Cannot answer questions about measurements
I have images of dogs and want to know their size. If I ask for the size of the pets, I always get "small" as an answer. If I then ask for specific heights, I get wrong answers. Anyone has an idea why this occurs and maybe how to cleverly prompt the model to still get correct results?
Hi
@mwieland
, this model can only answer questions for which it was trained. I had a similar experience than yourself for a different kind of question/domain.
What you need to do is create a dataset of dogs with questions and answers and then train (create a fine-tuned version) so it learns how to do this.
I created a dataset of 100 images with questions and answers for my problem and it performs at around 80-90% accuracy.
Depending on the complexity of the question, you might need more images.
I used this as starting point:
https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/ViLT/Fine_tuning_ViLT_for_VQA.ipynb
how do we do the Validation Accuracy?here?