Custom dataset fine tuning
Hello! Can anyone please tell me how we can use our own custom data to fine-tune Vilt-b32-finetuned-VQA? Answer will be really appreciable! Thank you !
I am also curious about this @ChinmayBH Did you figure it out?
@ChinmayBH Use this as starting point and then modify it so it can train on your own dataset:
I have the fine tuning notebook running but the training results are always very poor for me.
I was expecting to at least see the overfit. Should I expect this from the notebook as it exists?
Setup:
I am using the same dataset that the notebook is using. I wanted to get that working before trying a custom dataset.
I tried it without changes to the notebook and with a variety of settings for learning rate and epochs.
Running on Jupyter notebook. Torch 2.1.1_cu118
Example: dataset[0], the image looking through the net.
" what is this photo taken looking through? "
The type of result I get... Sometimes "net" might show up in the list but it is never the best answer.
0.338682621717453 yes
0.27205798029899597 no
0.036353662610054016 2
0.0306074358522892 skiing
0.02998974919319153 1