`blip` support for training
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://discuss.huggingface.co/t/finetune-blip-on-customer-dataset-20893/28446 
Before this PR, it was not possible to fine-tune BLIP on a custom dataset due to various reasons, mainly because the code did not supported "on-the-fly" right shifting of `decoder_input_ids`.
This PR also harmonizes some attributes inside `BlipForQuestionAnswering` --> I replaced  `decoder_bos_token_id ` by `decoder_start_token_id`  to make it consistent with T5 etc.

For all VQA models we should (at train time):
1- make sure `labels` is not None
2- create `decoder_input_ids` based on those (make sure the padding is always on the right side)
3- Infer on the text decoder 

I feel that we should probably add more tests and create a `VisualQuestionAnsweringMixin` in a follow up PR to make sure this is done for all VQA models (as I'd expect more VQA models to be added this year)

cc @NielsRogge @sgugger 

Comments:
Perfect, thanks for clarifying @sgugger ! 

_The documentation is not available anymore as the PR was closed or merged._

@younesbelkada @sgugger hi, thanks for contributing this code, but I found two possible bugs:

1. the code shift `labels` to `decoder_input_id` ([here](https://github.com/huggingface/transformers/pull/21021/files#diff-e483643fc206cde147f2483924507d9a407db540b01bf4028c72b8ec6cc3ffabR1209)) and the code shift `labels` when computing loss [(here)](https://github.com/huggingface/transformers/pull/21021/files#diff-00846f08e1b2a41509f5f669a49fc36baac8555a83da24c30f8ac9e7a9024d59R900) should only keep one, and I prefer to keep the former one and delete the later.
2. The BERT tokenizer has added a start token before the sequence, and the `_shift_right` function will add another one (pad), so it should use `forced_bos_token_id` like BART for generation.

Moreover, I think the [`reduction`](https://github.com/huggingface/transformers/pull/21021/files#diff-e483643fc206cde147f2483924507d9a407db540b01bf4028c72b8ec6cc3ffabR1227) function of `CrossEntropyLoss` should be set to `'mean'`, or you will get a loss more than tens or hundreds, which is uncommon and may affect the optimization.

Thanks for your valuable comments @StevenTang1998! @younesbelkada in any case it would probably be best to have verified this branch in a notebook on a toy image captioning dataset. Making the code as similar as possible to our other generative models (like T5, BART or GPT-2) would be great.

Hi @younesbelkada, I encountered the same error as mentioned by @dxlong2000.
I cloned this repository but the error is still there.

ValueError: Expected input batch_size (0) to match target batch_size (29).

Hi @faiqff94 
All the issues related to BLIP training should be resolved, if you follow what has been done in https://colab.research.google.com/drive/1lbqiSiA0sDF7JDWPeS0tccrM85LloVha?usp=sharing you should not get any issue. Can you share a reproducible handy script?

> Hi @faiqff94 All the issues related to BLIP training should be resolved, if you follow what has been done in https://colab.research.google.com/drive/1lbqiSiA0sDF7JDWPeS0tccrM85LloVha?usp=sharing you should not get any issue. Can you share a reproducible handy script?

Hi @younesbelkada, I have tried the colab's script in local PC, but got loss: nan in epoch-0

Any advice? Thanks