TROCR custom dataset
Author: Jim0530
Pull request: no

Hi everyone 
I would like to train a visionencoderdecoder on a custom dataset
Which I hope my decoder can decode different  words from custom vocabulary
How can I train my own model without using any pretrained model
Thanks 

Comments:
Hi!

Thanks for your interest in using `VisionEncoderDecoderModel`.

The steps to proceed would be:
1. train a new tokenizer on your custom text data, using [HuggingFace Tokenizers](https://github.com/huggingface/tokenizers). Could be a BPE (byte pair encoding) tokenizer for instance. Next, initialize your tokenizer with the new vocabulary.
```
tokenizer = ...
```
2. choose a text model to use as decoder, e.g. BERT, GPT2, RoBERTA. Let's choose BERT - here randomly initialized: 
```
from transformers import BertConfig, BertModel

decoder_config = BertConfig()
decoder = BertModel(decoder_config)
```
3. make sure that the embedding layer of the Transformer that you'll use as decoder has the same length as your newly trained tokenizer, i.e run the following:
```
decoder.resize_token_embeddings(len(tokenizer))
```
4. initialize a `VisionEncoderDecoderModel` with an image encoder (e.g. ViT) and your new decoder (here I'm also initializing ViT with randomly initialized weights - not really recommended, better to start from a pretrained one actually):
```
from transformers import ViTConfig, ViTModel, VisionEncoderDecoderModel

encoder_config = ViTConfig()
encoder = ViTModel(encoder_config)
model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder)
```
5. fine-tune on your custom dataset

Thanks a lot 
now training is okay.
But how do I do the inference? Without labels feed into the decoder.
I am not very familiar with the model.generate method.
Hope u can give me some documents to learn. Thanks you!!!

### model
> encoder = BeitModel(BeitConfig())
decoder = TrOCRForCausalLM(TrOCRConfig(39))
model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder)

### config

>model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.config.vocab_size = model.config.decoder.vocab_size
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 16
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4


### Inference

> pixel_values = train_dataset[0]["pixel_values"].unsqueeze(0)
labels = train_dataset[0]["labels"].unsqueeze(0)
model.generate(pixel_values)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_18930/3987862010.py in <module>
----> 1 model.generate(pixel_values)

~/anaconda3/envs/sccheng-essay/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28 

~/anaconda3/envs/sccheng-essay/lib/python3.8/site-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, max_new_tokens, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, **model_kwargs)
    905         if self.config.is_encoder_decoder:
    906             # add encoder_outputs to model_kwargs
--> 907             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
    908 
    909             # set input_ids as decoder_input_ids

~/anaconda3/envs/sccheng-essay/lib/python3.8/site-packages/transformers/generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, input_ids, model_kwargs)
    414                 if not (argument.startswith("decoder_") or argument.startswith("cross_attn"))
    415             }
--> 416             model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
    417         return model_kwargs
    418 

~/anaconda3/envs/sccheng-essay/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

TypeError: forward() got an unexpected keyword argument 'attention_mask'


Yes, that's because `BeitModel` currently doesn't take `attention_mask` as input to its forward method.

However, we're working on fixing this, as vision models don't actually need the attention mask.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

What if the training set is Non-English ? Bert and existing pretrained data won't work right?