Help training TrOCR
Author: radu-diaconescu13
Pull request: no

Hello,

Thank you for your hard work on the repo.

I want to train the newly added TrOCR for my use case on my dataset.

I am a little confused on how to proceed.
https://huggingface.co/transformers/training.html
The finetuning tutorial seems to be geared towards strictly NLP models.

Am I missing something? Is there some documentation somewhere related to how to train 
the TrOCR with images and text?

Any pointers or help would be greatly appreciated.

Thank you

Comments:
You're in luck, I just uploaded a notebook showcasing how to fine-tune TrOCR on a custom dataset.

You can find it here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb

Regarding documentation, this can be found in the docs of `VisionEncoderDecoderModel` [here](https://huggingface.co/transformers/master/model_doc/visionencoderdecoder.html) - soon available in the new v4.12 release.

In addition to that, it would be great to have a tutorial without transformers trainer for same task but with the `VisionEncoderDecoderModel `  :hugs:  

Do you mean using regular PyTorch?

yes or lightning

and if it possible also a way to export this model into ONNX (if it is possible - i know a bit trouble with .generate() /greedy search)

Sure, I can do that. It would be very similar, except that for computing metrics during evaluation one needs to use the `generate()` method instead of just doing a forward pass and argmaxing the logits.

@NielsRogge 
i have also created a [Colab Notebook](https://colab.research.google.com/drive/1PZz8oGH3pZEjDpMJex2nZ5p8MiYXzYA8?usp=sharing) with my try but fail at the val & predict step

this would be great :hugs: what do you think about the ONNX part ? Can you show some example also in a tutorial ?  I think this would be awesome and very useful also for EncoderDecoderModel and the speech counterpart

So i will keep a eye on your tutorials repo much thanks for this

@NielsRogge  Thank you for the very, very quick response. I will start looking into the notebook provided.


@felixdittrich92 fine-tuning TrOCR using native PyTorch notebook is here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb

@NielsRogge nice thank you :) 
but overall i do the same with VisonEncoderDecoderModel in val and it ooms (in val on batch_size=2) instantly (T4 16GB) any idea ?
I have also a very similar model (ViTSTR) where i have no problems to fit a batch_size of 64 (same data)

```
def validation_step(self, batch, batch_idx):
        image_tensors = batch['pixel_values']
        labels = batch['input_ids']

        plain_labels = self.tokenizer.batch_decode(labels, skip_special_tokens=True)

        loss, logits = self(pixel_values=image_tensors, labels=labels)

        preds = logits.argmax(-1) # [batch_size, seq_len]
        prob = torch.softmax(logits, -1).max().item()
      
        generated_ids = self.vit_bert_model.generate(image_tensors).detach() 
        plain_preds = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

        # compute word error rate
        word_error_rate = wer(predictions=plain_preds, references=plain_labels)
        # compute character error rate
        char_error_rate = cer(predictions=plain_preds, references=plain_labels) 

        self.log("val_loss", loss.detach(), prog_bar=True, on_step=False, on_epoch=True, sync_dist=self.sync_dist)
        self.log("val_acc", accuracy(preds, labels).detach(), prog_bar=True, on_step=False, on_epoch=True, sync_dist=self.sync_dist)
        self.log("word_error", torch.tensor(word_error_rate).detach(), prog_bar=True, on_step=False, on_epoch=True, sync_dist=self.sync_dist)
        self.log("char_error", torch.tensor(char_error_rate).detach(), prog_bar=True, on_step=False, on_epoch=True, sync_dist=self.sync_dist)

        return loss
```

```
self.vit_bert_model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(vit_backbone, 'bert-base-multilingual-cased', return_dict=True)
self.vit_bert_model.config.decoder.decoder_start_token_id = self.tokenizer.cls_token_id # 101
self.vit_bert_model.config.decoder.eos_token_id = self.tokenizer.sep_token_id # 102
self.vit_bert_model.config.decoder.pad_token_id = self.tokenizer.pad_token_id # 0
```

You can wrap everything in a `with torch.no_grad()` to save memory.

@NielsRogge 
Thats lightning if I'm not totally wrong, this will be done automatically in the val step :)
So i have no reason for this i pad the labels up to a size of 128

I use currently deit-base-distilled weights for the Encoder and bert-base-multilingual-cased as Decoder

Hello @NielsRogge 

Reopening this issue with a question.

Thank you for the notebook. It is very helpful.

I am trying to finetune the TrOCR to be able to recognize numbers of up to maximum 9 characters with and without a decimal point.

I have 2 questions: 
1. Do I need to train a custom tokenizer or is the pretrained Roberta one included with TrOCR good enough?
2. Do you have any recommendations regarding model parameters that would be better suited for my use case?
Thing like number of beams for beam search, max length, vocab size, etc? 

Thank you

Thank you so much for your hard work!
I'm starting work with NER and OCR. You helped me a lot.

To use this model in the Portuguese language, I need to train with a text base in Portuguese, right?
Do you know any tools to help me with the creation of this base?



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> Do I need to train a custom tokenizer or is the pretrained Roberta one included with TrOCR good enough?

You don't need to train a custom tokenizer, the one of RoBERTa can be used.

> Do you have any recommendations regarding model parameters that would be better suited for my use case?
Thing like number of beams for beam search, max length, vocab size, etc?

Not really, the authors used beam search with num_beams=5, other than that, all default parameters of the [generate](https://huggingface.co/docs/transformers/master/en/main_classes/model#transformers.generation_utils.GenerationMixin.generate) method were used.

> To use this model in the Portuguese language, I need to train with a text base in Portuguese, right?

Yes, the TrOCR model that Microsoft released was only trained on English image/text pairs, however they are planning to release a multilingual variant. The problem is that RoBERTa's tokenizer only includes tokens for the English language, so one would need to train the TrOCR model from scratch, starting from a multilingual (or Portuguese) text Transformer as decoder.

Thank you very much for the information!

I googled and found BERTibaum, link:
https://github.com/neuralmind-ai/portuguese-bert
This model is pre-trained in Portuguese.
Would it be possible to use this model in the decoder?

Em seg., 6 de dez. de 2021 às 12:20, NielsRogge ***@***.***>
escreveu:

> Do I need to train a custom tokenizer or is the pretrained Roberta one
> included with TrOCR good enough?
> You don't need to train a custom tokenizer, the one of RoBERTa can be used.
>
> Do you have any recommendations regarding model parameters that would be
> better suited for my use case?
> Thing like number of beams for beam search, max length, vocab size, etc?
> Not really, the authors used beam search with num_beams=5, other than
> that, all default parameters of the generate
> <https://huggingface.co/docs/transformers/master/en/main_classes/model#transformers.generation_utils.GenerationMixin.generate>
> method were used.
>
> To use this model in the Portuguese language, I need to train with a text
> base in Portuguese, right?
> Yes, the TrOCR model that Microsoft released was only trained on English
> image/text pairs, however they are planning to release a multilingual
> variant. The problem is that RoBERTa's tokenizer only includes tokens for
> the English language, so one would need to train the TrOCR model from
> scratch, starting from a multilingual (or Portuguese) text Transformer as
> decoder.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/14195#issuecomment-986876156>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AJHYQQ3CI4MNGCA2GKPCK43UPTIFDANCNFSM5G5DPWFQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>


-- 
*Celso Antônio M Lopes Júnior*
Computer Engineering - computational Intelligence
Researcher of Artificial Neural Networks, Deep Learning, and Computer Vision
+55 81 99246-9364 <+55%2081%2099246-9364>
***@***.***
_______________________________________________________


Hi,

Sorry for the late reply, but yes you can combine a vision encoder (like `google/vit-base-patch16-in21k`) with a language decoder in a different language (like `neuralmind/bert-base-portuguese-cased`), assuming you have enough data to fine-tune the model on image -portuguese text pairs.

Hello. 
Thank you very much for sharing code and models. It is really great and useful.
I am wondering whether I can ask a relevant question from here. 

I am trying to train a new VisionEncoderDecoder model for new language (Bahasa-Indonesian).
The initial performance is pretty bad (72% CER). I am wondering whether I can get some advices.

Based on my understanding, Bahasa-Indonesian uses Alphabet characters (without special characters such as German Epsilon). So my initial model is as follows:

Encoder: Pretrained ViT model (“google/vit-base-patch16-224”)
Decoder: Pretrained Indonesian Roberta Language model ("'cahya/roberta-base-indonesian-1.5G")

For updating decoder weights using Pertained Roberta model, I have a few questions.
Currently, the parameter names from Roberta models are different from Decoder model parameters, so we need some mapping process. I did the following steps, and i am wondering whether there are some errors.

encoder
encoder = ViTModel.from_pretrained(“google/vit-base-patch16-224”)

decoder
lmconfig = TrOCRConfig(vocab_size=indonesian_lm_vocab_size, …, ) # update accordingly by indonesian lm
decoder = CausualLM(lmcfg)
lm = RobertaForCausalLM.from_pretrained(“cahya/roberta-base-indonesian-1.5G”)
decoder = load_wts(decoder, lm)

def load_wts(decoder, lm):
    param_name_dict = {
        ‘attention.self.query.weight’: ‘self_attn.q_proj.weight’,
        ‘attention.self.query.bias’: ‘self_attn.q_proj.bias’,
        ‘attention.self.key.weight’: ‘self_attn.k_proj.weight’,
        ‘attention.self.key.bias’: ‘self_attn.k_proj.bias’,
        ‘attention.self.value.weight’: ‘self_attn.v_proj.weight’,
        ‘attention.self.value.bias’: ‘self_attn.v_proj.bias’,
        ‘attention.output.dense.weight’: ‘self_attn.out_proj.weight’,
        ‘attention.output.dense.bias’: ‘self_attn.out_proj.bias’,
        ‘attention.output.LayerNorm.weight’: ‘self_attn_layer_norm.weight’,
        ‘attention.output.LayerNorm.bias’: ‘self_attn_layer_norm.bias’,
        ‘output.LayerNorm.weight’: ‘final_layer_norm.weight’,
        ‘output.LayerNorm.bias’:‘final_layer_norm.bias’,
        ‘word_embeddings.weight’:‘embed_tokens.weight’,
        ‘position_embeddings.weight’: ‘embed_positions.weight’,
        ‘LayerNorm.weight’:‘layernorm_embedding.weight’,
        ‘LayerNorm.bias’: ‘layernorm_embedding.bias’
    }
   wts = lm.state_dict()
   new_wts = {} 
   dwts = decoder.state_dict()
   for key in wts.keys():
       nkey = rename_param(key)
       if nkey:
           new_wts[nkey] = wts[key]
   decoder.load_state_dict(new_wts, strict=False)
   return decoder

As paper mentioned, this does not update weights in the encoder-decoder attention layers
since they do not exist in Roberta language model.

If you have any advice, please let me know. Thank you a lot.

> You're in luck, I just uploaded a notebook showcasing how to fine-tune TrOCR on a custom dataset.
> 
> You can find it here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb
> 
> Regarding documentation, this can be found in the docs of `VisionEncoderDecoderModel` [here](https://huggingface.co/transformers/master/model_doc/visionencoderdecoder.html) - soon available in the new v4.12 release.

Are there any pretrained models of Japanese, Korean, etc.? Or how to train them? steve8000818@gmail.com

Hi!

If you want to train a TrOCR-like model on another language, you can initialize (also called "warm-start") the weights of the encoder and decoder of a `VisionEncoderDecoderModel` with those of any compatible checkpoint available on the hub. One then only adds randomly initialized cross-attention layers in the decoder, which need to be fine-tuned on a supervised dataset of image-text pairs. 

For instance, let's say you want to do OCR in Japanese, then you can initialize a `VisionEncoderDecoderModel` with ViT as encoder and https://huggingface.co/cl-tohoku/bert-base-japanese as decoder. This can be done as follows:

```
from transformers import VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained("google/vit-base-patch16-224-in21k", "cl-tohoku/bert-base-japanese")

# ensure that randomly initialized cross-attention layers are added
assert model.config.decoder.is_decoder is True
assert model.config.decoder.add_cross_attention is True
```

Thank you for your reply.
I wonder whether this tr is same as below link. Below link mainly uses [libtr.so or dll](https://github.com/myhub/tr/blob/master/tr/libtr.so) to realize Chinese ocr function, it is small and fast, any similar method or tool for Japanese, Korean?
https://github.com/myhub/tr

> Hi,
> 
> Sorry for the late reply, but yes you can combine a vision encoder (like `google/vit-base-patch16-in21k`) with a language decoder in a different language (like `neuralmind/bert-base-portuguese-cased`), assuming you have enough data to fine-tune the model on image -portuguese text pairs.

How about handwritten text in indonesian?

You can check languages here: https://huggingface.co/languages.

If you check Indonesian, there are currently 135 models available for this language. An example could be this one: https://huggingface.co/indobenchmark/indobert-base-p1

How to train crnn ocr model with transformer? steve8000818@gmail.com

> You can check languages here: https://huggingface.co/languages.
> 
> If you check Indonesian, there are currently 135 models available for this language. An example could be this one: https://huggingface.co/indobenchmark/indobert-base-p1

Am I doing right? @NielsRogge 

```python
from transformers import ViTFeatureExtractor, RobertaTokenizer, TrOCRProcessor

encode = 'google/vit-base-patch16-224-in21k'
decode = 'cahya/roberta-base-indonesian-1.5G'

feature_extractor=ViTFeatureExtractor.from_pretrained(encode)
tokenizer = RobertaTokenizer.from_pretrained(decode)
processor = TrOCRProcessor(feature_extractor=feature_extractor, tokenizer=tokenizer)

from transformers import TrOCRProcessor

train_dataset = IAMDataset(root_dir='../dataset/',
                           df=train_df,
                           processor=processor)
eval_dataset = IAMDataset(root_dir='../dataset/',
                           df=test_df,
                           processor=processor)

model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(encode, decode)

# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
# make sure vocab size is set correctly
model.config.vocab_size = model.config.decoder.vocab_size

config_decoder.is_decoder = True
config_decoder.add_cross_attention = True

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 64
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4
```

hello @NielsRogge, I'm trying to train TrOCR model for Arabic handwritten OCR, the model and preprocessor are as follows:

```
def load_model(from_disk: bool) -> VisionEncoderDecoderModel:
    model: VisionEncoderDecoderModel = VisionEncoderDecoderModel.from_encoder_decoder_pretrained("google/vit-base-patch16-224-in21k", "bhavikardeshna/xlm-roberta-base-arabic")#.from_pretrained(model_path)
    print(f"Using device {device}.")
    model.to(device)
    return model

def init_model_for_training(model: VisionEncoderDecoderModel, processor: TrOCRProcessor):
    model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
    model.config.pad_token_id = processor.tokenizer.pad_token_id
    model.config.vocab_size = model.config.decoder.vocab_size
    model.config.bos_token_id = processor.tokenizer.bos_token_id
    model.config.decoder_start_token_id = 0
    model.config.decoder.is_decoder = True
    model.config.decoder.add_cross_attention = True


def load_processor() -> TrOCRProcessor:
    feature_extractor=ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
    model_path = "bhavikardeshna/xlm-roberta-base-arabic"
    tokenizer = AutoTokenizer .from_pretrained(model_path)
    return TrOCRProcessor(feature_extractor=feature_extractor, tokenizer=tokenizer)

```
However, when I check the validation set predictions It's all in English like garbage.
what is the problem here?


Hi,

thanks for your interest in TrOCR! So you're instantiating the weights of the decoder with those of [bhavikardeshna/xlm-roberta-base-arabic](https://huggingface.co/bhavikardeshna/xlm-roberta-base-arabic), which looks like a multilingual model (XLM-RoBERTa), fine-tuned on an Arabic question answering dataset. I'd advise here to use a monolingual (Arabic-only) model instead. 

One option here is AraBERT: https://huggingface.co/aubmindlab/bert-base-arabertv2. This seems to be one of the most popular Arabic-only text models.

Also note that in your case, instantiating the model as follows:

```
from transformers import VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained("google/vit-base-patch16-224-in21k", "aubmindlab/bert-base-arabertv2")
```
is sufficient, you don't need to do `model.config.decoder.is_decoder = True` and `model.config.decoder.add_cross_attention = True` as the cross-attention layers are already automatically added (you can see that in the warning, which tells you that those layers will have randomly initialized weights).

I'd also check the data preparation really well (make sure to decode the labels just before feeding them to the model), to make sure you're really preparing the data appropriately.

Hi @NielsRogge .. Question please what did you mean by decoding the labels before feeding them to the model? should it be feed to the model in the training as text?

Also please, the MSE loss calculated basically by the model is so small while finetuning on a handwritten Arabic dataset (tends to 0) given a high CER, WER..

when I tried the crossentropy the loss values are huge values (number of 8 digits) so what could this behavior indicate?

Hi @NielsRogge,
When I train the model using huggingface intergrated deepspeed. There is an error.

`AttributeError: 'VisionEncoderDecoderConfig' object has no attribute 'hidden_size'`

Do you know how to config the hidden_size to avoid the error?

Hi @NielsRogge,

Congrats for the Great work for State of Art Tr-OCR, i have few questions,
1.How can be tune the TrOCR to perform on multi lines document,as currently it works only on single line text.
2.How can we train on HIndi Language
3.How can we export the OCR model to ONNX.
4.How to control latency as I was trying to do some batch processing on passing ROIs tensors stacked to the Base printed model


Thanks

Hi,

> 1.How can be tune the TrOCR to perform on multi lines document,as currently it works only on single line text.

I guess you can try fine-tuning the model on (image, multi-line text) pairs, and see whether the model is able to pick that up. Note that pre-training only happened on single-line text images.

> 2.How can we train on HIndi Language

See the thread above, it comes down to replacing the text decoder with a pre-trained one from the hub and fine-tune the model on (image, hindi text) pairs

> 3. How can we export the OCR model to ONNX.

We've recently added ONNX support for the VisionEncoderDecoderModel class, check details here: https://github.com/huggingface/transformers/pull/19254.

> 4.How to control latency as I was trying to do some batch processing on passing ROIs tensors stacked to the Base printed model

I'd recommend taking a look at our Optimum library, which allows to optimize/quantize Transformer-based models such as TrOCR: https://huggingface.co/docs/optimum/index

Thanks  @NielsRogge  for answering really appreciate your work, but can you please elaborate for 1 and 2 point above or provide some good references.

Has anyone achieved a successful implementation of this method for other languages rather than English? I have come across some difficulty in my efforts and, would appreciate some feedback as to whether its a viable option to pursue.

@vick998 well I am currently working on support to Multiple text line recognition support for Tr-OCR (English) and improving model inference. For other languages I would suggest it's worth trying focusing on pretrained language decoder and fine tuning the model in image -text pairs

@NielsRogge  Can eloborate on other language model development 

I have fine tuned the trocr small printed model on a custom dataset. the pytorch model provides a good accuracy and shows a decrease in CER. I want to convert the same to ONNX. After converting the model to onnx, my accuracy decreases by more than 20 %. Can anyone explain why this is happening.
Thanks


Hey, I am fine tuning TrOCR for OMR (Optical Music Recognition). So trying to read sheet music. I am using microsoft/trocr-small-stage1 as the pretrained model. I have about 80k single staff (single line) sheet music snippets (aka 80k images-text pair). With some changes to the tokenizer and some tweaks. I am getting about 63% ( Word level accuracy). I wanted to know how I can improve the performance for WER (word error rate). Can I used bert-base-uncased as a decoder?