High CER when Fine Tuning TrOCR Transformers to make an OCR for arabic Language
Author: DjouadaFarouk
Pull request: no

Hello Everyone ,
I need Help /Guidance regarding Creating and arabic OCR using Transformers . I'm using ViT as encoder and arabert as decoder with their pretrained weights using it like this : 
```
from transformers import VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
    "google/vit-base-patch16-224-in21k", "aubmindlab/bert-base-arabertv02"
)
```
i followed this [Tutorial](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb) by [NielsRogge](https://github.com/NielsRogge) ( Thank you Niels ) . My issue is that after training i get poor results such as Caracter error rate is higher than 70% . 
ive Tried different encoders , Different decoders , Different epochs, Different learning rate , but i don't know what i miss to make it work. 
<html>
<body>
<!--StartFragment-->

Step | Training Loss | Validation Loss | Cer
-- | -- | -- | --
200 | 4.380500 | 4.674773 | 0.693091
400 | 4.213600 | 4.367142 | 0.777522
600 | 4.257300 | 4.247403 | 0.756554
800 | 3.277700 | 4.185711 | 0.712005
1000 | 2.831100 | 4.275863 | 0.748850
1200 | 2.250200 | 4.342288 | 0.788509
1400 | 2.237800 | 4.589494 | 0.768880

<!--EndFragment-->
</body>
</html>

Comments:
cc @NielsRogge 

Hi,

I would first try to overfit a single batch, as explained in [Karpathy's blog post](http://karpathy.github.io/2019/04/25/recipe/). This makes sure everything is set up properly.

Several folks have shown to successfully fine-tune TrOCR from pre-trained encoder + decoder checkpoints, e.g. for Japanese: https://huggingface.co/spaces/Detomo/Japanese-OCR (or this repo: https://github.com/kha-white/manga-ocr).

See also this thread for more info: https://github.com/microsoft/unilm/issues/627.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am having the same problem with the Bengali language as the cer is around 87%-89%. @DjouadaFarouk did your issue got solved?

I got cer of 0.28 on Arabic using trocr small by using an Arabic tokenizer and initializing the embedding layer and fc layer in the decoder from scratch, I would also recommend starting the positional encoding layer in the decoder from scratch if you started from trocr small weights. 

> I got cer of 0.28 on Arabic using trocr small by using an Arabic tokenizer and initializing the embedding layer and fc layer in the decoder from scratch, I would also recommend starting the positional encoding layer in the decoder from scratch if you started from trocr small weights.

Hi there, can you share your code to accomplish that? i'm also trying to replace the tokenizer on the pretrained trocr-small or trocr-base?

processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")

tokenizer2 = AutoTokenizer.from_pretrained("aubmindlab/bert-base-arabertv2")
processor.tokenizer = tokenizer2

train = Data_boxes(path, 'images/train/', 'labels/train', 2, processor = processor)
test = Data_boxes(path,'images/validation/', 'labels/validation', 2, processor = processor)

#model defining
model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-small-stage1")

# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
# make sure vocab size is set correctly
#model.config.vocab_size = model.config.decoder.vocab_size

model.decoder.config.vocab_size = processor.tokenizer.vocab_size
model.config.vocab_size = model.config.decoder.vocab_size
model.decoder.output_projection = nn.Linear(256,  processor.tokenizer.vocab_size)
model.decoder.model.decoder.embed_tokens = nn.Embedding(processor.tokenizer.vocab_size, 256, padding_idx=1)

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 64
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4

hope this helps get you more intuitions.

``Hi there, I was following a similar approach, but I have problems to load the model for inference. During training model works very well, i have even a CER of 0.08 which is a good averall CER and after each epoch I made an extra inference on one image and at the end the transcripcion is a good one, but maybe I don't have the good code to save or reload the model, can you share your save and load code for a model training in this fashion, for instans this is the code I'm using: 

```python
processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
processor.tokenizer = AutoTokenizer.from_pretrained("roberta-latin-scratch/checkpoint-400000") #my own roberta model
processor.save_pretrained('./processor')

#other code to load and transform GT

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id

model.decoder.config.vocab_size = processor.tokenizer.vocab_size
model.config.vocab_size = model.config.decoder.vocab_size
model.decoder.output_projection = nn.Linear(1024, processor.tokenizer.vocab_size)
model.decoder.model.decoder.embed_tokens = nn.Embedding(processor.tokenizer.vocab_size, 1024, padding_idx=1)

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 200
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 2

trainer.train()
trainer.save_model("trainer_trocr/checkpoints")
model.save_pretrained("trainer_trocr/final_model")

```

So I save the processor, the checkpoints and the final model, after that to load the model for inference there are two options: 

```python
p = 'linea.png'
processor = TrOCRProcessor.from_pretrained("./processor")
model = VisionEncoderDecoderModel.from_pretrained("trainer_trocr/final_model")
```
or making like that: 

```python
p = 'linea.png'

processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
processor.tokenizer = AutoTokenizer.from_pretrained("roberta-latin-scratch/checkpoint-400000")
model = VisionEncoderDecoderModel.from_pretrained("trainer_trocr/final_model")

```

In both cases the output is absurd, in the first I get something like that: ["le is is is lele one, etc.], using the second one I get a series of trigramms line that: ["the the the of of of one one one, etc."].  Do you have some suggestions about all that? 


2 things you have to do:



##first 
You have to to do same updates as training before loading the weights for inference:

```
# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id

model.decoder.config.vocab_size = processor.tokenizer.vocab_size
model.config.vocab_size = model.config.decoder.vocab_size
model.decoder.output_projection = nn.Linear(1024, processor.tokenizer.vocab_size)
model.decoder.model.decoder.embed_tokens = nn.Embedding(processor.tokenizer.vocab_size, 1024, padding_idx=1)

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 200
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 2
```

##Second thing I load weights using this:
```
model = model.from_pretrained("/path/to/checkpoint/pytorch_model.bin")
```

I have a question, do the tokenizer and image processor get updated during training? Because I think no.

No the tokenizer and image processor are static things. Only the model weights get updated during training.

@magistermilitum I would recommend fitting the model on a single example to see where the issue happens. Do you make sure you prepare the image + text for the model in the same way during training vs inference? 

I faced the same problem before, I think @magistermilitum forgot to change the special tokens before loading the weights.

Thanks for your answers, I was actually updating the model during inferences like this: 

```python
from transformers import TrOCRProcessor, VisionEncoderDecoderModel, AutoTokenizer
from transformers import AutoModel
from PIL import Image
import torch
import torch.nn as nn

p = 'linea.png'

processor = TrOCRProcessor.from_pretrained("./processor")

model = VisionEncoderDecoderModel.from_pretrained("trainer_trocr/checkpoint-15900")

#Upading model parameters
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.decoder.config.vocab_size = processor.tokenizer.vocab_size
model.config.vocab_size = model.config.decoder.vocab_size
model.decoder.output_projection = nn.Linear(1024, processor.tokenizer.vocab_size)
model.decoder.model.decoder.embed_tokens = nn.Embedding(processor.tokenizer.vocab_size, 1024, padding_idx=1)

model.config.decoder.activation_function="gelu"
model.config.decoder.layernorm_embedding=True
model.config.decoder.max_position_embeddings=512
model.config.decoder.scale_embedding=False
model.config.decoder.use_learned_position_embeddings=True

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 200
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 2

#inference
image = Image.open(p)
image_rgb = image.convert('RGB')

pixels = processor(image_rgb, return_tensors="pt").pixel_values
generated_ids = model.generate(pixels)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_text)
```

But in this way I get a list of repeted trigrams as inference. So maybe is the manner in what I'm loading the model. What is exactly the code are you using to load the .bin file? AutoModel.from_pretrained?

```python
from transformers import AutoModel
model = AutoModel.from_pretrained("trainer_trocr/checkpoint-15900/")
```

try to use this:

```
model = model.from_pretrained("/path/to/checkpoint/pytorch_model.bin")
```

Yeah, I mean, "model.from_pretrained" is not a native transformers function, so are you using a one specific or maybe the general "AutoModel.from_pretrained"?

I mean try to load the binary file directly "pytorch_model.bin"

ok, thanks, I have a working code for inference: after updating the model parameters we must simply load the weights from the bin and updating the trocr model, but at the end as I modified the config arquitecture during training:

```python
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-base-handwritten')

#Other Code uptading the TrOCR architecture

state_dict = torch.load("trainer_trocr/checkpoint-15900/pytorch_model.bin")
model.load_state_dict(state_dict)

```



I have a question. For the tokenizer, did you make a new one with only the words in your new vocabulary? For example I am using TrOCR to finetune on my own dataset. The tokens are completely different from the pretrained vocabulary. Should I make my own tokenizer with only my new vocabulary? Should I extend the existing vocabulary with my new vocabulary. This part is really confusing 

> I have a question. For the tokenizer, did you make a new one with only the words in your new vocabulary? For example I am using TrOCR to finetune on my own dataset. The tokens are completely different from the pretrained vocabulary. Should I make my own tokenizer with only my new vocabulary? Should I extend the existing vocabulary with my new vocabulary. This part is really confusing

Regarding this issue, there are many directions if you have just some characters that are not included in the tokenizer just add them, but if you completely have a new vocab and can't find any pretrained tokenizer on hugging face then it is better to train your own tokenizer to find the best combination of subwords. There is a tutorial on this on hugging face. 

@Hesham-Aliy  
hello Aliy , can you share you notebook and which dataset have you used ?

@Hesham-Aliy  i had a doubt since you are using a diff language tokenizer you are initializing the decoder embedding layer and decoder output projection layer . But the trainable parameters would become very large in this case . I tried this and the no. of trainable param went upto 100 million . I was using a spanish tokenizer whose vocab size is 50,265 and the hidden size of trocr-base-handwritten is 1024 . Kindly suggest me another way to do this as in this way it is going very computationally expensive for me.