Help with TrOCR training for Spanish
Author: rubenaros
Pull request: no

### System Info

I'm using Torch 2.0 and Transformers 4.28.0 running in a Google Colab




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi 

Thanks @NielsRogge for your incredible work in Transformers. 

I'm working in develop a handwritten system recognition  in spanish, so i choose TrOCR for the traduction from handwrite to text in spanish.

I think I followed yours Notebooks examples to fine tune and inference with TrOCR and many post with people with the same problem when we need to train TrOCR in a diferent language, spanish for me.



The code to create dataset is:

 
class SpanishDataset(Dataset):
    

     def __init__(self, root_dir, df, processor, max_target_length=128):
        self.root_dir = root_dir
        self.df = df
        self.processor = processor
        self.max_target_length = max_target_length

     def __len__(self):
         return len(self.df)

     def __getitem__(self, idx):
         # get file name + text 
        file_name = self.df['Path'][idx]
        text = self.df['Text'][idx]
        # prepare image (i.e. resize + normalize)
        image = Image.open(self.root_dir + file_name).convert("RGB")
        pixel_values = self.processor(image, return_tensors="pt").pixel_values
        # add labels (input_ids) by encoding the text
        labels = self.processor.tokenizer(text, padding="max_length",max_length=self.max_target_length).input_ids
        # important: make sure that PAD tokens are ignored by the loss function
        labels = [label if label != self.processor.tokenizer.pad_token_id else -100 for label in labels]

        encoding = {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(labels)}
        return encoding


and 
 
```
 feature_extractor = AutoFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
decoder_tokenizer = AutoTokenizer.from_pretrained("bertin-project/bertin-roberta-base-spanish")

processor =TrOCRProcessor(feature_extractor=feature_extractor, tokenizer=decoder_tokenizer)

processor.save_pretrained('./processor')

processor = TrOCRProcessor.from_pretrained("./processor")


train_dataset = SpanishDataset(root_dir='/ShardDrives/MyDrive',
                           df=train_df,
                           processor=processor)
eval_dataset = SpanishDataset(root_dir='/ShardDrives/MyDrive',
                           df=test_df,
                           processor=processor) 
```

I  use this encoder and decoder:


```
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
    "google/vit-base-patch16-224-in21k", "bertin-project/bertin-roberta-base-spanish")

 
# set decoder config to causal lm
model.config.decoder.is_decoder = True
model.config.decoder.add_cross_attention = True
# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
# make sure vocab size is set correctly
model.config.vocab_size = model.config.decoder.vocab_size

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 64
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4


# ensure that randomly initialized cross-attention layers are added
assert model.config.decoder.is_decoder is True
assert model.config.decoder.add_cross_attention is True
```

I'm using cer as metric 

```
def compute_metrics(pred):
    labels_ids = pred.label_ids
    pred_ids = pred.predictions

    pred_str = processor.batch_decode(pred_ids, skip_special_tokens=True)
    labels_ids[labels_ids == -100] = processor.tokenizer.pad_token_id
    label_str = processor.batch_decode(labels_ids, skip_special_tokens=True)

    cer = cer_metric.compute(predictions=pred_str, references=label_str)

    return {"cer": cer}
```

and the training code is:

```
training_args = Seq2SeqTrainingArguments(
 
    evaluation_strategy="epoch",
    learning_rate=2e-4,    
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=100,
    output_dir="./",
    predict_with_generate=True,  
)   

# Training
trainer = Seq2SeqTrainer(
    model=model,
    tokenizer=processor.feature_extractor,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=default_data_collator
)
trainer.train()
```

But the Cer is bad (> 0.9) 

This is an output for a dataset element

```

train_dataset[0]

{'pixel_values': tensor([[[ 0.9922,  1.0000,  1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          ...,
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]],
 
         [[ 0.9922,  1.0000,  1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          ...,
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]],
 
         [[ 0.9922,  1.0000,  1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          ...,
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]]]),
 'labels': tensor([    0,  1323,   344,  2858, 11966,    66, 11507,  3298,   344, 14783,
            66,     2,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100])}
```

I have an error, but i don't know where, so any help or advice  is welcome.

Thank very much for everything

### Expected behavior

I expect to train a handwrite recognition system for spanish using TrOCR

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Working on that as well, I'm pretty sure the problem is the lenght of your dataset, if i'm not wrong, the tokeniser takes parts of the words and relates it to fragments of the image, so you need thousands of pics with different words so the model can get enought embbedings. Deduced that since the results with my train dataset were perfect, but for the eval dataset was terrible. You can also notice this since if you use the pretrained model without fine tuning and with an image with spanish text, the output will be mostly words that mostly sounds like english. Only way to fix this is to make a dataset as big as IAM

Maybe the batch size, or the learning rate