TrOCR Encoder &Decoder Replacement 
Author: Mohammed20201991
Pull request: no

Hi @NielsRogge thanks for the great tutorial  For TrOCR
1- can I get the whole datasets for IAM as the format (processed) used during fine-tuning because I only can see the test set ?
2- I want to replace the Encoder with **Vit or Swin or Diet** and the Decoder with **Bert** or **GPT-2** or another beast decoder In the original TrOCR or at least modify the decoder part 
but I got a huge CER= 76% Please can you suggest what a possibility to reach a beast result and later on wanna fine-tune with specific language 
import fun
[# modifying  the tokenizer 
processor.tokenizer = fun.AutoTokenizer.from_pretrained('bert-base-uncased')



def trocr_model_config(model):
    # set decoder config to causal lm
    model.config.decoder.is_decoder = True
    model.config.decoder.add_cross_attention = True


    # set special tokens used for creating the decoder_input_ids from the labels
    model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
    assert model.config.decoder_start_token_id == processor.tokenizer.cls_token_id
    model.config.pad_token_id = processor.tokenizer.pad_token_id
    # make sure vocab size is set correctly
    model.config.vocab_size = model.config.decoder.vocab_size

    # set beam search parameters
    model.config.eos_token_id = processor.tokenizer.sep_token_id
    model.config.max_length = 128
    model.config.early_stopping = True
    model.config.no_repeat_ngram_size = 3
    model.config.length_penalty = 2.0
    model.config.num_beams = 4
    return model

def main():
    df = load_dataset()
    print(df.head(4))
    train_dataset, eval_dataset = create_datasets(df)

    print("Number of training examples:", len(train_dataset))
    print("Number of validation examples:", len(eval_dataset))

    encoding = train_dataset[0]
    for k,v in encoding.items():
        print(k, v.shape)

    labels = encoding['labels']
    labels[labels == -100] = processor.tokenizer.pad_token_id
    label_str = processor.decode(labels, skip_special_tokens=True)
    print(label_str)
    
    model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained('google/vit-base-patch16-384','bert-base-uncased')
    # setting model configuration
    configured_model = trocr_model_config(model)

    training_args = Seq2SeqTrainingArguments(
        predict_with_generate=True,
        evaluation_strategy="steps",
        learning_rate=2e-5,
        num_train_epochs=12,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        fp16=True,
        output_dir= f'./models/vit_bert_IAM{datetime.now().strftime("%Y%m%d%H%M%S")}',
        logging_steps=100,
        save_steps=1000,
        eval_steps=500,    
        )

    # instantiate trainer
    trainer = Seq2SeqTrainer(
        model=configured_model,
        tokenizer= processor.feature_extractor,
        args=training_args,
        compute_metrics= compute_metrics,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator= default_data_collator,
    )

    trainer.train()

if __name__ == '__main__':
    main()


Comments:
@sgugger    #Bert #Good_new_ssue

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge ping on this one.

> can I get the whole datasets for IAM as the format (processed) used during fine-tuning because I only can see the test set

The TrOCR authors have released the dataset [here](https://github.com/microsoft/unilm/tree/master/trocr), so I'd recommend taking a look at the unilm issues and perhaps ask the TrOCR authors to release it

 > I want to replace the Encoder with Vit or Swin or Diet and the Decoder with Bert or GPT-2 or another beast decoder In the original TrOCR or at least modify the decoder part

Note that if you replace the encoder/decoder, you'll need to fine-tune the model on additional (image, text) pairs. For that I'd recommend checking out [this thread](https://github.com/huggingface/transformers/issues/15823). Also check my [demo notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR) for tutorials on fine-tuning TrOCR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.