m2m-100 finetuning messes up lang pairs
Author: ArtanisTheOne
Pull request: no

## Issue
Currently if I attempt to fine-tune M2M100 (many-to-many 100) on one language pair, what happens is the training data is convoluted and output translations from other language pairs are messed up. Further more, in the compute_metric which is ran when evaluate() is called and validation data is processed, the model/trainer returns predictions which, when decoded, are in a variety of languages, not in the target language which the reference text is in, making bleu calculations immensely low on trainer.evaluate(). I am hoping to be able to fine-tune specific language pairs and preserve the rest of the model's accuracy in it's other lang pairs.


## Environment info

- `transformers` version: 4.17.0
- Platform: `Windows-10-10.0.22000-SP0`
- Python version: 3.9.7
- PyTorch version (GPU?): `1.11.0+cu113 (True)`
- Tensorflow version (GPU?): `2.8.0 (True)`
- Flax version (CPU?/GPU?/TPU?): `not installed (NA)`
- Jax version: `not installed`
- JaxLib version: `not installed`
- Using GPU in script?: `Yes`
- Using distributed or parallel set-up in script?: `No`
- GPU Type: `RTX 3070`



Datasets used: [Totoeba Challenge EN-ES](https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/data/dev/eng-fra/dev.txt)


## Information

Model I am using:  **`M2M100 (418M)`**

The problem arises when using my own modified scripts of the official example scripts regarding the fine-tuning of translation models. Guide used is [here](https://huggingface.co/course/chapter7/4)

The tasks I am working on is finetuning the m2m-100 418M model on one language pair while preserving the other language pairs' weights and accuracy

## To reproduce

1. Use `Trainer` to finetune m2m-100 on one language pair with bitext data with script below

OR

1. Load [this finetuned m2m100 huggingface hub model](https://huggingface.co/NDugar/m2m100_418M-fr) and generate predictions for a variety of language pairs. (this model utilized the script below)
3. Try inferencing with english to french, then try influencing from english to russian, or any other language pair. You will receive messed up results with french vocab, punctuation and grammar. 

```python
from datasets import load_dataset, load_metric
from transformers import M2M100Tokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer
import torch


raw_datasets = load_dataset("text", data_files="./en-es.txt") #load bitext data file, can download from tatoeba challenge
split_datasets = raw_datasets["train"].train_test_split(train_size=0.9,test_size=0.001, seed=20) #i set the test size really low so I could view compute_metrics and evaluate faster to see messed up predictions clearer
split_datasets["validation"] = split_datasets.pop("test")

metric = load_metric("sacrebleu")

model_checkpoint = "facebook/m2m100_1.2B"
tokenizer = M2M100Tokenizer.from_pretrained(model_checkpoint)
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "es"

max_input_length = 128
max_target_length = 128

def preprocess_function(examples):
    inputs = []
    targets = []
    for ex in examples['text']: #split the dataset as it is still in bitext format (lang1 TAB lang2)
        split = ex.split("    ")
        if len(split) >= 2: #dont add empty lines or messed up data if there is any
            inputs.append(split[2]) #for totoeba challenge, first two inputs in array are the language codes
            targets.append(split[3]) 
        
    model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True)

    # Set up the tokenizer for targets
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(targets, max_length=max_target_length, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs
def compute_metrics(eval_preds): #compute bleu metrics of predictions and references
    #eval_preds made up op label_ids (reference translations) and predictions (the predicted translation) among other values
    reference_text = tokenizer.batch_decode(eval_preds.label_ids, skip_special_tokens=True) #reference translated texts
    translated_text = tokenizer.batch_decode(eval_preds.predictions, skip_special_tokens=True)
    metric_result = metric.compute(predictions=translated_text, references=reference_text) #compute bleu score
    print(metric_result) #will be outrageously low (i got 0.69 BLEU ~)
    return {"bleu": metric_result["score"]}

tokenized_datasets = split_datasets.map(
    preprocess_function,
    batched=True,
    remove_columns=split_datasets["train"].column_names,
) #use preprocess function on all data in dataset in batches of 1000

del raw_datasets 
del split_datasets #delete the text versions of dataset as has been converted to tensors be now

model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint) #load m2m100 model
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model) #load data collator

batch = data_collator([tokenized_datasets["train"][i] for i in range(1, 3)])

args = Seq2SeqTrainingArguments(
    f"m2m100_412M-test",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=1,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    eval_accumulation_steps=1,
    predict_with_generate=True,
    push_to_hub=False,
    do_train=True,
    do_eval=False,
    fp16=True,
)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

torch.cuda.empty_cache() #ensure torch cache is empty and ready

trainer.evaluate(max_length=max_target_length) #this will return a super super low bleu score, if you breakpoint the compute metrics function you'll see the decoded predictions are not in the target language but in an assortment of random ones, changing for every sentence
trainer.train() #training functions fine, however if you attempt to use the saved model after training from any other lang pair but en - es it will be messed up
```



## Expected behavior

The m2m-100 model and trainer should use the tokenized results to only train that specific language pair, (eg; en-es) instead of changing all language-pairs/weights with the target language. Hopefully you would be able to finetune language pairs while preserving the rest of the model, increasing the accuracy and translation output of specific highly used language pairs while keeping untouched language pairs alone and the same as original. 

@patrickvonplaten @LysandreJik


Comments:
cc @patil-suraj 

Hi @ArtanisTheOne ,

As specified in the [docs](https://huggingface.co/docs/transformers/model_doc/m2m_100), to generate output in a specific language with M2M100, we should set the `forced_bos_token_id` in `config` or pass it to `generate`. 
For example to generate translations in Spanish, in your script you should set.

```python
model.config.forced_bos_token_id = tokenizer.lang_code_to_id["es"]
```

> Hopefully you would be able to finetune language pairs while preserving the rest of the model, increasing the accuracy and translation output of specific highly used language pairs while keeping untouched language pairs alone and the same as original.

When we fine-tune a multilingual model on a specific language pair, it's very likely that the model's performance on other languages will drop. How to avoid this is more of a research question. 

> The m2m-100 model and trainer should use the tokenized results to only train that specific language pair, (eg; en-es) instead of changing all language-pairs/weights with the target language.

 It's a single model for all 100 language pairs and there are no language-pair specific weights, so when fine-tuning the model all weights will be updated. 

I understand, I’ll try to see if I can modify anything to achieve better results. Thanks for taking the time to respond.

@ArtanisTheOne as @patil-suraj said that modification bought you any changes in the results or there are any changes that you made and it worked for you?