m2m_100 finetuning not working (KeyError: none)
Author: sergej-schultenkaemper
Pull request: no

- `transformers` version: 4.5.0.dev0
- Python version: 3.8
- PyTorch version (GPU?): 1.7.1+cu110
- Using GPU in script?: Yes, RTX 3090
- Using distributed or parallel set-up in script?: No

I am trying to finetune m2m:

python3 run_translation.py \
    --model_name_or_path=facebook/m2m100_418M \
    --do_train \
    --do_eval \
    --source_lang de \
    --target_lang en \
    --fp16=True \
    --num_train_epochs 1 \
    --evaluation_strategy epoch \
    --dataset_name wmt15 \
    --dataset_config_name de-en \
    --output_dir /home/s/m2m_output/DE-EN \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate

And I'm getting this error:

All model checkpoint weights were used when initializing M2M100ForConditionalGeneration.

All the weights of M2M100ForConditionalGeneration were initialized from the model checkpoint at facebook/m2m100_418M.
If your task is similar to the task the model of the checkpoint was trained on, you can already use M2M100ForConditionalGeneration for predictions without further training.
Traceback (most recent call last):
  File "run_translation.py", line 562, in <module>
    main()
  File "run_translation.py", line 401, in main
    train_dataset = train_dataset.map(
  File "/home/s/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1120, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home/s/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1091, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "run_translation.py", line 382, in preprocess_function
    with tokenizer.as_target_tokenizer():
  File "/opt/conda/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/s/.local/lib/python3.8/site-packages/transformers/models/m2m_100/tokenization_m2m_100.py", line 299, in as_target_tokenizer
    self.set_tgt_lang_special_tokens(self.tgt_lang)
  File "/home/s/.local/lib/python3.8/site-packages/transformers/models/m2m_100/tokenization_m2m_100.py", line 312, in set_tgt_lang_special_tokens
    lang_token = self.get_lang_token(tgt_lang)
  File "/home/s/.local/lib/python3.8/site-packages/transformers/models/m2m_100/tokenization_m2m_100.py", line 318, in get_lang_token
    return self.lang_code_to_token[lang]
KeyError: None

@patrickvonplaten @patil-suraj Any ideas on how to fix this?

Thanks

Comments:
Hi @sergej-d 

The `run_translation.py` script now supports fine-tuning `M2M100` (see #11170), for this model you should now also pass the `--forced_bos_token` argument which is usually similar to the the `--target_lang` 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.