how to finetune huggingface MarianMT transformer model
Author: sravan-tadiparthi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
Python3.8
Linux RHEL 6.9

- `transformers` version: 4.10.3
- Platform: Linux RHEL
- Python version: 3.8
- PyTorch version (GPU?): 1.7.1
- Tensorflow version (GPU?):
- Using GPU in script?: No (for small dataset)
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:
MarianMT
- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): MarianMT

The problem arises when using:
* [ ] the official example scripts: (give details below) - i dont see script for translation
* [ ] my own modified scripts: (give details below) - i am using simple transformer library & with customized code too

The tasks I am working on is:
* [ ] my own task or dataset: (give details below) 

## To reproduce

Steps to reproduce the behavior:
I am very new with this transformers, requesting you to help me with this
1. Can you please help me code snippet with MarianMT fine-tune model for id (indonesia) to en (english) translation
(or)
2. i will share code snippet i am using, with this code i can able to train the model (it has not created vocab.json, source.spm & target.spm files) but while loading the model. it is expecting vocab files (vocab.json, source.spm & target.spm). Please help me to create these vocab files. Also let me know is this correct way i am following. Please put me in correct way
(trained model with 2 variations using Trainer API & simple transformers library)

#### Sample code i am using
****************** Using Trainer API *************************

from transformers import MarianTokenizer, MarianMTModel
from datetime import datetime

print("start time:", datetime.now())

path= "Lang Translation/opus-mt-id-en"

tokenizer = MarianTokenizer.from_pretrained(path)
model = MarianMTModel.from_pretrained(path)

from transformers import LineByLineTextDataset
train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="Lang Translation/dataset/train.json",
    block_size=128,
)

*** Sample text from train.json
{"translation": {"id": "Perahu Pinisi yang kokoh dapat dikatakan sebagai salah satu kerajinan kayu yang terkuat di Indonesia. Dibangun untuk bertahan di angin yang berkecamuk dan lautan yang berbadai, kapal yang abadi ini tidak hanya dapat bermanuver saat mengarungi perairan yang bergelombang, layar yang terbentang dapat menangkap dorongan angin \u2013 sebagai elemen yang penting dalam memberikan perlindungan.", "en": "The majestic Phinisi Yacht is said to be the strongest wooden craft in Indonesia. Built to survive raging winds and stormy seas, this timeless vessel is not only good at maneuvering in choppy waters, its sails can be angled to catch the winds \u2014 thereby harnessing the powerful elements to propel itself to safety."}}

test_dataset= LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="Lang Translation/dataset/test.json",
    block_size=128,
)

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False, mlm_probability=0.15
)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="results",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
)

trainer.train()

trainer.save_model("results")

********************** Using Simple Transformer Library **************************

import logging
import pandas as pd
from simpletransformers.seq2seq import Seq2SeqModel, Seq2SeqArgs

logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)


*******Sample text from train.csv
bahasa_text,english_text
"Perahu Pinisi yang kokoh dapat dikatakan sebagai salah satu kerajinan kayu yang terkuat di Indonesia. Dibangun untuk bertahan di angin yang berkecamuk dan lautan yang berbadai, kapal yang abadi ini tidak hanya dapat bermanuver saat mengarungi perairan yang bergelombang, layar yang terbentang dapat menangkap dorongan angin – sebagai elemen yang penting dalam memberikan perlindungan.","The majestic Phinisi Yacht is said to be the strongest wooden craft in Indonesia. Built to survive raging winds and stormy seas, this timeless vessel is not only good at maneuvering in choppy waters, its sails can be angled to catch the winds — thereby harnessing the powerful elements to propel itself to safety."

train_df = pd.read_csv("Lang Translation/train.csv", names=['input_text','target_text']).astype(str)
eval_df = pd.read_csv("Lang Translation/test.csv", names=['input_text','target_text']).astype(str)
train_df["prefix"] = ""
eval_df["prefix"] = ""

model_args = Seq2SeqArgs()
model_args.max_seq_length = 96
model_args.train_batch_size = 2
model_args.eval_batch_size = 1
model_args.num_train_epochs = 1
model_args.evaluate_during_training = True
model_args.evaluate_during_training_steps = 30000
model_args.use_multiprocessing = False
model_args.fp16 = False
model_args.save_steps = -1
model_args.save_eval_checkpoints = False
model_args.no_cache = True
model_args.reprocess_input_data = True
model_args.overwrite_output_dir = True
model_args.preprocess_inputs = False
model_args.num_return_sequences = 1
model_args.wandb_project = "Marian Bahasa-English Translation"
model = Seq2SeqModel(
    encoder_decoder_type="marian",
    encoder_decoder_name="Lang Translation/opus-mt-id-en",
    use_cuda=False,
    args=model_args)

model.train_model(train_df, eval_data=eval_df)

results = model.eval_model(eval_df)


***************************** Sample Input & Output **************************

I am begginer in using transformer, please help me with this either provide me fine-tune process of marianmt or validate my code & provide solution for creating vocabulary files


Comments:
Hi there!

Please use the [forum](https://discuss.huggingface.co) to ask such general questions, we use issues for bugs and feature requests. And the `run_translation.py` script [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch/translation) can be used to train/fine-tune Marian.

HI @patil-suraj ,

Thank you so much for quick response :).

as you suggested when i using run_translation.py script , am getting below error, can you please advice me

![image](https://user-images.githubusercontent.com/79854145/134467223-5a0a57bb-295e-45d7-96bb-9817d1bc5f89.png)



python run_translation.py \
--model_name_or_path '/Lang Translation/opus-mt-id-en' \
--do_train \
--do_eval \
--source_lang id \
--target_lang en \
--dataset_name '' \
--train_file '/Lang Translation/dataset/train.json' \
--test_file '/Lang Translation/dataset/test.json' \
--dataset_config_name id-en \
--output_dir '/outputs/marianmt/' \
--per_device_train_batch_size=4 \
--per_device_eval_batch_size=4 \
--overwrite_output_dir \
--predict_with_generate

*** Sample text from train.json (source : bahasa, target: english)
{"translation": {"id": "Perahu Pinisi yang kokoh dapat dikatakan sebagai salah satu kerajinan kayu yang terkuat di Indonesia. Dibangun untuk bertahan di angin yang berkecamuk dan lautan yang berbadai, kapal yang abadi ini tidak hanya dapat bermanuver saat mengarungi perairan yang bergelombang, layar yang terbentang dapat menangkap dorongan angin \u2013 sebagai elemen yang penting dalam memberikan perlindungan.", "en": "The majestic Phinisi Yacht is said to be the strongest wooden craft in Indonesia. Built to survive raging winds and stormy seas, this timeless vessel is not only good at maneuvering in choppy waters, its sails can be angled to catch the winds \u2014 thereby harnessing the powerful elements to propel itself to safety."}}

It would be nice if you could post the stack-trace as text :) 

Also, when you are passing your own train and test file, you don't need to pass `--dataset_name` and `--dataset_config_name`, that could be the issue.

Hi Thank you so much for your quick replies :)

as you suggested i have removed --dataset_name & --dataset_config_name parameters.. now i got below error. please help me

Traceback (most recent call last):
File "run_translation.py", line 551, in <module>
main()
File "run_translation.py", line 295, in main
raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/load.py", line 1084, in load_dataset
builder_instance = load_dataset_builder(
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/load.py", line 948, in load_dataset_builder
data_files = _resolve_data_files_locally_or_by_urls(".", data_files)
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/load.py", line 269, in _resolve_data_files_locally_or_by_urls
return {
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/load.py", line 270, in <dictcomp>
k: _resolve_data_files_locally_or_by_urls(base_path, v, allowed_extensions=allowed_extensions)
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/load.py", line 266, in _resolve_data_files_locally_or_by_urls
raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to resolve any data file that matches '/Lang Translation/dataset/train.json' at /app/aafa/lang_translation_poc

Hi Just tried this way now... for --train_file & --test_file--> given direct file name instead of path with filename. it has given some other error. For your reference i am copying total stack trace. please help

python run_translation.py \
> --model_name_or_path '/Lang Translation/opus-mt-id-en' \
> --do_train \
> --do_eval \
> --source_lang id \
> --target_lang en \
> --train_file 'train.json' \
> --test_file 'test.json' \
> --output_dir '/outputs/marianmt/' \
> --per_device_train_batch_size=4 \
> --per_device_eval_batch_size=4 \
> --overwrite_output_dir \
> --predict_with_generate
/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
return torch._C._cuda_getDeviceCount() > 0
09/23/2021 15:29:28 - WARNING - __main__ - Process rank: -1, device: cpu, n_gpu: 0distributed training: False, 16-bits training: False
09/23/2021 15:29:28 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
_n_gpu=0,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_steps=None,
evaluation_strategy=IntervalStrategy.NO,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
generation_max_length=None,
generation_num_beams=None,
gradient_accumulation_steps=1,
greater_is_better=None,
group_by_length=False,
hub_model_id=None,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=None,
ignore_data_skip=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=-1,
log_level=-1,
log_level_replica=-1,
log_on_each_node=True,
logging_dir=/outputs/marianmt/runs/Sep23_15-29-28_x01taafaapp1a.vsi.uat.dbs.com,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=3.0,
output_dir=/outputs/marianmt/,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=4,
per_device_train_batch_size=4,
predict_with_generate=True,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=None,
remove_unused_columns=True,
report_to=['tensorboard', 'wandb'],
resume_from_checkpoint=None,
run_name=/outputs/marianmt/,
save_on_each_node=False,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
sortish_sampler=False,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_legacy_prediction_loop=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
09/23/2021 15:29:38 - INFO - datasets.utils.file_utils - HEAD request to https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py timed out, retrying... [1.0]
09/23/2021 15:29:49 - WARNING - datasets.builder - Using custom data configuration default-3762545e4e6cb49f
09/23/2021 15:29:49 - INFO - datasets.builder - Generating dataset json (/home/devaafabg2/.cache/huggingface/datasets/json/default-3762545e4e6cb49f/0.0.0/d75ead8d5cfcbe67495df0f89bd262f0023257fbbbd94a730313295f3d756d50)
Downloading and preparing dataset json/default to /home/devaafabg2/.cache/huggingface/datasets/json/default-3762545e4e6cb49f/0.0.0/d75ead8d5cfcbe67495df0f89bd262f0023257fbbbd94a730313295f3d756d50...
100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 2/2 [00:00<00:00, 7025.63it/s]
09/23/2021 15:29:49 - INFO - datasets.utils.download_manager - Downloading took 0.0 min
09/23/2021 15:29:49 - INFO - datasets.utils.download_manager - Checksum Computation took 0.0 min
100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 2/2 [00:00<00:00, 513.94it/s]
09/23/2021 15:29:49 - INFO - datasets.utils.info_utils - Unable to verify checksums.
09/23/2021 15:29:49 - INFO - datasets.builder - Generating split train
Traceback (most recent call last):
File "run_translation.py", line 551, in <module>
main()
File "run_translation.py", line 295, in main
raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/load.py", line 1112, in load_dataset
builder_instance.download_and_prepare(
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/builder.py", line 636, in download_and_prepare
self._download_and_prepare(
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/builder.py", line 726, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/builder.py", line 1188, in _prepare_split
writer.write_table(table)
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/datasets/arrow_writer.py", line 426, in write_table
pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
File "pyarrow/table.pxi", line 1596, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 592, in pyarrow.lib._sanitize_arrays
File "pyarrow/array.pxi", line 329, in pyarrow.lib.asarray
File "pyarrow/table.pxi", line 277, in pyarrow.lib.ChunkedArray.cast
File "/app/aafa/lang_translation_poc/env/lib64/python3.8/site-packages/pyarrow/compute.py", line 297, in cast
return call_function("cast", [arr], options)
File "pyarrow/_compute.pyx", line 527, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 337, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<id: string, en: string> to struct using function cast_struct

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.