Train on custom translation dataset - machine translation
Author: SadGare
Pull request: no

After my reading the readme in pytorch translation example, it says

> And here is how you would use the translation finetuning on your own files, after adjusting the values for the arguments --train_file, --validation_file to match your setup:
> 
> python examples/pytorch/translation/run_translation.py \
>     --model_name_or_path t5-small \
>     --do_train \
>     --do_eval \
>     --source_lang en \
>     --target_lang ro \
>     --source_prefix "translate English to Romanian: " \
>     --dataset_name wmt16 \
>     --dataset_config_name ro-en \
>     --train_file path_to_jsonlines_file \
>     --validation_file path_to_jsonlines_file \
>     --output_dir /tmp/tst-translation \
>     --per_device_train_batch_size=4 \
>     --per_device_eval_batch_size=4 \
>     --overwrite_output_dir \
>     --predict_with_generate
> The task of translation supports only custom JSONLINES files, with each line being a dictionary with a key "translation" and its value another dictionary whose keys is the language pair. For example:_
> { "translation": { "en": "Others have dismissed him as a joke.", "ro": "Alții l-au numit o glumă." } }
> { "translation": { "en": "And some are holding out for an implosion.", "ro": "Iar alții așteaptă implozia." } }

I do have the jsonl file in the format above on my disk, but I just don't know how to get correct _dataset_name_ and _dataset_config_name_. And I tried to just ingnore them but it pops up

> run_translation.py --model_name_or_path t5-small --do_train --do_eval --source_lang Cls --target_lang Mdn --source_prefix "translate Classical to Modern: " --train_file C:\Users\Gare\PycharmProjects\Gare\Classical_Chinese\train.jsonl --validation_file C:\Users\Gare\PycharmProjects\Gare\Classical_Chinese\valid.jsonl --output_dir /tmp/Gare-translation --per_device_train_batch_size=4 --per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate
> 03/26/2022 20:47:20 - WARNING - __main__ - Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False
> 03/26/2022 20:47:20 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
> _n_gpu=1,
> adafactor=False,
> adam_beta1=0.9,
> adam_beta2=0.999,
> adam_epsilon=1e-08,
> bf16=False,
> bf16_full_eval=False,
> data_seed=None,
> dataloader_drop_last=False,
> dataloader_num_workers=0,
> dataloader_pin_memory=True,
> ddp_bucket_cap_mb=None,
> ddp_find_unused_parameters=None,
> debug=[],
> deepspeed=None,
> disable_tqdm=False,
> do_eval=True,
> do_predict=False,
> do_train=True,
> eval_accumulation_steps=None,
> eval_delay=0,
> eval_steps=None,
> evaluation_strategy=IntervalStrategy.NO,
> fp16=False,
> fp16_backend=auto,
> fp16_full_eval=False,
> fp16_opt_level=O1,
> generation_max_length=None,
> generation_num_beams=None,
> gradient_accumulation_steps=1,
> gradient_checkpointing=False,
> greater_is_better=None,
> group_by_length=False,
> half_precision_backend=auto,
> hub_model_id=None,
> hub_strategy=HubStrategy.EVERY_SAVE,
> hub_token=<HUB_TOKEN>,
> ignore_data_skip=False,
> label_names=None,
> label_smoothing_factor=0.0,
> learning_rate=5e-05,
> length_column_name=length,
> load_best_model_at_end=False,
> local_rank=-1,
> log_level=-1,
> log_level_replica=-1,
> log_on_each_node=True,
> logging_dir=/tmp/Gare-translation\runs\Mar26_20-47-20_GareNg,
> logging_first_step=False,
> logging_nan_inf_filter=True,
> logging_steps=500,
> logging_strategy=IntervalStrategy.STEPS,
> lr_scheduler_type=SchedulerType.LINEAR,
> max_grad_norm=1.0,
> max_steps=-1,
> metric_for_best_model=None,
> mp_parameters=,
> no_cuda=False,
> num_train_epochs=3.0,
> optim=OptimizerNames.ADAMW_HF,
> output_dir=/tmp/Gare-translation,
> overwrite_output_dir=True,
> past_index=-1,
> per_device_eval_batch_size=4,
> per_device_train_batch_size=4,
> predict_with_generate=True,
> prediction_loss_only=False,
> push_to_hub=False,
> push_to_hub_model_id=None,
> push_to_hub_organization=None,
> push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
> remove_unused_columns=True,
> report_to=['tensorboard'],
> resume_from_checkpoint=None,
> run_name=/tmp/Gare-translation,
> save_on_each_node=False,
> save_steps=500,
> save_strategy=IntervalStrategy.STEPS,
> save_total_limit=None,
> seed=42,
> sharded_ddp=[],
> skip_memory_metrics=True,
> sortish_sampler=False,
> tf32=None,
> tpu_metrics_debug=False,
> tpu_num_cores=None,
> use_legacy_prediction_loop=False,
> warmup_ratio=0.0,
> warmup_steps=0,
> weight_decay=0.0,
> xpu_backend=None,
> )
> 03/26/2022 20:47:40 - INFO - datasets.utils.file_utils - HEAD request to https://raw.githubusercontent.com/huggingface/datasets/2.0.0/datasets/jsonl/jsonl.py timed out, retrying... [1.0]
> 03/26/2022 20:48:45 - INFO - datasets.utils.file_utils - HEAD request to https://raw.githubusercontent.com/huggingface/datasets/master/datasets/jsonl/jsonl.py timed out, retrying... [1.0]
> Traceback (most recent call last):
>   File "examples/pytorch/translation/run_translation.py", line 624, in <module>
>     main()
>   File "examples/pytorch/translation/run_translation.py", line 322, in main
>     raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
>   File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\load.py", line 1671, in load_dataset
>     **config_kwargs,
>   File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\load.py", line 1492, in load_dataset_builder
>     data_files=data_files,
>   File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\load.py", line 1237, in dataset_module_factory
>     ) from None
> FileNotFoundError: Couldn't find a dataset script at C:\Users\Gare\Desktop\transformers\jsonl\jsonl.py or any data file in the same directory. Couldn't find 'jsonl' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/jsonl/jsonl.py

So what how can I help with it to just get my data trained?

Comments:
Hi @Gare-Ng , if you have the dataset in jsonl files

I don't think you need the `dataset_name and dataset_config_name.` parameters; you should use this instead to run the script

```
--train_file train.json \
--validation_file dev.json \
--test_file test.json \
```

@ToluClassics Thank for your prompt relpy, that is exactly what I believe it ought to be. But it still pops up same error 

> Traceback (most recent call last):
  File "run_translation.py", line 624, in <module>
    main()
  File "run_translation.py", line 322, in main
    raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\load.py", line 1671, in load_dataset
    **config_kwargs,
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\load.py", line 1492, in load_dataset_builder
    data_files=data_files,
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\load.py", line 1237, in dataset_module_factory
    ) from None
FileNotFoundError: Couldn't find a dataset script at C:\Users\Gare\PycharmProjects\Gare\transformers\examples\pytorch\translation\jsonl\jsonl.py or any data file in the same directory. Couldn't find 'jsonl' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/jsonl/jsonl.py

I konw that I need a config file to tell it some details about training, but I just do not know how.

I finally find a solution😎It seems that huggingface remove supprot for jsonl but in its readme it remains unchanged, misleading me for quite a long time😤
[https://github.com/huggingface/transformers/issues/10820](url)

Currently, running https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation.py with a custom JSON dataset still fails if the file extension is `jsonl`.

The script itself has a check for file extensions that incorrectly accepts `jsonl`:

https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation.py#L238

This issue could be re-opened until fixed

I am facing the same issue. Any solutions?