[s2s examples] dataset porting
Author: stas00
Pull request: no

We need to port these to jsonlines and ideally make them part of dataset hub:
https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
https://cdn-datasets.huggingface.co/translation/wmt_en_de.tgz
https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz

The problem is that nobody knows how these came to be - they are pre-processed but it's unclear how.

@sshleifer, do you by chance remember how these were created? If we put those on the new datasets hub - it'd be good to explain how these are different from normal wmt datasets. 

Also do you remember which wmtXX they came from?

Thank you!

----

The resolution is here: https://github.com/huggingface/transformers/issues/10044#issuecomment-779555741

Comments:
Here is a script to convert these:

```
import io
import json
import re

src_lang, tgt_lang = ["en", "ro"]

for split in ["train", "val", "test"]:
    recs = []
    fout = f"{split}.json"
    with io.open(fout, "w", encoding="utf-8") as f:
        for type in ["source", "target"]:
            fin = f"{split}.{type}"
            recs.append([line.strip() for line in open(fin)])
        for src, tgt in zip(*recs):
            out = {"translation": { src_lang: src, tgt_lang: tgt } }
            x = json.dumps(out, indent=0, ensure_ascii=False)
            x = re.sub(r'\n', ' ', x, 0, re.M)
            f.write(x + "\n")
```

The short answer is I don't recall, but my best guess is:

[en-de](https://github.com/pytorch/fairseq/blob/master/examples/translation/prepare-wmt14en2de.sh#L1)

[en-ro](https://github.com/rsennrich/wmt16-scripts/blob/master/sample/download_files.sh)

[cnn_dm](https://github.com/abisee/cnn-dailymail#option-1-download-the-processed-data)

That's perfect, @sshleifer! Thanks a lot!

I missed one more entry: https://cdn-datasets.huggingface.co/summarization/xsum.tar.gz

I found the main source at https://github.com/EdinburghNLP/XSum/tree/master/XSum-Dataset but not sure about what pre-processing if any went in.

Thank you!

Preprocessing: I don't actually know. The author sent me a link (in a github issue I can't find) which leads me to predict with high confidence that the preprocessing is the same as whatever is in the repo.

There is a larger *scraping wherever they are scraping is not deterministic* issue discussed in this thread https://github.com/huggingface/datasets/issues/672 (quentin and I run same code get different numbers)
which I was far too lazy to do anything more than complain about :)


OK, for xsum this is as good as it gets, and so I will document what you shared. Thank you!

I was able to to reproduce 100% the datasets you created with the instructions you provided:
- [en-de](https://github.com/pytorch/fairseq/blob/master/examples/translation/prepare-wmt14en2de.sh#L1)
- [en-ro](https://github.com/rsennrich/wmt16-scripts/blob/master/sample/download_files.sh)

This one doesn't match the instructions on the page you linked to:
- [cnn_dm](https://github.com/abisee/cnn-dailymail#option-1-download-the-processed-data)
the content is the same but the format is quite different after all the pre-processing steps were applied. Their results are all lower-cased and tagged with `

`, and it's word-level tokenized. Yours is just clean normal text. So you must have used a different process.

If I dig into it, it looks like your source is for sure just the original with new lines removed - in fact this is what it says in the README.md:
```
wget https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz
tar -xzvf cnn_dm_v2.tgz  # empty lines removed
```
target looks like a combination of `@highlight` entries into the abstract. So little pre-processing here, but I think I sorted it out.

I'm trying to preserve these datasets you created since they are awesome for monitoring any regressions in the code, because they avail themselves to a high bleu scores on even a small sample, so I constantly use them as a reference and I can quickly detect any problems if the score drops. Thank you for creating those in first place, @sshleifer!

With all datasets or pretty much any data I make, I include the scripts that created it (or a link to where it can be found), so that it's easy to know how things came to be. Mainly doing it for myself, since I am not good at remembering such details. And I don't like researching the same thing more than once.



To substitute the old datasets directories with new ones, replace:

* [x] `--data_dir wmt_en_de` => `--dataset_name wmt14 --dataset_config "de-en"` or if you want the highest score use: `--dataset_name stas/wmt14-en-de-pre-processed`
* [x] `--data_dir wmt_en_ro` => `--dataset_name wmt16 --dataset_config "ro-en"` 
* [x] `--data_dir cnn_dm` => `--dataset_name cnn_dailymail --dataset_config "3.0.0"`
* [x] `--data_dir xsum` => `--dataset_name xsum`

conversion to `datasets` status:

* [x] `stas/wmt14-en-de-pre-processed` https://huggingface.co/datasets/stas/wmt14-en-de-pre-processed (this dataset version scores about twice as good as the unprocessed one) 
* [x] `stas/wmt16-en-ro-pre-processed` https://huggingface.co/datasets/stas/wmt16-en-ro-pre-processed (this dataset version scores identical to the unprocessed one) 

I didn't bother porting the following 2, since their score is just slightly better than the unprocessed versions. So just use the following:
* [x] `--data_dir cnn_dm`  is just slightly better than `--dataset_name cnn_dailymail --dataset_config "3.0.0"`
* [x] `--data_dir xsum` is just slightly better than  `--dataset_name xsum`

Here are the full benchmarks where I verified that all but wmt14 are OK with unprocessed dataset versions:

```


###  wmt16-en-ro-pre-processed

export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir \
--adam_eps 1e-06 --do_eval --do_train --evaluation_strategy=steps  \
--label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 \
--max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS \
--per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler \
--task translation_en_to_ro  --val_max_target_length 128 --warmup_steps 500 \
--max_train_samples 2000 --max_val_samples 500 \
--dataset_name stas/wmt16-en-ro-pre-processed --source_prefix "translate English to Romanian: "

02/16/2021 00:01:55 - INFO - __main__ -     val_bleu = 24.1319


vs normal wmt16-en-ro dataset:


export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --do_eval --do_train --evaluation_strategy=steps  --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro  --val_max_target_length 128 --warmup_steps 500 --max_train_samples 2000 --max_val_samples 500 --dataset_name wmt16 --dataset_config "ro-en" --source_prefix "translate English to Romanian: "

02/15/2021 23:59:56 - INFO - __main__ -     val_bleu = 24.1319


results: the preprocessed scores identically as the non-preprocessed one



### wmt14-en-de-pre-processed

export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir \
--adam_eps 1e-06 --do_eval --do_train --evaluation_strategy=steps  \
--label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 \
--max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS \
--per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler \
--task translation_en_to_de  --val_max_target_length 128 --warmup_steps 500 \
--max_train_samples 2000 --max_val_samples 500 \
--dataset_name stas/wmt14-en-de-pre-processed --source_prefix "translate English to English: "


02/19/2021 11:53:46 - INFO - __main__ - eval_bleu = 22.2348

vs normal wmt14-en-de dataset:

export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --do_eval --do_train --evaluation_strategy=steps  --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_de  --val_max_target_length 128 --warmup_steps 500 --max_train_samples 2000 --max_val_samples 500 --dataset_name wmt14 --dataset_config "de-en"

02/19/2021 11:55:37 - INFO - __main__ - eval_bleu = 10.5513

results: the preprocessed one scores significantly better


# cnn_dailymail

wget https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz
tar -xzvf cnn_dm_v2.tgz  # empty lines removed
mv cnn_cln cnn_dm
export BS=16 MODEL=t5-small; rm -r output_dir; PYTHONPATH=src USE_TF=0  CUDA_VISIBLE_DEVICES=0 examples/legacy/seq2seq/finetune_trainer.py --model_name_or_path $MODEL --output_dir output_dir --adam_eps 1e-06 --data_dir cnn_dm --do_eval --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS  --eval_steps 25000  --sortish_sampler  --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 50 --n_train 2000 --n_val 500 --predict_with_generate --task  summarization


2021-02-19 15:16:41 | INFO | __main__ | ***** val metrics *****
2021-02-19 15:16:41 | INFO | __main__ |   val_gen_len = 54.3
2021-02-19 15:16:41 | INFO | __main__ |   val_loss = 3432.908
2021-02-19 15:16:41 | INFO | __main__ |   val_n_objs = 500
2021-02-19 15:16:41 | INFO | __main__ |   val_rouge1 = 30.2151
2021-02-19 15:16:41 | INFO | __main__ |   val_rouge2 = 11.1576
2021-02-19 15:16:41 | INFO | __main__ |   val_rougeL = 21.545
2021-02-19 15:16:41 | INFO | __main__ |   val_rougeLsum = 27.1914
2021-02-19 15:16:41 | INFO | __main__ |   val_runtime = 70.1847
2021-02-19 15:16:41 | INFO | __main__ |   val_samples_per_second = 7.124


vs normal cnn_dailymail 3.0.0 dataset

export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir \
--adam_eps 1e-06 --do_eval --do_train --evaluation_strategy=steps  \
--label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 \
--max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS \
--per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler \
--task  summarization --val_max_target_length 128 --warmup_steps 500 \
--max_train_samples 2000 --max_val_samples 500 \
--dataset_name cnn_dailymail --dataset_config "3.0.0"


02/19/2021 15:02:13 - INFO - __main__ -   ***** val metrics *****
02/19/2021 15:02:13 - INFO - __main__ -     eval_gen_len              =  74.902
02/19/2021 15:02:13 - INFO - __main__ -     eval_loss                 =  4.7365
02/19/2021 15:02:13 - INFO - __main__ -     eval_rouge1               = 28.3215
02/19/2021 15:02:13 - INFO - __main__ -     eval_rouge2               =  9.8609
02/19/2021 15:02:13 - INFO - __main__ -     eval_rougeL               = 20.1687
02/19/2021 15:02:13 - INFO - __main__ -     eval_rougeLsum            = 25.0959
02/19/2021 15:02:13 - INFO - __main__ -     eval_runtime              = 37.8969
02/19/2021 15:02:13 - INFO - __main__ -     eval_samples              =     500
02/19/2021 15:02:13 - INFO - __main__ -     eval_samples_per_second   =  13.194


results: the preprocessed one scores slightly better



# xsum

wget https://cdn-datasets.huggingface.co/summarization/xsum.tar.gz
tar -xzvf xsum.tar.gz

export BS=16 MODEL=t5-small; rm -r output_dir; PYTHONPATH=src USE_TF=0  CUDA_VISIBLE_DEVICES=0 examples/legacy/seq2seq/finetune_trainer.py --model_name_or_path $MODEL --output_dir output_dir --adam_eps 1e-06 --data_dir xsum --do_eval --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS  --eval_steps 25000  --sortish_sampler  --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 50 --n_train 2000 --n_val 500 --predict_with_generate --task  summarization

2021-02-19 15:25:32 | INFO | __main__ |   val_gen_len = 42.7
2021-02-19 15:25:32 | INFO | __main__ |   val_loss = 2272.3525
2021-02-19 15:25:32 | INFO | __main__ |   val_n_objs = 500
2021-02-19 15:25:32 | INFO | __main__ |   val_rouge1 = 20.6343
2021-02-19 15:25:32 | INFO | __main__ |   val_rouge2 = 2.8416
2021-02-19 15:25:32 | INFO | __main__ |   val_rougeL = 14.3483
2021-02-19 15:25:32 | INFO | __main__ |   val_rougeLsum = 14.8529
2021-02-19 15:25:32 | INFO | __main__ |   val_runtime = 51.8796
2021-02-19 15:25:32 | INFO | __main__ |   val_samples_per_second = 9.638

vs normal cnn_dailymail 3.0.0 dataset

export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir \
--adam_eps 1e-06 --do_eval --do_train --evaluation_strategy=steps  \
--label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 \
--max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS \
--per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler \
--task  summarization --val_max_target_length 128 --warmup_steps 500 \
--max_train_samples 2000 --max_val_samples 500 \
--dataset_name xsum

02/19/2021 15:23:38 - INFO - __main__ -     epoch                     =     1.0
02/19/2021 15:23:38 - INFO - __main__ -     eval_gen_len              =  56.858
02/19/2021 15:23:38 - INFO - __main__ -     eval_loss                 =  5.2487
02/19/2021 15:23:38 - INFO - __main__ -     eval_rouge1               = 18.0063
02/19/2021 15:23:38 - INFO - __main__ -     eval_rouge2               =   2.276
02/19/2021 15:23:38 - INFO - __main__ -     eval_rougeL               = 12.8842
02/19/2021 15:23:38 - INFO - __main__ -     eval_rougeLsum            = 13.9633
02/19/2021 15:23:38 - INFO - __main__ -     eval_runtime              = 31.2343
02/19/2021 15:23:38 - INFO - __main__ -     eval_samples              =     500
02/19/2021 15:23:38 - INFO - __main__ -     eval_samples_per_second   =  16.008

results: the preprocessed one scores slightly better
```