[s2s] wmt download script use less ram
Author: stas00
Pull request: yes

a few enhancement to https://github.com/huggingface/transformers/pull/6403

the main change:
- rewrite not to load 100GB into RAM - wmt19 is huge!

and then a few small things:
- replaced the default dataset to wmt16 as it's much much smaller than wmt19 to experiment with (also it seems that at least wmt19-ru-en is missing a test dataset, while wmt16-run-en has it)
- added lang defaults so it's easy to start experimenting with
- moved tqdm to where a detailed progress can be seen
- added some extra notes

@sshleifer 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=h1) Report
> Merging [#6405](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b9ecd92ee4fcd515f542c73593a4b6fa0b2c81fc&el=desc) will **decrease** coverage by `2.10%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6405/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6405      +/-   ##
==========================================
- Coverage   80.10%   78.00%   -2.11%     
==========================================
  Files         149      149              
  Lines       27680    27680              
==========================================
- Hits        22173    21591     -582     
- Misses       5507     6089     +582     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `28.94% <0.00%> (-67.11%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `26.98% <0.00%> (-52.81%)` | :arrow_down: |
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `33.33% <0.00%> (-24.33%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `48.79% <0.00%> (-18.08%)` | :arrow_down: |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `63.95% <0.00%> (-14.54%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `13.76% <0.00%> (-14.38%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `65.68% <0.00%> (-6.16%)` | :arrow_down: |
| [src/transformers/modelcard.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGNhcmQucHk=) | `82.71% <0.00%> (-2.47%)` | :arrow_down: |
| [src/transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kaXN0aWxiZXJ0LnB5) | `96.19% <0.00%> (-1.64%)` | :arrow_down: |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `95.20% <0.00%> (-1.37%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/huggingface/transformers/pull/6405/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=footer). Last update [b9ecd92...de72b0a](https://codecov.io/gh/huggingface/transformers/pull/6405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


further testing showed that chunking doesn't make much of a difference - writing one record at a time is almost as fast as writing in chunks of 10K records - I think it's the reading that's the bottleneck here, which we can't optimize. So I removed the chunking from this PR.

`wmt19-ru-en` with 37M records converted in ~40mins on my machine using this PR.