[s2s run_eval] new features
Author: stas00
Pull request: yes

this PR adds several features to `run_eval.py` and adds a new script `run_eval_search.py` 

* adding 2 new args:
```
  --dump-args           print the custom hparams with the results
  --info [INFO]         use in conjunction w/ --dump-args to print with the results whatever other info you'd like, 
                        e.g. lang=en-ru. If no value is passed, the current datetime string will be used.
```
* changed `parse_numeric_n_bool_cl_kwargs` to support bool args, so now we can pass ` --early_stopping true`

* added a new wrapper script `run_eval_search.py` that performs parametric search using  `run_eval.py`.

Here is the new section in `README.md` that explains all the additions:


#### run_eval tips and tricks


When using `run_eval.py`, the following features can be useful:

* if you running the script multiple times and want to make it easier to track what arguments produced that output, use `--dump-args`. Along with the results it will also dump any custom params that were passed to the script. For example if you used: `--num_beams 8 --early_stopping true`, the output will be:
   ```
   {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True}
   ```

   `--info` is an additional argument available for the same purpose of tracking the conditions of the experiment. It's useful to pass things that weren't in the argument list, e.g. a language pair `--info "lang:en-ru"`. But also if you pass `--info` without a value it will fallback to the current date/time string, e.g. `2020-09-13 18:44:43`.

   If using `--dump-args --info`, the output will be:
   
   ```
   {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True, 'info': '2020-09-13 18:44:43'}
   ```

   If using `--dump-args --info "pair:en-ru chkpt=best`, the output will be:
   
   ```
   {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True, 'info': 'pair=en-ru chkpt=best'}
   ```
      


* if you need to perform a parametric search in order to find the best ones that lead to the highest BLEU score, let `run_eval_search.py` to do the searching for you.

   The script accepts the exact same arguments as `run_eval.py`, plus an additional argument `--search`. The value of `--search` is parsed, reformatted and fed to ``run_eval.py`` as additional args.

   The format for the ``--search`` value is a simple string with hparams and the values to try, e.g.:
   ```
    --search "num_beams=5:10 length_penalty=0.8:1.0:1.2 early_stopping=true:false"
   ```
   which will generate `12` `(2*3*2)` searches for a product of each hparam. For example the example that was just used will invoke `run_eval.py` repeatedly with:
   
   ```
    --num_beams 5 --length_penalty 0.8 --early_stopping true
    --num_beams 5 --length_penalty 0.8 --early_stopping false
    [...]
    --num_beams 10 --length_penalty 1.2 --early_stopping false
   ```
   
   On completion this function prints a markdown table of the results sorted by the best BLEU score and the winning arguments.
```
bleu  | num_beams | length_penalty | early_stopping
----- | --------- | -------------- | --------------
26.71 |         5 |            1.1 |              1
26.66 |         5 |            0.9 |              1
26.66 |         5 |            0.9 |              0
26.41 |         5 |            1.1 |              0
21.94 |         1 |            0.9 |              1
21.94 |         1 |            0.9 |              0
21.94 |         1 |            1.1 |              1
21.94 |         1 |            1.1 |              0

Best score args:
stas/wmt19-en-ru data/en-ru/val.source data/en-ru/test_translations.txt --reference_path data/en-ru/val.target --score_path data/en-ru/test_bleu.json --bs 8 --task translation --num_beams 5 --length_penalty 1.1 --early_stopping True
```
@sshleifer 

Comments:
After signature discussion is complete, lets add some test coverage.
Can copy/modify [`test_run_eval`](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_seq2seq_examples.py#L287)

>     * Important that we can `utils.json_load` the dumped files.

Do you see a reason why it should fail to do so? 

but the point is moot if we merge into one dict as discussed above.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=h1) Report
> Merging [#7109](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/90cde2e938638e64a8696a12b79ee5f52364b162?el=desc) will **increase** coverage by `1.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/7109/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #7109      +/-   ##
==========================================
+ Coverage   79.62%   80.63%   +1.00%     
==========================================
  Files         168      168              
  Lines       32284    32284              
==========================================
+ Hits        25706    26031     +325     
+ Misses       6578     6253     -325     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `25.13% <0.00%> (-73.83%)` | :arrow_down: |
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `66.66% <0.00%> (-23.43%)` | :arrow_down: |
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `20.53% <0.00%> (-21.21%)` | :arrow_down: |
| [src/transformers/modeling\_transfo\_xl\_utilities.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsX3V0aWxpdGllcy5weQ==) | `52.98% <0.00%> (-13.44%)` | :arrow_down: |
| [src/transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsLnB5) | `67.10% <0.00%> (-12.67%)` | :arrow_down: |
| [src/transformers/modeling\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19tb2JpbGViZXJ0LnB5) | `79.21% <0.00%> (-10.25%)` | :arrow_down: |
| [src/transformers/tokenization\_dpr.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZHByLnB5) | `53.15% <0.00%> (-4.51%)` | :arrow_down: |
| [src/transformers/configuration\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2JhcnQucHk=) | `90.00% <0.00%> (-4.00%)` | :arrow_down: |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `90.52% <0.00%> (-3.52%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.46% <0.00%> (-1.26%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/7109/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=footer). Last update [90cde2e...875eac8](https://codecov.io/gh/huggingface/transformers/pull/7109?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


(after we sort out these few points you pointed out)

I have discovered more work/functionality is needed for the new script - need to be able to recover from CUDA OOM, want to tap into `--info`, etc. So probably, let's merge this first round, and I will continue working on improving it in the future PRs. The changes of this PR for run_eval are all good to go and can be used right away. unless you want to merge all the return values from `run_generate` into one dict, instead of returning a tuple of 2 separate ones. Let me know and I will attend to that first then.

And, yes, all the tests will be added too - it's on my short todo list.

Yes one dict please!

OK, test added.

`test_run_eval_search` is about 3 times slower than `test_run_eval` (it runs eval 4 times). Still OK to leave it as a normal test (no slow)?

`test_run_eval`:
```
Results (15.45s):
       3 passed

real    0m16.193s
user    0m13.079s
sys     0m1.010s
```
`test_run_eval_search`
```
Results (51.18s):
       3 passed

real    0m51.962s
user    0m42.695s
sys     0m1.688s
```

and currently supporting:
```
task_score_names = {
    "translation": ["bleu"],
    "translation_en_to_de": ["bleu"],
    "summarization": ["rouge1", "rouge2", "rougeL"],
}
```


How do I go about creating a `TINY_FSMT` model, @sshleifer?

this looks like a hack:
```
try:
    from .utils import calculate_bleu, calculate_rouge, parse_numeric_n_bool_cl_kwargs, use_task_specific_params
except ImportError:
    from utils import calculate_bleu, calculate_rouge, parse_numeric_n_bool_cl_kwargs, use_task_specific_params
```
would it be better to use `dirname(__file__)` and insert that into `sys.path` so it works no matter where it gets invoked from?

Actually adding `examples/seq2seq/conftest.py` that will insert a resolved `examples/seq2seq` into `sys.path` is probably a better fix as it'll solve this issue for all tests under that folder. (see `examples/conftest.py`). and then copy that into each of the examples folders that will tweak their corresponding path. (`pytest` always runs any `conftest.py` it finds first)

If you're in agreement I will make another PR to fix those.



Please mark your test `@slow` mine should probably be `@slow` too. Ideally we have one fast one.


`conftest` great idea! If you change imports, its important to verify that scripts actually run afterwards (you can get the unittests passing without try/except).

Tiny model code (roughly):
```python
tok = FSMTTokenizer.from_pretrained('facebook/wmt19-en-de')
config = FMSTConfig(decoder_layers=1, encoder_layers=1, vocab_size=vocab_size, d_model=4, encoder_ffn_dim=4, decoder_ffn_dim=4, encoder_attention_heads=1, decoder_attention_heads=1)
tiny_model = FSMTModel(config)
print(tiny_model.num_parameters())
# Test it
src_text, tgt_text = ['I am a small frog.'], ['Ich bin ein kleiner frosch.']
batch = tok.prepare_seq2seq_batch(src_text, tgt_texts=tgt_text)
outputs = tiny_model(**batch, return_dict=True)
print(outputs.loss)
tiny_model.save_pretrained('tiny_fsmt_en_de')
# transformers-cli upload tiny_fsmt_en_de stas
```

> Please mark your test `@slow` mine should probably be `@slow` too. Ideally we have one fast one.

Probably having just one of of the models is good enough for a fast test, since it checks that the script works. And then do all the others slow?

> `conftest` great idea! If you change imports, its important to verify that scripts actually run afterwards (you can get the unittests passing without try/except).

Will do.


👍 

> Tiny model code (roughly):

Thank you!

don't we want to:
1. include the tokenizer files in the s3 model? (vocabs and merges?)
2. would it make sense to create a tiny vocab and use that for tokenizer? Otherwise with these small settings I still get 336552 params. 

1) yes
2) tiny vocab would definitely be better, but is not required! tiny_mbart vocab size is around 300K also.

OK, here is the full script:
```
from transformers import FSMTTokenizer, FSMTConfig, FSMTForConditionalGeneration
mname = "facebook/wmt19-en-de"
tokenizer = FSMTTokenizer.from_pretrained(mname)
# get the correct vocab sizes, etc. from the master model
config = FSMTConfig.from_pretrained(mname)
config.update(dict(
    d_model=4,
    encoder_layers=1, decoder_layers=1,
    encoder_ffn_dim=4, decoder_ffn_dim=4,
    encoder_attention_heads=1, decoder_attention_heads=1))
tiny_model = FSMTForConditionalGeneration(config)
print(f"num of params {tiny_model.num_parameters()}")
# Test it
batch = tokenizer.prepare_seq2seq_batch(["Making tiny model"])
outputs = tiny_model(**batch, return_dict=True)
# Save
tiny_model.save_pretrained('tiny-wmt19-en-de')
tokenizer.save_pretrained('tiny-wmt19-en-de')
```

I end up with 3.1MB of files:

```
l tiny-wmt19-en-de/
total 3.1M
-rw-rw-r-- 1 stas stas 2.0K Sep 17 11:30 config.json
-rw-rw-r-- 1 stas stas 308K Sep 17 11:30 merges.txt
-rw-rw-r-- 1 stas stas 1.4M Sep 17 11:30 pytorch_model.bin
-rw-rw-r-- 1 stas stas   85 Sep 17 11:30 special_tokens_map.json
-rw-rw-r-- 1 stas stas  111 Sep 17 11:30 tokenizer_config.json
-rw-rw-r-- 1 stas stas 747K Sep 17 11:30 vocab-src.json
-rw-rw-r-- 1 stas stas 747K Sep 17 11:30 vocab-tgt.json
```

Is this reasonable for non-slow tests? 
> tiny vocab would definitely be better, but is not required! tiny_mbart vocab size is around 300K also.

Or should I make a custom small vocab? It'd shave off about half the size. Probably letter-long bpe codes, so that it could tokenize any input still.

Here currently we have 1.5M in dict files.

I would try it and give up if it's hard.
I've given up on trying to shrink sentencepiece vocabs.