[example scripts] inconsistency around eval vs val
Author: stas00
Pull request: no
Labels: name: Examples

name: WIP

* `val` == validation set (split)
* `eval` == evaluation (mode)

those two are orthogonal to each other - one is a split, another is a model's run mode.

the trainer args and the scripts are inconsistent around when it's `val` and when it's `eval` in variable names and metrics. 

examples:
* `eval_dataset` but `--validation_file`
* `eval_*` metrics key for validation dataset - why the prediction is then `test_*` metric keys?
* `data_args.max_val_samples` vs `eval_dataset` in the same line

the 3 parallels:
- `train` is easy - it's both the process and the split
- `prediction` is almost never used in the scripts it's all `test` - var names and metrics and cl args
- `eval` vs `val` vs `validation` is very inconsistent. when writing tests I'm never sure whether I'm looking up `eval_*` or `val_*` key. And one could run evaluation on the test dataset.

Perhaps asking a question would help and then a consistent answer is obvious:

Are metrics reporting stats on a split or a mode? 
A. split - rename all metrics keys to be `train|val|test`
B. mode - rename all metrics keys to be `train|eval|predict`

Thank you.

@sgugger, @patil-suraj, @patrickvonplaten 

Comments:
While what you say make sense, I'm unsure it warrants a new change of argument names on all example scripts as it seems more cosmetic to me.

The `TrainingArguments` have the proper mode already (`--do_train`, `--do_eval`, `--do_predict`) so it's only the examples. We're less attached to no breaking changes there for now but we will soon need them to be production ready, so if we want to do this, it should be done by end of next week if possible.

Also cc @LysandreJik 

> we will soon need them to be production ready

This examples-are-not-production and examples-are-production back and forth depending on the context is incredibly difficult to sustain.

I'm proposing to improve clarity and consistency so that the user can have an easier understanding, and if these are honest examples then the goal is to improve that - exemplification. If you make a slide presentation and people find typos in it, you don't say, but I already showed it to a group so the typos have to remain. Examples are that slide presentation that try to do the best exemplification of the core code. At least based on much feedback I received on my PRs.

And if we want to have programs that are production quality see my rfc https://github.com/huggingface/transformers/issues/10155. I really hope this project will make a clear cut decision wrt this back and forth and stand by it. 

My fantasy is that there will be:
1. examples - change and improve those any time to make things easier - these are living and working tutorials. Nothing is fixed in stone here and which are a subject to evolve at any moment. No fixed API, but really easy to understand the code.
2. apps - production quality programs that are part of the core, with well thought out API, clean refactored code, thorough testing and tests relying on these apps to do testing.


Not really my area of expertise here, but I do agree with @stas00 -> I think we should keep the liberty of quickly adapting the examples

> Are metrics reporting stats on a split or a mode?
> A. split - rename all metrics keys to be `train|val|test`
> B. mode - rename all metrics keys to be `train|eval|predict`

So what it should be? - either A or B - the current `train|eval|test` is a very odd amalgamation of A and B. Unless we just say that the `validation` set is really an `evaluation` set and let it be.

This impacts the results json files too in example scripts.


This really is broken. I was just trying to write some code that was using the splits and again run into this some args are "*val*" others "*eval*" :( so can't even write automated attribute retrieval by split and have to write the code as:

```
def get_actual_samples(self, split):

    if not split in ["train", "eval", "test"]:
        raise ValueError(f"Unknown split {split}")

    dataset = getattr(self, f"{split}_dataset")
    split_fixed = split if split != "eval" else "val"
    max_samples_arg = getattr(self.args, f"max_{split_fixed}_samples")
    max_samples = max_samples_arg if max_samples_arg is not None else len(dataset)
    return min(max, len(dataset))
```

as you can see this is so strange.

Please, please, please - make your vote and let's make the examples use either the splits or the modes and not the mix of both:

A. split - rename all metrics keys to be `train|val|test`
B. mode - rename all metrics keys to be `train|eval|predict`

To remind currently it's:
- `train|val-eval|test`

If option B is chosen we rename all cl arg keys + metrics: "val" => "eval" and "test" => "predict"
If option A is chosen we rename all cl arg keys + metrics: "eval" => "val" 

My vote is B:  `train|eval|predict` because we are reporting on and configuring a specific Trainer mode and not the split.




I vote for B, for consistency with `do_train`, `do_eval`, `do_predict`.

For examples: switching an arg name can be done without taking precautions for BC as long as the README is updated at the same time, but for `TrainingArguments `(if any is concerned), a proper deprecation cycle has to be made.

@bhadreshpsavani, would this be something you'd like to work on by chance? If you haven't tired of examples yet.

Hi @stas00,
Ya i will be happy to work more.
Actually I was looking for some issues to work on!

Awesome! Thank you, @bhadreshpsavani!

So the changes we need are:

1. use `eval` instead of `val`
2. use `predict` instead of `test`

in cl args and variable names in example scripts (only the active ones, please ignore legacy/research subdirs).

I hope this will be a last rename in awhile.


Hi @stas00, 
In the dataset we have `validation` as a key for proper conversion shall we also need to change it to `evaluation`?
For `validation_file` we can either change it to `evalution_file` or `eval_file` or keep it as it is.

No the key in the dataset dictionary is "validation", so it should be `validation_file`.

While testing my changes I come to know that few example scripts are not working fine before my changes!
Here is the List:
```
/language-modeling/run_clm.py 
/language-modeling/run_plm.py
/question-answering/run_qa_beam_search.py
```
Please check/run this [colab](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/TestingAllHuggingfaceScripts_.ipynb) for instant testing.
When I run the above script locally in ubuntu my system got freeze.

I have no luck using colab today, it doesn't connect at all, so I can't test.

I run the clm test as you posted it on my own machine and it worked just fine.

Is it only colab that it's failing on? 

Hi @stas00,
It actually not giving any error but after a few epochs of training, it gives something like this ^C, and it's not doing further stages like eval and predict. 

Does it silently abort the run w/o any traceback? That often means the system run out of RAM and the kernel killed the process - often you get no response. colab is notorious for giving a tiny amount of RAM.

I hope colab will start working for me again and I will see if I can reproduce this.

for hanging there is this trick, add this to the beginning of the program:

```
import faulthandler
faulthandler.dump_traceback_later(20, repeat=True)
```
now every 20 secs it will print out where each thread is (traceback). super handy!


There is also the on demand version:
```
# register and then kill the process w/ stack trace
import faulthandler, signal
faulthandler.register(signal.SIGUSR1)
# kill the stuck process
kill -USR1 PID
```
but it often doesn't work.

`py-spy` is another handy one but it requires `sudo` so won't work on colab. Unless you start the program with it:
```
# trace a running python application - e.g. when it's hanging or very slow and you want to see the backtrace - one way is using a sighandler - but that requires killing it and already having it installed
pip install py-spy
sudo py-spy top --pid PID
# if one has no sudo, start the program via
py-spy -- python myprogram.py
# and then it will attached without sudo
# https://github.com/benfred/py-spy#when-do-you-need-to-run-as-sudo
```


Ya, I think it silently aborted the run w/o any traceback. Might be because it is occupying the entire ram somehow.
Similar behavior I observed when I run a really big docker image locally.

I will definitely try this command and dig more!

Thanks a lot for your input. This is really insightful! I will note down this as well :)

Yes, this is almost always the case in colab. It's too bad they don't have a simple widget that shows real time memory usage.

For your personal machine, always have a huge swap file if you do ML dev. Like 100GB. It will save the day.

Also there is a way to protect your desktop from your RAM-hungry commands - you can look into `cgroups`. If you machine crashes a lot because of run away training or jupyter, this is another super useful addition. But there are other way to manage resources.

ok, so I had to disable a `Privacy Badger` firefox extension and colab started working.

First, make a habit to start colab with:
```
!free -h
```
sometimes I get 12GB RAM, other times 25GB, 12GB is typically too low for much.

So `run_clm` works just fine  even on 12GB. I had to use a small bs so edited your cmd lines to limit bs:

```
!python examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 \
--dataset_name wikitext \
--max_train_samples 5 \
--max_val_samples 5 \
--dataset_config_name wikitext-2-raw-v1 \
--do_train \
--do_eval \
--output_dir /tmp/test-clm \
--per_device_eval_batch_size 2 \
--per_device_train_batch_size 2 \
--overwrite_output_dir
```
this worked too:
```
!python examples/pytorch/language-modeling/run_plm.py \
--model_name_or_path xlnet-base-cased \
--dataset_name wikitext \
--max_train_samples 5 \
--max_val_samples 5 \
--dataset_config_name wikitext-2-raw-v1 \
--do_train \
--do_eval \
--output_dir /tmp/test-clm \
--per_device_eval_batch_size 2 \
--per_device_train_batch_size 2 \
--overwrite_output_dir
```
and so did:
```
!python examples/pytorch/question-answering/run_qa.py \
--model_name_or_path distilbert-base-uncased \
--train_file tests/fixtures/tests_samples/SQUAD/sample.json \
--validation_file tests/fixtures/tests_samples/SQUAD/sample.json \
--test_file tests/fixtures/tests_samples/SQUAD/sample.json \
--do_train \
--do_eval \
--do_predict \
--max_train_samples 5 \
--max_val_samples 5 \
--max_test_samples 5 \
--learning_rate 3e-5 \
--max_seq_length 384 \
--doc_stride 128 \
--version_2_with_negative \
--output_dir /tmp/debug_squad/ \
--per_device_eval_batch_size 2 \
--per_device_train_batch_size 2 \
--overwrite_output
```