The multi-node / multi-gpu training and repeat logging on each process
Author: stas00
Pull request: no
Labels: name: WIP

How do we deal with repetitive warnings that can't be shut off on a multi-node/multi-gpu environment?

e.g. at BigScience we started using HF Tokenizer and now **this gets repeated hundreds of times**:
```
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
```

It comes from:
https://github.com/huggingface/transformers/blob/efea0f868bd381244e3cef51b388293e41a36d1e/src/transformers/tokenization_utils_base.py#L1934-L1936

The only way for me to fix this is to push the logging level to ERROR on the replicas:
```
        if args.rank == 0:
            transformers.utils.logging.set_verbosity(logging.INFO)
        else:
            transformers.utils.logging.set_verbosity(logging.ERROR)
```
but then if there is actually a real warning in some process, then I won't see it.

Any good suggestions here?

Thank you!

p.s. As a background story: I have each component we use in Megatron-DeepSpeed spitting just a few dozens of these which then get multiplied by say 512 or 1024 times. And the log file becomes unusable and makes the troubleshooting when things crash a very difficult experience. Hence I really need to find a way not to log anything that is not really pertinent to a specific replica process. Moreover many processes aren't replicas of rank 0 process and do unique things, e.g. in the pipeline setup. But in the case of tokenizer it is the same on all processes.

@sgugger, @LysandreJik 

Comments:
I don't know of any other way to limit the logs.

We could switch it to a `warnings.warn`, which should only be sent once. 

Otherwise, we can also use the `self.deprecation_warnings` dictionary available here:

https://github.com/huggingface/transformers/blob/efea0f868bd381244e3cef51b388293e41a36d1e/src/transformers/tokenization_utils_base.py#L1470-L1472

This can store warnings, like done here:

https://github.com/huggingface/transformers/blob/efea0f868bd381244e3cef51b388293e41a36d1e/src/transformers/tokenization_utils_base.py#L1494-L1498

To make sure that the warnings are not sent more than once.

This unfortunately won't help. We are talking multi-process here, so the processes don't know anything about each other's doings. 

OK, I have an idea. Many of our warnings are there to help new users know that something might not be right, or they may need to do something. This is great! 

But when a user runs the same command line then Nth time, they don't want to see that warning anymore, because if it were important to them they would have done something about it already and the warning should have disappeared.

So I propose a new feature that turns all the advisory warnings off. Probably via env var.
```
HF_ADVISORY_WARNINGS=0 my_program.py
```

This is a feature for users who know what they are doing and actually pay attention to the logs.

Thoughts?


I am fine with that on principle, we just need to have an API designed that makes it easy to use (e.g. not adding three lines of codes to issue a warning).

Yes, of course! It should be a single wrapper, like:

```
- logger.warning("Special tokens have been added in the vocabulary...) 
+ logger.warning_advice("Special tokens have been added in the vocabulary...) 
```

and:

```
# logging.py

advisory_warnings = code to check ENV[HF_ADVISORY_WARNINGS]

def warning_advice(*args, **kwargs):
    if not advisory_warnings: return
    logger.warning(*args, **kwargs)
```

Something like that?

This looks like a great solution to me. Wdyt @LysandreJik ?

Sounds great to me!