switching some low-level log.info's to log.debug?
State: closed | Author: stas00 (CONTRIBUTOR)
Labels: name: enhancement

In https://github.com/huggingface/transformers/pull/12276 we are now changing the examples to have `datasets` on the same log level as `transformers`, so that one setting can do a consistent logging across all involved components.

The trouble is that now we get a ton of these:

```
06/23/2021 12:15:31 - INFO - datasets.utils.filelock -   Lock 139627640431136 acquired on /home/stas/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow.lock
06/23/2021 12:15:31 - INFO - datasets.arrow_writer -   Done writing 50 examples in 12280 bytes /home/stas/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow.
06/23/2021 12:15:31 - INFO - datasets.arrow_dataset -   Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
06/23/2021 12:15:31 - INFO - datasets.utils.filelock -   Lock 139627640431136 released on /home/stas/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow.lock
```

May I suggest that these can be `log.debug` as it's no informative to the user.

More examples: these are not informative - too much information:
```
06/23/2021 12:14:26 - INFO - datasets.load -   Checking /home/stas/.cache/huggingface/datasets/downloads/459933f1fe47711fad2f6ff8110014ff189120b45ad159ef5b8e90ea43a174fa.e23e7d1259a8c6274a82a42a8936dd1b87225302c6dc9b7261beb3bc2daac640.py for additional imports.
06/23/2021 12:14:27 - INFO - datasets.builder -   Constructing Dataset for split train, validation, test, from /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e814712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a

```

While these are:
```
06/23/2021 12:14:27 - INFO - datasets.info -   Loading Dataset Infos from /home/stas/.cache/huggingface/modules/datasets_modules/datasets/wmt16/0d9fb3e814712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a
06/23/2021 12:14:27 - WARNING - datasets.builder -   Reusing dataset wmt16 (/home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e814712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a)
```

I also realize that `transformers` examples don't have do use `info` for `datasets` to let the default `warning` keep logging to less noisy.

But I think currently the log levels are slightly misused and skewed by 1 level. Many `warnings` will better be `info`s and most `info`s be `debug`.

e.g.:

```
06/23/2021 12:14:27 - WARNING - datasets.builder -   Reusing dataset wmt16 (/home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e814712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a)
```

why is this a warning? it is informing me that the cache is used, there is nothing to be worried about. I'd have it as `info`.

Warnings are typically something that's bordering error or the first thing to check when things don't work as expected.

infrequent info is there to inform of the different stages or important events.

Everything else is debug.

At least the way I understand things. 


Comments: Hi @stas00, thanks for pointing out this issue with logging.

I agree that `datasets` can sometimes be too verbose... I can create a PR and we could discuss there the choice of the log levels for different parts of the code.