[run_clm] handling large inputs
Author: stas00
Pull request: no

This could probably use a bit of sorting out:

> [WARNING|tokenization_utils_base.py:3138] 2021-04-06 21:29:29,790 >> Token indices sequence length is longer than the specified maximum sequence length for this model (1462828 > 1024). Running this sequence through the model will result in indexing errors

This doesn't look right and it looks quite scary. `1462828 > 1024`

This happens when someone feeds a huge input per entry, e.g. this happened when feeding 
```
BS=1; rm -rf output_dir; PYTHONPATH=src USE_TF=0 examples/language-modeling/run_clm.py \
--model_name_or_path distilgpt2 --do_train --output_dir output_dir --num_train_epochs 1  \
--per_device_train_batch_size 1 --block_size 128 --train_file finetune-gpt2xl/train.csv
```

It comes from: https://github.com/Xirider/finetune-gpt2xl - which shows how to train gpt-neo.
The csv file's only record is a small book https://raw.githubusercontent.com/Xirider/finetune-gpt2xl/main/train.csv.

So the whole single input is 1462828 tokens.

Either `run_clm.py` should slice it up or truncate it or do something about it, so that that warning won't show up. I'm not sure what the design is supposed to be for handling huge inputs.

Here is the trace to where the warning comes from:

```
  File "examples/language-modeling/run_clm.py", line 444, in <module>
    main()
  File "examples/language-modeling/run_clm.py", line 322, in main
    tokenized_datasets = datasets.map(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/datasets/dataset_dict.py", line 431, in map
    {
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/datasets/dataset_dict.py", line 432, in <dictcomp>
    k: dataset.map(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1289, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1260, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "examples/language-modeling/run_clm.py", line 320, in tokenize_function
    return tokenizer(examples[text_column_name])
  File "/mnt/nvme1/code/huggingface/transformers-gpt-neo-nan/src/transformers/tokenization_utils_base.py", line 2254, in __call__
    return self.batch_encode_plus(
  File "/mnt/nvme1/code/huggingface/transformers-gpt-neo-nan/src/transformers/tokenization_utils_base.py", line 2439, in batch_encode_plus
    return self._batch_encode_plus(
  File "/mnt/nvme1/code/huggingface/transformers-gpt-neo-nan/src/transformers/tokenization_utils.py", line 549, in _batch_encode_plus
    batch_outputs = self._batch_prepare_for_model(
  File "/mnt/nvme1/code/huggingface/transformers-gpt-neo-nan/src/transformers/tokenization_utils.py", line 597, in _batch_prepare_for_model
    outputs = self.prepare_for_model(
  File "/mnt/nvme1/code/huggingface/transformers-gpt-neo-nan/src/transformers/tokenization_utils_base.py", line 2790, in prepare_for_model
    self._eventual_warn_about_too_long_sequence(encoded_inputs["input_ids"], max_length, verbose)
  File "/mnt/nvme1/code/huggingface/transformers-gpt-neo-nan/src/transformers/tokenization_utils_base.py", line 3135, in _eventual_warn_about_too_long_sequence
    traceback.print_stack()
```

Thank you!

@sgugger 

Comments:
We don't care since we are not feeding those inputs to the model, they are split afterwards: the preprocessing has two stages, first tokenize everything then create the blocks of size `block_size`. We can't truncate large inputs because then the special tokens would be added weirdly and there is no way I know of of deactivating that warning.

I don't see it as being bad, like the warning when we load a checkpoint in an architecture that doesn't completely fit and some weights are randomly initialized. We can add it to the README documenting the script if you find it that scary.

Thank you for explaining why this is done in this way, @sgugger 

Here are a few possible solutions:

1. Perhaps the tokenizer utils shouldn't be in charge of dispensing warnings out of context? If you propose the user to ignore a warning, it's a good idea to ask ourselves why are we issuing this warning in the first place?
 
2. Pass a new flag like `unchunked_long_input=True` which will skip the warning as now we know it's by design.

And to explain why `1462828 > 1024` looks scary because it's not "warning: 2048 > 1024` - ok, I'm feeding a slightly longer input - I know what I'm doing. `1462828` is much much larger and it makes no ready sense as it can be an arbitrary huge number which doesn't immediately correlate to anything. And I saw that warning when dealing with NaN, so I immediately had a question - could that somehow be connected? It's not but when you deal with an error, every warning could potentially be flagging the pointer to the cause of the error.

Looking more, there is a `verbose` keyword argument that we could pass to this tokenization call that will then not issue the warning.

Indeed, this:
```
diff --git a/examples/language-modeling/run_clm.py b/examples/language-modeling/run_clm.py
index a49c815e2..682a9e328 100755
--- a/examples/language-modeling/run_clm.py
+++ b/examples/language-modeling/run_clm.py
@@ -317,7 +317,9 @@ def main():
     text_column_name = "text" if "text" in column_names else column_names[0]

     def tokenize_function(examples):
-        return tokenizer(examples[text_column_name])
+        # clm input could be much much longer than block_size - so silence the warning about size
+        # mismatch - and all other warnings with it as a side-effect
+        return tokenizer(examples[text_column_name], verbose=False)

     tokenized_datasets = datasets.map(
         tokenize_function,
```

turns it off.

But it turns off all warnings and bunch of info and error logs too.

What is the purpose of `verbose`? I see it's used to control:
* logger.info
* logger.error
* logger.warning - (mostly deprecation warnings)

so if it's turned off it could mask some possible issues. It looks more like `silent` flag. Still not sure it should silence `logger.error`.


Well we will have to choose one of the two ;-) I'm not really fond of adding yet another flag for silencing some things but not others so either the warning stays or we may miss other important infos.

What we can do is add a debug flag to `run_clm` and if the flag is off, we pass `verbose=False` and skip some other things like the part where we show some random samples. If the flag is on, we leave everything as is.

You consider this silencing, whereas I feel that this should not be happening in the first place. If a given function may receive an input that is very long by design, it should be instructed that it is so and have no qualms about it. The all-or-nothing approach to software communicating back to users is a difficult one.

At the same time I trust that if you were to design this area of the software from scratch you'd have done so considering this possibility and I hear that you just don't feel that it's important enough to warrant a special flag the way things are now.

If we are on the same page, then I'd rather not turn it all off.

I think that perhaps a sort of band-aid would be to catch the warning in `run_clm.py` and if it's indeed matching that the input is too long not print it.

Maybe the warning should be rephrased, but it should still be there in my opinion since in 95% of the cases, the user will be using the tokenzier outputs directly inside the model, this is really an uncommon use case here. So it says something not necessarily warranted in those 5% but will help the rest of the time, I think it's a good trade-off.

Catching the warning in the script sounds like a good idea, to not scare a beginner.

By no means I was suggesting to disable the warning in the general case.

It's far from trivial to capture and hide it though, since it's not a warning, but logger warning. And while it's probably possible to disable the default logger handler and then re-enable it, somehow it feels unsafe to do.

How about this:

```
    def tokenize_function(examples):
        from transformers.testing_utils import CaptureLogger
        tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base")
        with CaptureLogger(tok_logger) as cl:
            output = tokenizer(examples[text_column_name])
        # clm input could be much much longer than block_size
        if "Token indices sequence length is longer than the" in cl.out:
            tok_logger.warning("^^^^^^^^^^^^^^^^ Please ignore the warning above - it's just a long input")
        return output
```

So we end up with an extra warning:

```
[WARNING|tokenization_utils_base.py:3143] 2021-04-07 21:09:22,144 >> Token indices sequence length is longer than the specified maximum sequence length for this model (1462828 > 1024). Running this sequence through the model will result in indexing errors
[WARNING|run_clm.py:326] 2021-04-07 21:13:14,300 >> ^^^^^^^^^^^^^^^^ Please ignore the warning above - it's just a long input
```

unfortunately I'm not sure how to make it look as it's coming from the same file, so it's not aligned - tried to add some `^^^^^` to make it loud and clear the helper warning is about the one above and to sort of align it.

Not great, but at least it explains what happens.

Also I'm not sure if it's safe to import from `testing_utils` in examples (any extra dependencies?)

Thoughts.


Yeah, it's not ideal to import from `testing_utils` but I don't see any import that is not part of the standard lib there so I don't think it adds any dependence.

For the rest, I like your solution, and I don't think it's a problem if the warnings don't come from the same file. It's very clear to me.