[run_clm] tokenize_function clarification makes it non-hashable => no-reusing cache
Author: VictorSanh
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master at commit acc851e1ff92835d2a3ee9774d9d0abfda6e3f36 (from yesterday)
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@stas00 since you opened the PR #11145 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)


## To reproduce

I am running the minimal command:
```bash
CUDA_VISIBLE_DEVICES=0 python examples/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name ./data/bk --block_size 1024 \
    --do_train \
    --output_dir debug --overwrite_output_dir \
    --preprocessing_num_workers 5
```

When it gets to line [331](https://github.com/huggingface/transformers/blob/60607465708814fe22aaa18b26a3aab3df110c1c/examples/language-modeling/run_clm.py#L331), datasets.map gives this warning:

> [WARNING|tokenization_utils_base.py:3143] 2021-04-09 15:48:53,408 >> Token indices sequence length is longer than the specified maximum sequence length for this model (191443 > 1024). Running this sequence through the model will result in indexing errors
> [WARNING|run_clm.py:333] 2021-04-09 15:48:53,408 >> ^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model.
> 04/09/2021 15:48:53 - WARNING - 17900 - datasets.fingerprint - Parameter 'function'=<function tokenize_function at 0x7f747662c268> of the transform datasets.arrow_dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

Basically, something went wrong when trying to hash the `tokenize_function` (to produce the cache file name) => it doesn't use the pre-processed cache for the next launch.

The `tokenize_function` was originally
```python
    def tokenize_function(examples):
        output = tokenizer(examples[text_column_name])
        return output
```
and became:
```python
    def tokenize_function(examples):
        tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base")
        with CaptureLogger(tok_logger) as cl:
            output = tokenizer(examples[text_column_name])
        # clm input could be much much longer than block_size
        if "Token indices sequence length is longer than the" in cl.out:
            tok_logger.warning(
                "^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model."
            )
        return output
```


Comments:
Thank you for the report, @VictorSanh!

I can reproduce the problem separately:

```
    import transformers
    from transformers import AutoTokenizer
    from transformers.testing_utils import CaptureLogger
    tokenizer = AutoTokenizer.from_pretrained("t5-small")
    def tokenize_function(examples):
        tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base")
        with CaptureLogger(tok_logger) as cl:
            output = tokenizer(examples[text_column_name])
        # clm input could be much much longer than block_size
        if "Token indices sequence length is longer than the" in cl.out:
            tok_logger.warning(
                "^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model."
            )
        return output
    def tokenize_function2(examples):
        return tokenizer(examples[text_column_name])
```

This works (original function)
```
from datasets.fingerprint import Hasher
hasher = Hasher()
hasher.update(tokenize_function2)
```

This crashes:
```
from datasets.fingerprint import Hasher
hasher = Hasher()
hasher.update(tokenize_function)
```

```
TypeError: cannot pickle '_LazyModule' object
```

I thought I made a mistake on my side, but I saw this problem yesterday in a totally different situation:
https://github.com/huggingface/datasets/issues/2194

Let me investigate some more and will get back to you.

Until then to enable your work please just put back:
```
    def tokenize_function(examples):
        return tokenizer(examples[text_column_name])
```

This should fix the problem: https://github.com/huggingface/transformers/pull/11168



you rock!

> This should fix the problem: #11168

I modified my code according to your way, but still didn't solve the problem.
I run the official example scripts run_clm.py with multiprocessing



```
Traceback (most recent call last):
  File "/usr/local/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/media/cfs/gonglixing/9Nctl/gpt_v2/run_clm_v2.py", line 484, in init_process
    fn(rank, size)
  File "/media/cfs/gonglixing/9Nctl/gpt_v2/run_clm_v2.py", line 350, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/datasets/dataset_dict.py", line 489, in map
    for k, dataset in self.items()
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/datasets/dataset_dict.py", line 489, in <dictcomp>
    for k, dataset in self.items()
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1693, in map
    transformed_shards = [r.get() for r in results]
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1693, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get
    raise self._value
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/multiprocess/pool.py", line 424, in _handle_tasks
    put(task)
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/dill/_dill.py", line 498, in dump
    StockPickler.dump(self, obj)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/dill/_dill.py", line 1496, in save_function
    obj.__dict__, fkwdefaults), obj=obj)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/media/cfs/gonglixing/.pylib/lib/python3.6/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/anaconda3/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle _LazyModule objects

```

Since we can't see your custom code, it's hard to tell why you have a problem. At least checking your traceback it doesn't match the current `run_clm.py` version in master. Perhaps you are running an unmodified code that still has the original problem?

Perhaps give a try to `run_clm.py` in master?

If it doesn't work, please open a new Issue and give us all the required details to be able to reproduce the problem. And tag me to it. Thank you.

> Since we can't see your custom code, it's hard to tell why you have a problem. At least checking your traceback it doesn't match the current `run_clm.py` version in master. Perhaps you are running an unmodified code that still has the original problem?
> 
> Perhaps give a try to `run_clm.py` in master?
> 
> If it doesn't work, please open a new Issue and give us all the required details to be able to reproduce the problem. And tag me to it. Thank you.

I tried the `run_clm.py`  in master, but it still doesn't work. I will create a new issue.
Thanks for your reply!