Tokenizers throwing warning "The current process just got forked, Disabling parallelism to avoid deadlocks.. To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)"
Author: saahiluppal
Pull request: no
Labels: name: Core: Tokenization

I know this warning is because the transformer library is updated to 3.x.
I know the warning saying to set TOKENIZERS_PARALLELISM = true / false

My question is where should i set TOKENIZERS_PARALLELISM = true  / false
is this when defining tokenizers like 
```
tok = Tokenizer.from_pretrained('xyz', TOKENIZERS_PARALLELISM=True) // this doesn't work
```
or is this when encoding text like
```
tok.encode_plus(text_string, some=some, some=some, TOKENIZERS_PARALLELISM = True) // this also didn't work
```

Suggestions anyone?

Comments:
This might help you: https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning

I suspect this may be caused by loading data. In my case, it happens when my dataloader starts working.

This is happening whenever you use `multiprocessing` (Often used by data loaders). The way to disable this warning is to set the `TOKENIZERS_PARALLELISM` environment variable to the value that makes more sense for you. By default, we disable the parallelism to avoid any hidden deadlock that would be hard to debug, but you might be totally fine while keeping it enabled in your specific use-case.

You can try to set it to `true`, and if your process seems to be stuck, doing nothing, then you should use `false`.

We'll improve this message to help avoid any confusion (Cf https://github.com/huggingface/tokenizers/issues/328)

I may be a rookie, but it seems like it would be useful to indicate that this is an environment variable in the warning message.

You are totally right! In the latest version `3.0.2`, the warning message should be a lot better, and it will trigger only when necessary.

Hi, sorry to bump this thread... 

I'm having the same problem however, the tokenizer is used only in my model. 

Data loading is made with multiple workers but it is only loading raw text which is then given to the model and only the model uses the tokenizer. 
I don't have multi model or whatever, just a classic pytorch model. 

Thus I was wondering how can I have the  warning. 

Thanks in advance, 
Have a great day :)  

You must be using a tokenizer before using `multiprocessing`. When your process gets forked, you see this message because it detects that a fork is happening and that some kind of parallelism was used before.

@n1t0, 
Thanks a lot for the fast reply, 
I guess it detect a fork even if it's safe for me to do so... Yes my process is forked but not the tokenizer. 

Then I will use the env variable to remove the warning. 

I use ```tokenizer``` in my data loader.

If that is the source of this problem (hence disabling the parallelization --> hence slow training), then what is the solution? 

Using ```tokenizer``` in the pre-processing step? 

After testing, it is found that when the data in a dataloader is processed by the token, and the datalodaer jumps out before it is finished, this warning will be triggered; 
I give a code example:
```
# for example, following code will trigger the warning
for texts in train_dataloader:
    _ = tokenizer.batch_encode_plus(texts)
    # loader has not been traversed
    # but texts are used
    break 
for texts in test_dataloader:
    # warning ...
    pass or break

# and following code will not trigger the warning
for texts in train_dataloader:
    # loader has not been traversed
    # but texts are not used
    break 
for texts in test_dataloader:
    # No warning 
    pass or break
```

@hbchen121 my dataloader processes the text in init function

During data loading time, directly input_ids and attention masks are fetched, yet I get this warning.

Despite [the documentation](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html) saying that `use_fast` defaults to `False`, adding `use_fast=False` so that it's `AutoTokenizer.from_pretrained(model_name, use_fast=False)` removed this warning for me. If I just use `AutoTokenizer.from_pretrained(model_name)`, the warning pops up again.

I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?

cc @ArthurZucker

> I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?

@hzphzp there is an explanation in SO
https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996

> > I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?
> 
> @hzphzp there is an explanation in SO https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996

 Thank you!

Though each notebook runs fine by itself, I get this warning when running multiple notebooks via `nbdev_test` (https://github.com/fastai/nbdev).  Shortly afterwards it crashes due to out-of-memory.  

I assume it has something to do with multiprocessing in `nbdev_test`, even when setting `--n_workers 1`.

This gets a warning about disabling parallelism to avoid locks:
>  ```nbdev_test --n_workers 1 --pause 10 --do_print --file_glob "*nb"```

This works fine:
> ```$ for x in `ls nbs/*nb`; do nbdev_test --n_workers 1 --do_print --path "$x"; done```

> Despite [the documentation](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html) saying that `use_fast` defaults to `False`, adding `use_fast=False` so that it's `AutoTokenizer.from_pretrained(model_name, use_fast=False)` removed this warning for me. If I just use `AutoTokenizer.from_pretrained(model_name)`, the warning pops up again.

`use_fast` default is `True`, which enables a fast Rust base tokeniser if available, if not a Python base tokeniser will be triggered.