Voicelab

/

herbert-base-cased-sentiment

like 6

Hi, I get this warning when I run the snippet in loop:

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

I was looking for solution here https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning and here https://github.com/huggingface/transformers/issues/5486

I set TOKENIZERS_PARALLELISM=false in my bashrc, but it did't help.

Is this warning means creating new tokenizer procces at every iteration and polluting my memory?