Ignored unknown kwarg option direction, while running run_mlm.py (pytorch)
Author: saahiluppal
Pull request: no

transformers_version == '4.16.0.dev0'

using script ```run_mlm.py``` (pytorch)
with following parameters
```
!python run_mlm.py \
    --output_dir "./test-roberta-base" \
    --model_type "roberta" \
    --config_name "./test-roberta-base" \
    --tokenizer_name "./test-roberta-base" \
    --train_file "./training_data/test.txt" \
    --max_seq_length 128 \
    --line_by_line \
    --do_train \
    --weight_decay 0.01 \
    --per_device_train_batch_size 256 \
    --per_device_eval_batch_size 256 \
    --learning_rate 3e-4 \
    --warmup_steps 1000 \
    --overwrite_output_dir \
    --num_train_epochs 18 \
    --adam_beta1 0.9 \
    --adam_beta2 0.98 \
    --logging_steps 500 \
    --save_steps 2500 \
    --eval_steps 2500
```

when i pass in the parameter ```--line_by_line```, this happens
```
Running tokenizer on dataset line_by_line:   0%| | 66/1536000 [00:01<6:13:50, 68Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Running tokenizer on dataset line_by_line:   0%| | 76/1536000 [00:01<5:35:58, 76Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Running tokenizer on dataset line_by_line:   0%| | 85/1536000 [00:01<6:44:46, 63Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Ignored unknown kwarg option direction
Running tokenizer on dataset line_by_line:   0%| | 95/1536000 [00:01<6:01:37, 70Ignored unknown kwarg option direction
```

my train_file is in the form
```
sentence1
sentence2
sentence3
and so on...
```

is this expected?

PS: 
using custom tokenizer made via
```
tokenizer = ByteLevelBPETokenizer(lowercase=True)

tokenizer.train(
    files=paths,
    vocab_size=50265,
    min_frequency=10,
    special_tokens=["
", "<pad>", "
", "<unk>", "<mask>"]
)

tokenizer.save("./test-roberta-base/tokenizer.json")
```

Comments:
cc @sgugger 

Looks like a problem in the tokenizers. What's your version of the Tokenizers library?

@sgugger
tokenizers version = '0.9.4'

I think you might need to upgrade it.

Okay, upgraded tokenizers to 0.11.0.
i can confirm that running run_mlm.py script doesn't throw that error anymore.

Thanks.

In my case 0.11.0 still threw error but 0.11.5 worked

Same problem
```
# Name                    Version                   Build  Channel
tokenizers                0.10.3           py38hb317417_1
```

Same issue even with tokenizers version 0.11.6 (using [keyBERT](https://maartengr.github.io/KeyBERT/guides/quickstart.html))

However, I see this warning while I run this in a notebook, but not in the shell.

I had to downgrade to transformers 4.15.0 (and left tokenizers at 0.10.3)

Using conda, you can't install tokenizers version 0.11.x with transformers 4.x. I tried forcing it, but got errors regarding openssl v3 when trying to import tokenizers.

```
conda search transformers --info -c conda-forge
...
transformers 4.16.2 pyhd8ed1ab_0
...
dependencies: 
  - tokenizers >=0.10.1,<0.11
```

and

```
conda search transformers --info -c huggingface
...
transformers 4.14.1 pyhd3eb1b0_0
...
dependencies: 
  - tokenizers >=0.10.1,<0.11
```


Downgrading to transformers 4.15.0 fixed the issue for me! Thanks!

Downgrading 4.15.0 helped. Thank you

Downgrading transformers`conda install -c huggingface -c conda-forge transformers==4.15.0` didn't work for me, tokenizers however

Having the same issue, is this just an annoying superfluous warning or does it affect the results? Because in the former case I will just ignore it.