Tokenizer not working
Author: GuillemGSubies
Pull request: no
Labels: name: Should Fix

## Environment info

- `transformers` version: 4.3.2
- Platform: Ubuntu 16.04.6 LTS
- Python version: 3.8.8

### Who can help

- tokenizers: @n1t0, @LysandreJik


## Information

To reproduce:

```
conda create --name=env1 python=3.8 jupyter transformers tokenizers -y -c conda-forge
conda activate env1
```
```
~$ python
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoTokenizer
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased", do_lower_case=False, strip_accents=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guillem.garcia/.conda/envs/cosas/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 395, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/guillem.garcia/.conda/envs/cosas/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    return cls._from_pretrained(
  File "/home/guillem.garcia/.conda/envs/cosas/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1860, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/guillem.garcia/.conda/envs/cosas/lib/python3.8/site-packages/transformers/models/bert/tokenization_bert_fast.py", line 199, in __init__
    self.backend_tokenizer.normalizer = pre_tok_class(**pre_tok_state)
TypeError: PyBertNormalizer.__new__() got an unexpected keyword argument: do_lower_case
```
The most weird thing is that running that exact command in a jupyter notebook does not raise any error. Also `AutoTokenizer.from_pretrained("bert-base-cased", do_lower_case=False)` works so it seems to be something related to strip accents.


Comments:
Hi! We do not maintain the conda-forge versions of transformers and tokenizers. We maintain the versions that are on the `huggingface` channel.

I just tried with the `huggingface` channel and I get no such errors:

```
conda create --name=env1 python=3.8 jupyter transformers tokenizers -y -c huggingface && conda activate env1
```
See the result:

```
~ (🌟) 🤗 python                                                                     (env1) 10:00:33 ~
Python 3.8.5 (default, Sep  4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoTokenizer
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased", do_lower_case=False, strip_accents=False)
Downloading: 100%|█████████████████████████████████████████████████| 213k/213k [00:00<00:00, 2.71MB/s]
Downloading: 100%|█████████████████████████████████████████████████| 436k/436k [00:00<00:00, 4.67MB/s]
Ignored unknown kwargs option do_lower_case
>>>
```

I got a similar problem with [another BERT model](https://huggingface.co/ltgoslo/norbert)

Everything is OK if the `tokenizer_config.json` contains only this:
```
{
  "do_lower_case": false
}
```

But as soon as another line is added:
```
{
  "do_lower_case": false,
  "do_basic_tokenize": false
}
```
the `AutoTokenizer.from_pretrained("ltgoslo/norbert")` ends in 
`unexpected keyword argument: do_lower_case`,
 which is weird, since the argument obviously is valid, if given alone.

I see the same problem even on the HuggingFace Model Hub itself:
`Can't load tokenizer using from_pretrained, please update its configuration: PyBertNormalizer.__new__() got an unexpected keyword argument: do_lower_case`

What is wrong with the `AutoTokenizer` + `do_basic_tokenize` combination? Locally, everything is fine if I use
`tokenizer = BertTokenizer.from_pretrained("ltgoslo/norbert")`
or
`tokenizer = AutoTokenizer.from_pretrained("ltgoslo/norbert", use_fast=False)`

Hi @akutuzov, the `do_basic_tokenize` is a python tokenizer only attribute, what behavior do you want from it?

You get the error because the `AutoTokenizer` tries to load a fast tokenizer by default.

Thanks @LysandreJik , this is my impression as well. But two questions then:

1. If the problem is with the `do_basic_tokenize`, why the warning says  `unexpected keyword argument: do_lower_case`?
2. Is it possible to tell the `AutoTokenizer` **not** to load the fast tokenizer by default for a particular model? Anything I can put in the `config.json` or `tokenizer_config.json`? Since it seems that fast tokenizers sometimes lack the functionality which is there in the python tokenizers, it would be great to have some way to enforce using the python ones. 

In our case, we need `do_basic_tokenize=False`, since we would like to avoid punctuation splitting.

I have just encountered the same problem. It appears only with Version 4.3.x. For now you could switch back to 4.2.x like me. In version 4.2.2 there is just this output: 
`Ignored unknown kwargs option do_lower_case`

Everythink works as expected in 4.2.2!

@NebelAI I am using 4.2.2. It does not work as expected: the `do_basic_tokenize` parameter is silently ignored by the `AutoTokenizer`, which instead produces a strange warning about `do_lower_case`.
I see it as problematic behavior.


@akutuzov You are right. My problem goes in the same direction but is not identical to yours, sorry.

I was capable of using `AutoTokenizer.from_pretrained(file, use_fast=True)` with `tokenizer.json` as file input for quite some time. After upgrading to 4.3.3 I was facing this weird exception you mentioned at the beginning. So my attempt only works if you are using tokenizer.json which has been created by tokenizers lib.

Bu still ... this error needs to be fixed.

Re-opening this as the issue isn't solved.

related to https://github.com/huggingface/transformers/issues/10121

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I believe this issue has been fixed by https://github.com/huggingface/transformers/pull/10686

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.