Can't load pre-trained Flaubert model
Author: LoicH
Pull request: no
Labels: name: Need more information

name: wontfix

name: Core: CLI

name: Core: Tokenization

# 🐛 Bug


## Information

Model I am using (Bert, XLNet ...): Flaubert

Language I am using the model on (English, Chinese ...): French

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load a pre-trained model
2.
3.

I'm following the guide from https://huggingface.co/transformers/model_doc/flaubert.html#flaubertmodel: 
```
import transformers
tokenizer = transformers.FlaubertTokenizer.from_pretrained('flaubert-base-cased')
```

```
Traceback (most recent call last):
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-05c64572fe39>", line 2, in <module>
    tokenizer = transformers.FlaubertTokenizer.from_pretrained('flaubert-base-cased')
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\tokenization_utils.py", line 309, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\tokenization_utils.py", line 410, in _from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name 'flaubert-base-cased' was not found in tokenizers model name list (flaubert-large-cased, flaubert-base-uncased, flaubert-small-cased, flaubert-base-cased). We assumed 'flaubert-base-cased' was a path, a model identifier, or url to a directory containing vocabulary files named ['merges.txt', 'vocab.json'] but couldn't find such vocabulary files at this path or url.

```

## Expected behavior

`tokenizer` should be a `FlaubertTokenizer` object

## Environment info

Well calling `python transformers-cli env` gave me another error:

```
(venv) C:\Users\PLHT09191\Documents\work\dev\Classif_Annonces\venv\Scripts>python transformers-cli env
Traceback (most recent call last):
  File "transformers-cli", line 4, in <module>
    __import__('pkg_resources').run_script('transformers==2.4.1', 'transformers-cli')
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\setuptools-40.8.0-py3.5.egg\pkg_resources\__init__.py", line 666, in run_script
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\setuptools-40.8.0-py3.5.egg\pkg_resources\__init__.py", line 1446, in run_script
  File "c:\users\myself\documents\work\dev\classif_annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\EGG-INFO\scripts\transformers-cli", line 6, in <module>
    from transformers.commands.user import UserCommands
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\commands\user.py", line 163
    entries: List[os.DirEntry] = list(os.scandir(rel_path))
           ^
SyntaxError: invalid syntax
```

     
- `transformers` version: 2.4.1
- Platform: Windows 64 bits
- Python version: Python 3.5.2
- PyTorch version (GPU?): torch.__version__ = 1.4.0+cpu
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
On the second issue, the CLI is Python 3.6+ only. We'll document this better in the future, cc @LysandreJik 

On the first issue, looks like your traceback might be truncated. Did you paste all of it?

> 
> 
> On the first issue, looks like your traceback might be truncated. Did you paste all of it?

Yes indeed I forgot the last lines, don't know why... I edited my original post to include the full traceback: 

> Traceback (most recent call last):
>   File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
>     exec(code_obj, self.user_global_ns, self.user_ns)
>   File "<ipython-input-2-05c64572fe39>", line 2, in <module>
>     tokenizer = transformers.FlaubertTokenizer.from_pretrained('flaubert-base-cased')
>   File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\tokenization_utils.py", line 309, in from_pretrained
>     return cls._from_pretrained(*inputs, **kwargs)
>   File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\tokenization_utils.py", line 410, in _from_pretrained
>     list(cls.vocab_files_names.values()),
> OSError: Model name 'flaubert-base-cased' was not found in tokenizers model name list (flaubert-large-cased, flaubert-base-uncased, flaubert-small-cased, flaubert-base-cased). We assumed 'flaubert-base-cased' was a path, a model identifier, or url to a directory containing vocabulary files named ['merges.txt', 'vocab.json'] but couldn't find such vocabulary files at this path or url.
> 

I can't replicate this issue (FlaubertTokenizer) in either v2.4.0 or v2.4.1, does it arise when you simply do 

```py
from transformers import FlaubertTokenizer
tokenizer= FlaubertTokenizer.from_pretrained("flaubert-base-cased")
```
?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.