Issue 1:
🐛 Bug

Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

I'm trying to run run_language_modelling.py with my own tokenizer

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

!pip install transformers --upgrade

from transformers import AutoTokenizer
my_tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")
!mkdir my_tokenizer
my_tokenizer.save_pretrained("my_tokenizer")
my_tokenizer2 = AutoTokenizer.from_pretrained("./my_tokenizer")

Stack Trace:

---------------------------------------------------------------------------
OSError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
246 resume_download=resume_download,
--> 247 local_files_only=local_files_only,
248 )

4 frames
/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
260 # File, but it doesn't exist.
--> 261 raise EnvironmentError("file {} not found".format(url_or_filename))
262 else:

OSError: file ./my_tokenizer/config.json not found

During handling of the above exception, another exception occurred:

OSError Traceback (most recent call last)
<ipython-input-16-b699c4abfe8e> in <module>()
3 get_ipython().system('mkdir my_tokenizer')
4 my_tokenizer.save_pretrained("my_tokenizer")
----> 5 my_tokenizer2 = AutoTokenizer.from_pretrained("./my_tokenizer")

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
184 config = kwargs.pop("config", None)
185 if not isinstance(config, PretrainedConfig):
--> 186 config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
187
188 if "bert-base-japanese" in pretrained_model_name_or_path:

/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
185 """
186 config_dict, _ = PretrainedConfig.get_config_dict(
--> 187 pretrained_model_name_or_path, pretrained_config_archive_map=ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, **kwargs
188 )
189

/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
268 )
269 )
--> 270 raise EnvironmentError(msg)
271
272 except json.JSONDecodeError:

OSError: Can't load './my_tokenizer'. Make sure that:

- './my_tokenizer' is a correct model identifier listed on 'https://huggingface.co/models'

- or './my_tokenizer' is the correct path to a directory containing a 'config.json' file

Expected behavior

AutoTokenizer should be able to load the tokenizer from the file.

Possible duplicate of #1063 #3838

Environment info

transformers version:

Platform: Google Colab

Python version: 3.6.9

PyTorch version (GPU?): N/A

Tensorflow version (GPU?): N/A

Using GPU in script?: No

Using distributed or parallel set-up in script?: No

Issue 2:
🐛 Bug

Hi,

I noticed some strange behavior with the fast tokenizers in v2.5.0, which I think is a bug:

It seems BertTokenizerFast is ignoring the pad_to_max_length argument, as shown below:

>>> from transformers import AutoTokenizer, BertTokenizer
>>> tok_auto = AutoTokenizer.from_pretrained('bert-base-uncased')
>>> tok_bert = BertTokenizer.from_pretrained('bert-base-uncased')
>>> a, b = 'Sentence 1', 'Sentence 2'
>>>
>>> tok_bert.encode(a, b, max_length=10, pad_to_max_length=True)
[101, 6251, 1015, 102, 6251, 1016, 102, 0, 0, 0] # <-- Expected behavior
>>> tok_auto.encode(a, b, max_length=10, pad_to_max_length=True)
[101, 6251, 1015, 102, 6251, 1016, 102] # <-- Actual behavior

Also, can some please explain the reason for the warning below that's raised when I set pad_to_max_length=False (which is only there in the fast tokenizer)?

>>> tok_auto.encode(a, b, max_length=10, pad_to_max_length=False)
Disabled padding because no padding token set (pad_token: [PAD], pad_token_id: 0).
To remove this error, you can add a new pad token and then resize model embedding:
tokenizer.pad_token = '<PAD>'
model.resize_token_embeddings(len(tokenizer))
[101, 6251, 1015, 102, 6251, 1016, 102]
>>> tok_bert.encode(a, b, max_length=10, pad_to_max_length=False)
[101, 6251, 1015, 102, 6251, 1016, 102] # <-- No warning

Thanks!