Issue 1:
Example:

>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
... print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'

As you can see, after the word "naïve", the offsets go off the rails.

Issue 2:
🐛 Bug

Information

Model I am using (Bert, XLNet ...): Bert (could happen with other ones, don't know)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

from transformers import AutoTokenizer

pretrained_model_name = "bert-base-cased"

fast_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
slow_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name, use_fast=False)

text = "naïve"

assert fast_tokenizer.encode(text) == slow_tokenizer.encode(text)

With the slow, it only strips accents if lowercase is enabled (maybe a bug?):

transformers/src/transformers/tokenization_bert.py

Line 346
in
e676764

token = self._run_strip_accents(token)

With the fast one, it'd never strip accents:

https://github.com/huggingface/tokenizers/blob/python-v0.5.0/bindings/python/tokenizers/implementations/bert_wordpiece.py#L23

transformers/src/transformers/tokenization_bert.py

Lines 557 to 565
in
e676764

BertWordPieceTokenizer(

vocab_file=vocab_file,

add_special_tokens=add_special_tokens,

unk_token=unk_token,

sep_token=sep_token,

cls_token=cls_token,

handle_chinese_chars=tokenize_chinese_chars,

lowercase=do_lower_case,

),

I'd be cool to have that flag also, in both tokenizers.

Finally, this warning seems odd for the simple code from above:

>>> assert fast_tokenizer.encode(text) == slow_tokenizer.encode(text)
Disabled padding because no padding token set (pad_token: [PAD], pad_token_id: 0).
To remove this error, you can add a new pad token and then resize model embedding:
tokenizer.pad_token = '<PAD>'
model.resize_token_embeddings(len(tokenizer))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError

Maybe here the if pad_to_max_length should be nesting the rest of the if?

transformers/src/transformers/tokenization_utils.py

Lines 80 to 95
in
e676764

if pad_to_max_length and (pad_token and pad_token_id >= 0):

tokenizer.enable_padding(

max_length=None,

direction=padding_side,

pad_id=pad_token_id,

pad_type_id=pad_token_type_id,

pad_token=pad_token,

)

else:

logger.warning(

"Disabled padding because no padding token set (pad_token: {}, pad_token_id: {}).\n"

"To remove this error, you can add a new pad token and then resize model embedding:\n"

"\ttokenizer.pad_token = '<PAD>'\n\tmodel.resize_token_embeddings(len(tokenizer))".format(

pad_token, pad_token_id

)

)

Didn't check in the other transformer models.

Expected behavior

The 2 tokenizer outputs (slow and fast) should be the same.

The tokenizers should allow you to choose if to strip accents or not.

That warning shouldn't appear, IMHO.

Environment info

transformers version: 2.5.0

Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid

Python version: 3.7.4

PyTorch version (GPU?): 1.4.0 (True)

Tensorflow version (GPU?): 2.0.0 (False)

Using GPU in script?: No

Using distributed or parallel set-up in script?: No