Issue 1:
Environment info

transformers version: 4.5.0

Platform: Windows 10 (replicated on Ubuntu 18.04 as well)

Python version: 3.6.8

PyTorch version (GPU?): n/a

Tensorflow version (GPU?): n/a

Using GPU in script?: no

Using distributed or parallel set-up in script?: no

Full package list:

certifi==2020.12.5
chardet==4.0.0
click==7.1.2
dataclasses==0.8
filelock==3.0.12
idna==2.10
importlib-metadata==3.10.0
joblib==1.0.1
numpy==1.19.5
packaging==20.9
protobuf==3.15.7
pyparsing==2.4.7
regex==2021.4.4
requests==2.25.1
sacremoses==0.0.44
sentencepiece==0.1.95
six==1.15.0
tokenizers==0.10.2
tqdm==4.60.0
transformers==4.5.0
typing-extensions==3.7.4.3
urllib3==1.26.4
zipp==3.4.1

Who can help

@LysandreJik

Information

Model I am using (Bert, XLNet ...): n/a

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

>>> tokenizer = XLMRobertaTokenizerFast.from_pretrained('xlm-roberta-base')
>>> tokenizer.save_pretrained('test')
('test\\tokenizer_config.json', 'test\\special_tokens_map.json', 'test\\sentencepiece.bpe.model', 'test\\added_tokens.json')
>>> tokenizer_loaded = XLMRobertaTokenizerFast.from_pretrained('test')
>>> tokenizer('hello world', return_offsets_mapping=True)
{'input_ids': [0, 33600, 31, 8999, 2], 'attention_mask': [1, 1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 4), (4, 5), (6, 11), (0, 0)]}
>>> tokenizer_loaded('hello world', return_offsets_mapping=True)
{'input_ids': [0, 33600, 31, 8999, 2], 'attention_mask': [1, 1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 4), (4, 5), (5, 11), (0, 0)]}

Expected behavior

The loaded tokenizer should produce the same offset mapping as the original tokenizer. It seems like the tokenizer loaded from disk does not account for spaces in its offset mapping.

Issue 2:
Environment info

transformers` version: 4.2.1

Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Python version: 3.6.9

PyTorch version (GPU?): 1.7.0+cu101 (False)

Tensorflow version (GPU?): 2.4.0 (False)

Using GPU in script?: yes

Using distributed or parallel set-up in script?: no

Who can help

@mfuntowicz

@stefan-it

Information

Model I am using is XLM-RoBERTa.

The problem arises when using XLMRobertaTokenizerFast tokenizer.

The tasks I am working on is token-classification. In order to align the labels with the sub-word units I have used the code snippet provided here: https://huggingface.co/transformers/custom_datasets.html [ Fine-tuning with custom datasets/Token Classification with W-NUT Emerging Entities ].

When trying to align the labels with the encodings, it throws: "ValueError: NumPy boolean array indexing assignment cannot assign X input values to the Y output values where the mask is true."

This behavior is due to tokenizing punctuation. Moreover comma ( ' , ' ) gets tokenized into '' and ',' ( having offset values (0,1) ) Similar behavior happens with dot. However, some other punctuation marks are producing only one token (i.g. ' : ' -> ':').

In addition, the offset_mapping value for ':' is different in different sentences resulting either in (0,0) or (0,3) tuple. The problem is that padding tokens have offset tuple with values (0,0) which are excluded from alignment, but in this case I have to preserve the punctuation since it is POS tagging problem.

To reproduce

print("Token: {} Offset_mapping: {}".format(train_encodings[338].tokens[67], train_encodings[338].offsets[67]))
# Token: ▁... Offset_mapping: (0, 0)
print("Token: {} Offset_mapping: {}".format(train_encodings[20].tokens[2], train_encodings[20].offsets[2]))
# Token: ▁... Offset_mapping: (0, 3)

Moreover, although I fixed this issue by writing my own masks, I found new issue: the blank space which denotes start of the word is tokenized as separate token instead of being together with the starting sub-token.

To reproduce

tokenizer = XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-base")
model= XLMRobertaForTokenClassification.from_pretrained("xlm-roberta-base")
s = "Je često kritizirao vladu ."
print(tokenizer.tokenize(s))
# output: ['▁Je', '▁često', '▁krit', 'izira', 'o', '▁', 'vlad', 'u', '▁', '.']

Expected behavior

Punctuation marks should be consistently tokenized and having offset values different from padding tokens.

The first sub-word token should be with preceding blank space everywhere.