Issue 1:
Environment info

transformers version: 4.2.2

Platform: Linux-4.15.0-45-generic-x86_64-with-glibc2.10

Python version: 3.8.3

PyTorch version (GPU?): 1.6.0 (True)

Tensorflow version (GPU?): not installed (NA)

Using GPU in script?: No

Using distributed or parallel set-up in script?: No

Who can help

Information

Model I am using (Bert, XLNet ...):

The problem arises when using:

my own modified scripts: (give details below)

The tasks I am working on is:

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

The code

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
words = ['(cid:3)', '하셨습니까', '하다']
tokenizer.batch_encode_plus(
[words],
max_length=512,
truncation=True,
padding=True,
is_split_into_words=True,
return_offsets_mapping=True,
return_special_tokens_mask=True,
return_tensors="pt",
)

The offset_mapping in the output is

tensor([[[0, 0], # [CLS]
[0, 1], # for '('
[1, 4], # for 'cid'
[4, 5], # for ':'
[5, 6], # for '3'
[6, 7], # for ')'
[0, 5], # for '하셨습니까'
[0, 1], # for '하'
[0, 1], # for '하'
[1, 2], # for '다'
[1, 2], # for '다'
[0, 0]]])

As you could find, it generates four tokens for 하다. The output is correct according to Byte pair encoding. However, it generates duplicated [0,1] and [1,2]s, which changes the structure of the outputs (for regular tokens, it can only have one [0,x], which can be used to project the encoded tokens back to their original positions). Therefore, we need extra indicators for positions where Byte-pair encoding is used.

Expected behavior

An additional output showing the mapping for input_ids -> original_token_ids . In this case, it should be something like:

[0, 1, 1, 1, 1, 1, 2, 3, 3, 3, 3 0]

Therefore, we could use this map to figure out byte code embedding is used for the 3rd token.

Updated - @n1t0

Issue 2:
Environment info

transformers version: 4.12.2

Platform: Linux-5.4.0-87-generic-x86_64-with-glibc2.10

Python version: 3.8.8

PyTorch version (GPU?): 1.9.1+cu111 (True)

Tensorflow version (GPU?): not installed (NA)

Flax version (CPU?/GPU?/TPU?): not installed (NA)

Jax version: not installed

JaxLib version: not installed

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: No

Who can help

@LysandreJik

Information

Model I am using (Bert, XLNet ...): BertLMHeadModel

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

Instantiate a BertLMHeadModel. The model contains, among others, a linear layer under the path cls.predictions.decoder . Specifically, it contains 2 parameters: cls.predictions.decoder.weight and cls.predictions.decoder.bias.

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("bert-base-uncased", is_decoder=True)
print(model.cls.predictions.decoder)

Get a list of model parameters.

parameter_keys = list(dict(model.named_parameters()).keys())

The decoder parameters are missing.

print("cls.predictions.decoder.weight" in parameter_keys)
print("cls.predictions.decoder.bias" in parameter_keys)

Note that the parameters appear under the cls module:

print(list(dict(model.cls.named_parameters()).keys()))

Note that the cls.decoder module appears to be registerred.

print(list(dict(model.named_modules()).keys()))

Expected behavior

Decoder parameters should be incldued in the model parameters.