Issue 1:
Environment info

transformers version:4.3.3

Platform:Pytorch

Python version:3.7.0

PyTorch version (GPU?):GPU

Tensorflow version (GPU?):

Using GPU in script?:

Using distributed or parallel set-up in script?:NO

Who can help

Information

Model I am using (Bert, XLNet ...):

The problem arises when using:

the official example scripts: (give details below)

https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1.python run_mlm.py --model_name_or_path release_model/ --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir test-mlm --max_seq_length 128

2.Got an error:

[INFO|trainer.py:1408] 2021-03-15 11:10:32,884 >> Saving model checkpoint to test-mlm
[INFO|configuration_utils.py:304] 2021-03-15 11:10:32,886 >> Configuration saved in test-mlm/config.json
[INFO|modeling_utils.py:817] 2021-03-15 11:10:33,863 >> Model weights saved in test-mlm/pytorch_model.bin
Traceback (most recent call last):
File "run_mlm.py", line 475, in <module>
main()
File "run_mlm.py", line 450, in main
trainer.log_metrics("train", metrics)
AttributeError: 'Trainer' object has no attribute 'log_metrics'

Expected behavior

The script finish without error.

Issue 2:
I've been trying to work out why I keep getting a CUDA assert in a specific mini-batch when training RoBERTa from scratch. I finally tracked it down after switching to CPU.

I don't understand why padding_idx is added to incremental_indices below? - edit: I do in that the embedding needs a padding mask but it I'm not sure it's the correct way to do it.

In my case padding_idx=3. And one of my input_ids rows was truncated. Say I have input_ids = [[4,5,6],[4,3,3]], this results in mask=[[1,1,1],[1,0,0]] and incremental index=[[1,2,3],[1,0,0]]. Adding padding_idx then produces [[4,5,6],[4,3,3]].

The issue is self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size) so for any sequences which are truncated adding anything to the indices results in an index which is greater than the embedding dim.

Perhaps you can argue that max_position_embeddings is supposed to be larger than the largest possible sequence so this doesn't happen? There is a check in run_mlm.py that data_args.max_seq_length > tokenizer.model_max_length but it seems that in actual fact to avoid a very hard to track down error you must have tokenizer.model_max_length < data_args.max_seq_length

def create_position_ids_from_input_ids(input_ids, padding_idx):
"""
Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols
are ignored. This is modified from fairseq's `utils.make_positions`.

Args:
x: torch.Tensor x:

Returns: torch.Tensor
"""
# The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
mask = input_ids.ne(padding_idx).int()
incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
return incremental_indices.long() + padding_idx