Fine tune masked language model on custom dataset 'index out of range in self'
Author: dragonlee97
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hi  I am training a mlm model with reference of this tutorial, https://huggingface.co/blog/how-to-train. However,I got the error of 'index out of range in self' but I already set max_length as well as block size in my code.  I am also not clear how to load and prepare data, should I mask certain words by myself? and pass the masked label while training by myself?  Because herehttps://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm  I see '– Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size]' I am not sure using Trainer and DataCollatorForLanguageModeling could solve this.
## Details
<!-- Description of your issue -->
To reprpduce: 

```
tokenizer = BertTokenizer.from_pretrained('./bert-large-cased',truncation = True, padding=True, max_length=100)
model = BertForMaskedLM.from_pretrained(
    "./bert-large-cased", 
    output_attentions = False, 
    output_hidden_states = True
)
device = 'cuda' if torch.cuda.is_available() else 'cpu' # Tell pytorch to run this model on the GPU.
model = model.to(device)
model.train()

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,mlm=True, mlm_probability=0.15  #I don't know if this is the only way to set up mask for mlm task....
)

from transformers import LineByLineTextDataset
train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./na_en_train.txt", #I reorgnasized data to line by line form as the tutorial, which is stupid. but  I also tried TensorDataset, it got errors
    block_size=100,
)
eval_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./na_en_test.txt",
    block_size=100,
)
from transformers import Trainer, TrainingArguments
training_args = TrainingArguments(
    output_dir="./results",
    overwrite_output_dir=True,
    num_train_epochs=3,
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=16,
    save_steps=10000,
    save_total_limit=2
)
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    prediction_loss_only=True,
)
trainer.train()

```

#Error Message:
---------------------------------------------------------------------------

> IndexError                                Traceback (most recent call last)
> <ipython-input-116-3435b262f1ae> in <module>
> ----> 1 trainer.train()
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path)
>     497                     continue
>     498 
> --> 499                 tr_loss += self._training_step(model, inputs, optimizer)
>     500 
>     501                 if (step + 1) % self.args.gradient_accumulation_steps == 0 or (
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/transformers/trainer.py in _training_step(self, model, inputs, optimizer)
>     620             inputs["mems"] = self._past
>     621 
> --> 622         outputs = model(**inputs)
>     623         loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
>     624 
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
>     548         .. note::
>     549             This method modifies the module in-place.
> --> 550 
>     551         Args:
>     552             device (:class:`torch.device`): the desired device of the parameters
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states, **kwargs)
>    1081             encoder_attention_mask=encoder_attention_mask,
>    1082             output_attentions=output_attentions,
> -> 1083             output_hidden_states=output_hidden_states,
>    1084         )
>    1085 
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
>     548         .. note::
>     549             This method modifies the module in-place.
> --> 550 
>     551         Args:
>     552             device (:class:`torch.device`): the desired device of the parameters
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states)
>     751 
>     752         embedding_output = self.embeddings(
> --> 753             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
>     754         )
>     755         encoder_outputs = self.encoder(
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
>     548         .. note::
>     549             This method modifies the module in-place.
> --> 550 
>     551         Args:
>     552             device (:class:`torch.device`): the desired device of the parameters
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
>     176 
>     177         if inputs_embeds is None:
> --> 178             inputs_embeds = self.word_embeddings(input_ids)
>     179         position_embeddings = self.position_embeddings(position_ids)
>     180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
>     548         .. note::
>     549             This method modifies the module in-place.
> --> 550 
>     551         Args:
>     552             device (:class:`torch.device`): the desired device of the parameters
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
>     112             assert list(_weight.shape) == [num_embeddings, embedding_dim], \
>     113                 'Shape of weight does not match num_embeddings and embedding_dim'
> --> 114             self.weight = Parameter(_weight)
>     115         self.sparse = sparse
>     116 
> 
> /opt/conda/envs/rapids/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
>    1722     if not torch.jit.is_scripting():
>    1723         if type(input) is not Tensor and has_torch_function((input,)):
> -> 1724             return handle_torch_function(hardswish, (input,), input, inplace=inplace)
>    1725     if inplace:
>    1726         return torch._C._nn.hardswish_(input)
> 
> IndexError: index out of range in self
> **A link to original question on the forum/Stack Overflow**: https://discuss.huggingface.co/t/fine-tune-masked-language-model-on-custom-dataset/747

Comments:
same question

same issues coming for me as well

same issue

Does this still happen in the latest transformers version? Could you put the output of `transformers-cli` env here? Thanks.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

Reopening. Still having same issue?