RuntimeError: index out of range at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237
Author: rvoak
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert-base-uncased

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

I am trying to fine-tune BERT for sentiment analysis on the IMDB Dataset. Most of the code is based on this blog: http://mccormickml.com/2019/07/22/BERT-fine-tuning/

I create my model as follows:
```
from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
    num_labels = 2, # The number of output labels--2 for binary classification.
    output_attentions = False, # Whether the model returns attentions weights.
    output_hidden_states = False, # Whether the model returns all hidden-states.
)

```

In my training loop, I do the following:
```
for step, batch in enumerate(train_dataloader):
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)
     
        model.zero_grad()

       loss, logits = model(b_input_ids, 
                             token_type_ids=None, 
                             attention_mask=b_input_mask, 
                             labels=b_labels)
```

I get the following error: 
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-77ffde63bb0d> in <module>()
    109                              token_type_ids=None,
    110                              attention_mask=b_input_mask,
--> 111                              labels=b_labels)
    112 
    113         # Accumulate the training loss over all of the batches so that we can

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels)
   1030                             position_ids=position_ids,
   1031                             head_mask=head_mask,
-> 1032                             inputs_embeds=inputs_embeds)
   1033 
   1034         pooled_output = outputs[1]

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734 
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    185         if inputs_embeds is None:
    186             inputs_embeds = self.word_embeddings(input_ids)
--> 187         position_embeddings = self.position_embeddings(position_ids)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    189 

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115 
    116     def extra_repr(self):

~/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1465         # remove once script supports set_grad_enabled
   1466         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1467     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1468 
   1469 

RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:237
```



<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.3.0
- Platform: Mac OS
- Python version: 3.6
- PyTorch version (GPU?): 1.2.0, no GPU
- Tensorflow version (GPU?): 1.2.0, no GPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
How big are your batches? BERT can only accept tensors of maximum length of 512. Is it possible one of your inputs is longer than 512?

I had the same issue while my `per_gpu_eval_batch_size=8` and `per_gpu_train_batch_size=8 `. 

Yes, it's unrelated to batch size. Is your sequence length longer than 512?

Also having this issue, getting a different error when I try to use GPU but I understand that the error is caused by the same thing and the CPU stack trace is just more informative. My sequence lengths are definitely <512

> I had the same issue while my `per_gpu_eval_batch_size=8` and `per_gpu_train_batch_size=8 `.

I got the same error when I use batch_size = 8. After I change it to 64, then no errors. 

> Also having this issue, getting a different error when I try to use GPU but I understand that the error is caused by the same thing and the CPU stack trace is just more informative. My sequence lengths are definitely <512

did you find the solution?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

RuntimeError: index out of range: Tried to access index 0 out of table with 202 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

is there anyone knows what's the problem ? , I checked the embedding matrix of shape (203, 16), it shows index 0 out of range, how it is possible???