Error while trying to retrieve BERT embeddings for a custom task
Author: adithya8
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
I am using BERT [base-uncased].
Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Here's the error:
```
I0615 15:19:52.531945 140468956133120 file_utils.py:41] PyTorch version 1.1.0 available.
I0615 15:19:53.723086 140468956133120 file_utils.py:57] TensorFlow version 2.0.0 available.
bert-base-cased
I0615 15:19:53.860575 140468956133120 configuration_utils.py:256] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json from cache at /users2/user1/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.9da767be51e1327499df13488672789394e2ca38b877837e52618a67d7002391
I0615 15:19:53.861034 140468956133120 configuration_utils.py:292] Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "eos_token_ids": null,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": true,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

I0615 15:19:53.934896 140468956133120 configuration_utils.py:256] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /users2/user1/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517
I0615 15:19:53.935213 140468956133120 configuration_utils.py:292] Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "eos_token_ids": null,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30522
}

I0615 15:19:54.006190 140468956133120 tokenization_utils.py:501] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /users2/user1/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I0615 15:19:54.133543 140468956133120 modeling_utils.py:461] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin from cache at /users2/user1/.cache/torch/transformers/35d8b9d36faaf46728a0192d82bf7d00137490cd6074e8500778afed552a67e5.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "eos_token_ids": null,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": true,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

len(input_ids):  636
Num Batches: 40
---------------------------------------
---------------------------------------
---------------------------------------
Traceback (most recent call last):
  File "./dlatkInterface.py", line 2020, in <module>
    main()
  File "./dlatkInterface.py", line 1013, in main
    args.feattable = fe.addBERTTable_(modelName = args.bertmodel, aggregations=args.bertaggs, layersToKeep=args.bertlayers, noContext=args.bertnocontext, layerAggregations = args.bertlayeraggs, wordAggregations=args.transwordaggs, valueFunc = args.valuefunc)
  File "/users2/user1/NLP/dlatk/dlatk/featureExtractor.py", line 1327, in addBERTTable_
    encAllLayers = model(input_ids = input_ids_padded, attention_mask = attention_mask_padded,  token_type_ids = token_type_ids_padded)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users2/user1/.local/lib/python3.5/site-packages/transformers/modeling_bert.py", line 783, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users2/user1/.local/lib/python3.5/site-packages/transformers/modeling_bert.py", line 173, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:193
```

I am just trying to retrieve the embeddings from the layers that I want and store it in a list. Here's the code block that hits the error:

```
config = AutoConfig.from_pretrained(modelName, output_hidden_states=True)
tokenizer = AutoTokenizer.from_pretrained(tokenizerName)
model = AutoModel.from_pretrained(modelName, config=config)
cuda = False
model.eval()
batch_size=16
.
.
.
num_batches = int(np.ceil(len(input_ids)/batch_size))
encSelectLayers = []
print ('len(input_ids): ',len(input_ids))
print ('Num Batches:', num_batches)

for i in range(num_batches):
    input_ids_padded = pad_sequence(input_ids[i*batch_size:(i+1)*batch_size], batch_first = True, padding_value=tokenizer.pad_token_id)
    token_type_ids_padded = pad_sequence(token_type_ids[i*batch_size:(i+1)*batch_size], batch_first = True, padding_value=0)
    attention_mask_padded = pad_sequence(attention_mask[i*batch_size:(i+1)*batch_size], batch_first = True, padding_value=0)

    if cuda:
        input_ids_padded = input_ids_padded.to('cuda') 
        token_type_ids_padded = token_type_ids_padded.to('cuda') 
        attention_mask_padded = attention_mask_padded.to('cuda')

    input_ids_padded = input_ids_padded.long()
    token_type_ids_padded = token_type_ids_padded.long()
    attention_mask_padded = attention_mask_padded.long()
    
    #print (input_ids_padded.shape, token_type_ids_padded.shape, attention_mask_padded.shape)
    #print (input_ids_padded)
    #print (token_type_ids_padded)
    #print (attention_mask_padded)
    print ('---------------------------------------')
    with torch.no_grad():

        encAllLayers = model(input_ids = input_ids_padded, attention_mask = attention_mask_padded,  token_type_ids = token_type_ids_padded)
        encAllLayers = encAllLayers[-1]            
        for lyr in layersToKeep: #Shape: (num Layers, num_batches, batch_size, max Seq len, 768)
            encSelectLayers.append([encAllLayers[int(lyr)].detach().cpu().numpy()])

        del encAllLayers
attention_mask_padded.shape)
print (np.array(encSelectLayers).shape)
```


## Environment info

- `transformers` version: 2.5.1
- Platform:  Linux-4.4.0-171-generic-x86_64-with-debian-stretch-sid
- Python version: 3.5.2
- CUDA version: 10.1
- PyTorch version (GPU?): 1.1 (True)
- Tensorflow version (GPU?): 2.0 (True)
- Using GPU in script: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi! The error seems to come from your data processing. It's hard to help you without knowing what your inputs are, how `pad_sequence` works, and how you tokenize your inputs.

In recent transformers versions, the tokenizer can take care of truncating/padding and do so for input ids, attention masks and token type ids. Using the encode/encode_plus methods to do this would reduce the risk of errors when pre-processing.

Thank you for your response @LysandreJik 
I would like to add a few more details pertaining to this error here. 
`pad_sequence` is the method from [torch.nn.utils.rnn](https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pad_sequence.html#torch-nn-utils-rnn-pad-sequence). 
With respect to using `encode/encode_plus`, we are processing the text into multiple segments when they go past the max tokens limit. Hence we would still need to process the output of `encode_plus` to fit it in the token limit. [Which is what we follow right now but using the `tokenize`, `convert_tokens_to_ids` and `create_token_type_ids_from_sequences`.

This error doesn't occur when I replace the the three lines of loading config, tokenizer and model (using `AutoConfig`,` AutoTokeizer` and `AutoModel` respectively) with just `BertTokenizer` and `BertModel` directly. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.