text: 
  
  TITLE
  Is the GPT-2 forward too different from Bert or RoBerta?
  
  COMMENTS
  5
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I am using some Transformers (Bert, RoBerta, etc.) in my project.
  When including the `GPT-2` as shown below:
  
  ```python
  from transformers import GPT2Tokenizer, GPT2Model
  import torch
  
  # inits model and tokenizer
  tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
  tokenizer.add_special_tokens({'pad_token': '[PAD]'})
  model = GPT2Model.from_pretrained('gpt2')
  
  # inputs
  inputs = tokenizer.encode(text="Hello, my dog is cute", max_length=12, padding="max_length",
                                           truncation=True)
  #[15496, 11, 616, 3290, 318, 13779, 50257, 50257, 50257, 50257, 50257, 50257]
  
  # input_ids tensor and attention mask
  features = torch.tensor([inputs])
  # tensor([[15496,    11,   616,  3290,   318, 13779, 50257, 50257, 50257, 50257,
  #        50257, 50257]])
  attention_mask = (features < 50257).int()
  # tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]], dtype=torch.int32)
  
  # outputs
  outputs = model(
      input_ids=features, 
      attention_mask=attention_mask
  )
  
  ```
  
  I have had the following error:
  
  ```python
  IndexError                                Traceback (most recent call last)
  <ipython-input-31-763ba5835cf8> in <module>()
  ----> 1 outputs = model(input_ids=features, attention_mask=attention_mask)
  
  4 frames
  /usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
      725             result = self._slow_forward(*input, **kwargs)
      726         else:
  --> 727             result = self.forward(*input, **kwargs)
      728         for hook in itertools.chain(
      729                 _global_forward_hooks.values(),
  
  /usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, use_cache, output_attentions, output_hidden_states, return_dict)
      679 
      680         if inputs_embeds is None:
  --> 681             inputs_embeds = self.wte(input_ids)
      682         position_embeds = self.wpe(position_ids)
      683         hidden_states = inputs_embeds + position_embeds
  
  /usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
      725             result = self._slow_forward(*input, **kwargs)
      726         else:
  --> 727             result = self.forward(*input, **kwargs)
      728         for hook in itertools.chain(
      729                 _global_forward_hooks.values(),
  
  /usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
      124         return F.embedding(
      125             input, self.weight, self.padding_idx, self.max_norm,
  --> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
      127 
      128     def extra_repr(self) -> str:
  
  /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
     1850         # remove once script supports set_grad_enabled
     1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
  -> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
     1853 
     1854 
  
  IndexError: index out of range in self
  ```