ValueError: Wrong shape for input_ids (shape torch.Size([18])) or attention_mask (shape torch.Size([18]))
Author: youssefavx
Pull request: no

## Environment info
 
- `transformers` version: 3.1.0
- Platform: Darwin-18.0.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No (Running on Macbook Pro MacOS Mojave 10.14)

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik 
 
## Information

Model I am using (Bert, XLNet ...): Bert Multilingual

The problem arises when using:
* [ ] the official example scripts: (give details below)

Hi, this is an error I'm getting running a package someone else created called [simalign](https://github.com/cisnlp/simalign/). I believe their package works with transformers 2.3.0 (I tried it and it worked). 

But after trying to upgrade to the newest transformers because I really wanted the fill-mask feature which was more recently released. 

(Actually I wanted a way to get the probability of a word in a certain position given the words around it  (left and right) in a sentence and have yet to find a way to do that). 

At the same time I also want to run simalign in the same application (so I can't downgrade to the previous transformers make it work again).

I'm not exactly sure what is causing the error (as I'm not experienced enough) but I'll share the traceback below here:

```
>>> import simalign
>>> 
>>> source_sentence = "Sir Nils Olav III. was knighted by the norwegian king ."
>>> target_sentence = "Nils Olav der Dritte wurde vom norwegischen König zum Ritter geschlagen ."
>>> model = simalign.SentenceAligner()
2020-09-13 18:02:40,806 - simalign.simalign - INFO - Initialized the EmbeddingLoader with model: bert-base-multilingual-cased
I0913 18:02:40.806071 4394976704 simalign.py:47] Initialized the EmbeddingLoader with model: bert-base-multilingual-cased
>>> result = model.get_word_aligns(source_sentence.split(), target_sentence.split())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simalign/simalign.py", line 181, in get_word_aligns
    vectors = self.embed_loader.get_embed_list(list(bpe_lists))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simalign/simalign.py", line 65, in get_embed_list
    outputs = [self.emb_model(in_ids.to(self.device)) for in_ids in inputs]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simalign/simalign.py", line 65, in <listcomp>
    outputs = [self.emb_model(in_ids.to(self.device)) for in_ids in inputs]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/transformers/modeling_bert.py", line 806, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/transformers/modeling_utils.py", line 248, in get_extended_attention_mask
    input_shape, attention_mask.shape
ValueError: Wrong shape for input_ids (shape torch.Size([18])) or attention_mask (shape torch.Size([18]))
```

I'd love it if anyone could tell me if I could replace a line in the code somewhere to fix this. 

Here's a sample of that code region where that error occurs:

```
def get_embed_list(self, sent_pair):
		if self.emb_model is not None:
			sent_ids = [self.tokenizer.convert_tokens_to_ids(x) for x in sent_pair]
			inputs = [self.tokenizer.prepare_for_model(sent, return_token_type_ids=True, return_tensors='pt')['input_ids'] for sent in sent_ids]

			outputs = [self.emb_model(in_ids.to(self.device)) for in_ids in inputs]
			# use vectors from layer 8
			vectors = [x[2][self.layer].cpu().detach().numpy()[0][1:-1] for x in outputs]
			return vectors
		else:
			return None
```

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

I'm trying to use simalign in conjunction with finding the probabilities of a word in a sentence given its position in that sentence 

(meaning I want to assess the probability of the word 'to' and 'too' in this sentence: "I went to the store"

Given a function like this:

```
find_probability_of_word_in_given_sentence('to', f'I went {given_word} the store')
find_probability_of_word_in_given_sentence('too', f'I went {given_word} the store')
```

I'd want an output like this:
```
to: 0.849283
too: 0.021412
```

And I don't know if there's a way to do that with transformers version 2.3.0, since it does not have the `fill-mask` feature (although that feature does not return word probabilities for a given word, but word predictions.

## To reproduce

Steps to reproduce the behavior:

1. Install simalign
2. Upgrade transformers
3. Run simalign example here: https://github.com/cisnlp/simalign/blob/master/examples/align_example.py

## Expected behavior

Simalign works as it does with transformers 2.3.0, returns a list of tuples with numbers.


Comments:
I had the same error, I had it downgraded `pip install transformers==3.0.2` in order to work.

My problem was not that I needed to downgrade but the fact that I needed new functionality from the newest transformers to work while also working with the features of simalign. I solved this bug here: https://github.com/cisnlp/simalign/issues/10#issuecomment-694407502