Repository: huggingface/transformers

Issue:
Error on `PreTrainedTokenizerBase.batch_encode_plus` with `return_overflowing_tokens=True, truncation=True`
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2 (master branch)
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.8.1
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

 tokenizers: @mfuntowicz

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the below code

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer.batch_encode_plus(
    ["foo", "bar " * 1000], return_overflowing_tokens=True, truncation=True, padding=True
)
```

raises the following error:

```
Traceback (most recent call last):
  File "foo.py", line 4, in <module>
    tokenizer.batch_encode_plus(
  File "/Users/user/work/transformers/src/transformers/tokenization_utils_base.py", line 2121, in batch_encode_plus
    return self._batch_encode_plus(
  File "/Users/user/work/transformers/src/transformers/tokenization_utils.py", line 534, in _batch_encode_plus
    batch_outputs = self._batch_prepare_for_model(
  File "/Users/user/work/transformers/src/transformers/tokenization_utils.py", line 606, in _batch_prepare_for_model
    batch_outputs = self.pad(
  File "/Users/user/work/transformers/src/transformers/tokenization_utils_base.py", line 2305, in pad
    assert all(
AssertionError: Some items in the output dictionnary have a different batch size than others.
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

No error


Discussion:
Try `padding=True` ?
@patil-suraj Same error occurs

Resolving patch:
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -2440,6 +2440,7 @@ def prepare_for_model(
         total_len = len_ids + len_pair_ids + (self.num_special_tokens_to_add(pair=pair) if add_special_tokens else 0)
 
         # Truncation: Handle max sequence length
+        overflowing_tokens = []
         if truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE and max_length and total_len > max_length:
             ids, pair_ids, overflowing_tokens = self.truncate_sequences(
                 ids,
@@ -2448,9 +2449,10 @@ def prepare_for_model(
                 truncation_strategy=truncation_strategy,
                 stride=stride,
             )
-            if return_overflowing_tokens:
-                encoded_inputs["overflowing_tokens"] = overflowing_tokens
-                encoded_inputs["num_truncated_tokens"] = total_len - max_length
+
+        if return_overflowing_tokens:
+            encoded_inputs["overflowing_tokens"] = overflowing_tokens
+            encoded_inputs["num_truncated_tokens"] = total_len - max_length
 
         # Add special tokens
         if add_special_tokens:


Test patch:
diff --git a/tests/test_tokenization_common.py b/tests/test_tokenization_common.py
--- a/tests/test_tokenization_common.py
+++ b/tests/test_tokenization_common.py
@@ -1352,6 +1352,18 @@ def test_prepare_for_model(self):
 
             self.assertEqual(input_dict, prepared_input_dict)
 
+    def test_batch_encode_plus_overflowing_tokens(self):
+        tokenizers = self.get_tokenizers(do_lower_case=False)
+        for tokenizer in tokenizers:
+            string_sequences = ["Testing the prepare_for_model method.", "Test"]
+
+            if tokenizer.pad_token is None:
+                tokenizer.add_special_tokens({"pad_token": "[PAD]"})
+
+            tokenizer.batch_encode_plus(
+                string_sequences, return_overflowing_tokens=True, truncation=True, padding=True, max_length=3
+            )
+
     @require_torch
     @require_tf
     def test_batch_encode_plus_tensors(self):