batch_encode_plus with pad_to_max_length but no max_length is not padding the output
Author: muggin
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using BERT:

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:                                                                                                                                             
```
import torch
import numpy as np
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

seq1 = "This is a short sequence"
seq2 = "This will be a much longer sequence, so the short one requires padding"

input = [seq1, seq2]

# Explicitly specified padding length
max_len = 20
tck_temp = tokenizer.batch_encode_plus(input, max_length=max_len, pad_to_max_length=True)
inp_ids = tck_temp['input_ids']
assert len(inp_ids[0]) == len(inp_ids[1]) == max_len, "Both inputs should have length equal to 20"

# Implicit padding length set to models max length
model_max_len = tokenizer.max_len
tck_temp = tokenizer.batch_encode_plus(input, pad_to_max_length=True)inp_ids = tck_temp['input_ids']
assert len(inp_ids[0]) == len(inp_ids[1]) == model_max_len, "Both inputs should have length equal to %d" % model_max_len
```

## Expected behavior
According to the documentation, `batch_encode_plus` with `pad_to_max_length=True` should pad sequence to models maximal length, if the `max_length` is not explicitly specified.
The attached script should run without raising Exception.

From documentation
"If no max length is specified, the padding is done up to the model’s max length."

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.7.0
- Platform: Linux-4.15.0-74-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi,

Is anyone working on this or is it open for someone to take?
I was able to reproduce the issue.
If not being worked on by anyone, I would like to take it up.

Thanks

I'm facing similar issues with batch_encode_plus:
```
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-cased')
a = ['short sentence', 'Larger sentence than short sentence']
input_ids = torch.tensor(tokenizer.batch_encode_plus(a, pad_to_max_length=True)['input_ids'])
```
It doesn't work for me, it return this error:
`ValueError: expected sequence of length 2 at dim 1 (got 6)`

Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This should now be fixed on master with the updated tokenizer API