Unexpected logits shape on prediction with TFRobertaForSequenceClassification
Author: steindor
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
  

- `transformers` version: 4.0.0
- Platform: Linux-4.9.0-11-amd64-x86_64-with-debian-9.11
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0a0+bf2bbd9 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Distributed

I am using the the TFRobertaForSequenceClassification to create a classifier. According to the documentation, the logits output should have a shape of (batch_size, num_labels) which makes sense. I however get (batch_size, seq_length, num_labels).

Code to reproduce:

```
from transformers import TFRobertaForSequenceClassification, RobertaConfig
import numpy as np

seq_len = 512

classifier = TFRobertaForSequenceClassification(RobertaConfig())

#create random inputs for demo
input_ids = np.random.randint(0,10000, size=(seq_len,))
attention_mask = np.random.randint(0,2, size=(seq_len,))
token_type_ids = np.random.randint(0,2, size=(seq_len,))

#make a prediction with batch_size of 1
output = classifier.predict([input_ids, attention_mask, token_type_ids])

print(output.logits.shape) -> prints out (512,2)
```

## Expected behavior

Logits in the shape of (batch_size,num_labels) or (1,2)



Comments:
Hello! The main issue here is that your arrays are of shape `(seq_length)`, whereas they should be of shape `(batch_size, seq_length)`, even if the batch size is 1.

Updating your code to reflect that:

```py
from transformers import TFRobertaForSequenceClassification, RobertaConfig
import numpy as np

bs = 1
seq_len = 510

classifier = TFRobertaForSequenceClassification(RobertaConfig())

#create random inputs for demo
input_ids = np.random.randint(0,10000, size=(bs, seq_len,))
attention_mask = np.random.randint(0,2, size=(bs, seq_len,))
token_type_ids = np.random.randint(0,2, size=(bs, seq_len,))

#make a prediction with batch_size of 1
output = classifier.predict([input_ids, attention_mask, token_type_ids])

print(output.logits.shape)  # ->  outputs (1, 2)
```

However, there seems to be an error as the model cannot handle a sequence length of 512 when used this way. @jplu running the above code with a sequence length of 512 results in the following error:

```
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[0,510] = 512 is not in [0, 512)
	 [[node tf_roberta_for_sequence_classification/roberta/embeddings/position_embeddings/embedding_lookup (defined at /home/lysandre/Workspaces/Python/transformers/src/transformers/models/roberta/modeling_tf_roberta.py:199) ]] [Op:__inference_predict_function_8030]

Errors may have originated from an input operation.
Input Source operations connected to node tf_roberta_for_sequence_classification/roberta/embeddings/position_embeddings/embedding_lookup:
 tf_roberta_for_sequence_classification/roberta/embeddings/add (defined at /home/lysandre/Workspaces/Python/transformers/src/transformers/models/roberta/modeling_tf_roberta.py:122)

Function call stack:
predict_function
```

Using a smaller sequence length doesn't raise the error. Do you mind weighing in on the issue?

Yep, you are limited to 510 tokens + 2 extra tokens (beginning + end)

After talking about it a bit offline with @jplu we realize there might be an issue with the `predict` method when passing in the values as a list. Could you try passing them as a dictionary instead?

Doing this instead:

```py
output = classifier.predict({"input_ids": input_ids, "attention_mask": attention_mask, "token_type_ids": token_type_ids})
```

Hei! Thank you for the feedback. I passed the parameters as a dict with everything else unchanged but still get the output as (seq_len, num_labels) unfortunately. 

Can you try this:
```
from transformers import TFRobertaForSequenceClassification, RobertaConfig
import numpy as np

bs = 1
seq_len = 510

classifier = TFRobertaForSequenceClassification(RobertaConfig())
input_ids = np.random.randint(0,10000, size=(bs, seq_len,))
attention_mask = np.random.randint(0,2, size=(bs, seq_len,))
token_type_ids = np.zeros(shape=(bs, seq_len,))
classifier.predict({"input_ids": input_ids, "attention_mask": attention_mask, "token_type_ids": token_type_ids})
```

Yes, this works with seq_len = 510. It might help stating this behaviour in the docs or perhaps raise an error or show a warning when one tries to input an unbatched sample.  Also a bit confusing that seq_len needs to be 510 and not 512 to account for the extra tokens (and the error received when one tries with 512 is a bit murky). Anyway, thanks for the help. I'll go ahead and close this.