Training the TFGPT2LMHeadModel with model.fit produces error
Author: bjourne
Pull request: no

MWE:
```python
from transformers.modeling_tf_gpt2 import TFGPT2LMHeadModel
from transformers.configuration_gpt2 import GPT2Config
from tensorflow.data import Dataset
import tensorflow as tf

data = tf.random.uniform(shape=[10000], dtype = tf.int32, maxval = 100)
src = tf.constant(data)
def split_input_target(chunk):
    return chunk[:-1], chunk[1:]
ds = Dataset.from_tensor_slices(src) \
            .batch(256 + 1, drop_remainder = True) \
            .map(split_input_target)
model = TFGPT2LMHeadModel(GPT2Config())
loss = ['sparse_categorical_crossentropy'] + [None] * 12
model.compile(loss = loss, metrics = ['sparse_categorical_accuracy'])
model.fit(ds)
```
This generates the error `ValueError: Dimensions must be equal, but are 256 and 12 for '{{node Equal_1}} = Equal[T=DT_FLOAT, incompatible_shape_error=true](Cast_6, Cast_7)' with input shapes: [256,1], [2,256,12,1].` Is there any explanation somewhere on how to train like this?





Comments:
Hello!

Your way of computing the loss is wrong. I suggest you to look at how we compute it [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_gpt2.py#L650) and [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L125). You should also rewrite your metric the same way.

I know but changing the loss metric won't fix the ValueError.

It is. Just look at how we do it. Including in the tests.

Met the same error before, it works for me with removing the `metrics` params when `model.compile`. But I think it's a bad idea.

Hi, there. I found it won't raise this error if `batch size` == `n_head`. For example, LysandreJik's [gits](https://gist.github.com/LysandreJik/c958925768eb6a9a72609ea99561d1cb)  works only `BATCH_SIZE = 12`

~~Hey, guys. If the output of other layers was used rarely, can we add a controller to select whether return multi-layer logits or not? (correct me if it's necessary to return multi-layer logits.)~~

~~If we can do this, this [pull request](https://github.com/huggingface/transformers/pull/8584) may help.~~

@bjourne  Hey guy, I think adding `output_attentions=False`  and `output_hidden_states=False` may help:
```
model = TFGPT2LMHeadModel(GPT2Config(output_attentions=False, output_hidden_states=False, use_cache=False))
```

I haven't had the chance to try that. Maybe someone else can? Will the trained network with `output_hidden_states=True` though? You need to set it to True when generating text.

Hello,

I got the same error.
When I tried to set `output_attentions = False`, it didn't seem to be in GPT2Config.
Looking at the [documentation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2config), GPT2Config doesn't have that parameter, where should I set it?

> Hello,
> 
> I got the same error.
> When I tried to set `output_attentions = False`, it didn't seem to be in GPT2Config.
> Looking at the [documentation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2config), GPT2Config doesn't have that parameter, where should I set it?

`output_attentions` is a Parameter of `PretrainedConfig` which is the SuperClass of `GPT2Config`, [see](https://huggingface.co/transformers/main_classes/configuration.html#transformers.PretrainedConfig) 

Thank you for your reply.

> PretrainedConfig which is the SuperClass of GPT2Config

I see, I understand.
I noticed that the version of the transformer I'm using was out of date :cry: 

> GPT2Config(output_attentions=False, output_hidden_states=False, use_cache=False)

I used the latest version and it worked without any errors!
Your advice was very helpful, thank you!

This issue has been stale for 1 month.