TFAutoModel does not work with gpt2 and .generate
Author: ehrencrona
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no

### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import TFAutoModel, AutoTokenizer

model = TFAutoModel.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

tokens = tokenizer(["hey there"], return_tensors='tf')

model.generate(input_ids=tokens['input_ids'], attention_mask=tokens['attention_mask'])
```

will return `AttributeError: 'TFBaseModelOutputWithPastAndCrossAttentions' object has no attribute 'logits'`

Full stack trace:

```
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-45-eb17c4f14b9f>](https://localhost:8080/#) in <module>()
----> 1 output = model.generate(input_ids=tokens['input_ids'], attention_mask=tokens['attention_mask'])
      2 output

3 frames
[/usr/local/lib/python3.7/dist-packages/transformers/generation_tf_utils.py](https://localhost:8080/#) in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, output_scores, output_attentions, output_hidden_states, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, **model_kwargs)
    594                 return_dict_in_generate=return_dict_in_generate,
    595                 forced_bos_token_id=forced_bos_token_id,
--> 596                 forced_eos_token_id=forced_eos_token_id,
    597             )
    598 

[/usr/local/lib/python3.7/dist-packages/transformers/generation_tf_utils.py](https://localhost:8080/#) in _generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, seed, output_scores, output_attentions, output_hidden_states, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, **model_kwargs)
   1589                 output_scores=output_scores,
   1590                 return_dict_in_generate=return_dict_in_generate,
-> 1591                 **model_kwargs,
   1592             )
   1593         elif is_sample_gen_mode:

[/usr/local/lib/python3.7/dist-packages/transformers/generation_tf_utils.py](https://localhost:8080/#) in greedy_search(self, input_ids, max_length, pad_token_id, eos_token_id, logits_processor, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, **model_kwargs)
   2082         # 1st generation step has to be run before to initialize `past`
   2083         generated, finished_sequences, next_tokens, cur_len, model_kwargs = greedy_search_body_fn(
-> 2084             generated, finished_sequences, input_ids, cur_len, model_kwargs
   2085         )
   2086 

[/usr/local/lib/python3.7/dist-packages/transformers/generation_tf_utils.py](https://localhost:8080/#) in greedy_search_body_fn(generated, finished_sequences, next_tokens, cur_len, model_kwargs)
   2025                 output_hidden_states=output_hidden_states,
   2026             )
-> 2027             next_token_logits = outputs.logits[:, -1]
   2028 
   2029             # Store scores, attentions and hidden_states when required

AttributeError: 'TFBaseModelOutputWithPastAndCrossAttentions' object has no attribute 'logits'
```

### Expected behavior

Not sure if you consider this to be a bug, but it is a stumbling block for beginners.

If you use `TFAutoModel` to load `gpt2` you will get a `TFGPT2Model`. This class has a `generate` method but it doesn't work (because it expects a linear layer to generate logits, i.e. it only works on `TFGPT2LMHeadModel`).

I'd argue that it's a bug because if `TFGPT2Model` doesn't support generation, then it shouldn't have a `generate` method.

Possible alternative fixes:

 * Throw an easier-to-understand error in this situation
 * Make `TFGPT2Model` not implement `generate`
 * Have `TFAutoModel` return a `TFGPT2LMHeadModel` (though this would be a breaking change)

Comments:
cc @gante as well

Hi @ehrencrona 👋 The correct class to use for generation with decoder-only models is `TFAutoModelForCausalLM`. You can use it the same way as `TFAutoModel` but, contrarily to it, it has a language modeling head.

As for the suggested fixes -- I agree `generate` should not exist here (or better yet, that the error should be informative, as new users might not know which class to use). I've added that to the list of generate goodies to add in the near future :) 

Thank you for flagging the issue and for the suggestions!