Documentation Clarification: Autoregressive Models using GenerationMixin
Author: JoeREISys
Pull request: no

In the documentation on HuggingFace and within the source code for autoregressive models in comments, it shows to use the `generate` method from the GenerationMixin. Here is an example in the code for Llama model.


```
def forward(
        self,
        input_ids: torch.LongTensor = None,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_values: Optional[List[torch.FloatTensor]] = None,
        inputs_embeds: Optional[torch.FloatTensor] = None,
        labels: Optional[torch.LongTensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, CausalLMOutputWithPast]:
        r"""
        Args:
            labels (torch.LongTensor of shape (batch_size, sequence_length), *optional*):
                Labels for computing the masked language modeling loss. Indices should either be in [0, ...,
                config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored
                (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].
        Returns:
        Example:

        ###python
        >>> from transformers import AutoTokenizer, LlamaForCausalLM

        >>> model = LlamaForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
        >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)

        >>> prompt = "Hey, are you conscious? Can you talk to me?"
        >>> inputs = tokenizer(prompt, return_tensors="pt")

        >>> # Generate
        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
        "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
        """
```


However, autoregressive models shouldn't need beam search as casual language modeling output should be able to directly decode the tokens. In tracing the model inheritance, there is no connection to `GenerationMixin` either to expose a generate method nor an implementation of the generate method for this models. What am I missing?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Review the documentation of autoregressive models
2. Review the source code of autoregressive models
3. Autoregressive models do not have `GenerationMixin` but have comment to use `.generate` method.

### Expected behavior

1. The source code of the models reflects the comments with `GenerationMixin` implementation or `.generate` method implementation or the comments and model documentation reflect how to use for inference if `GenerationMixin` is not used.

Comments:
I was confusing training vs inference. At training time, the model autoregressively trains by shifting the labels. It has the entire input and output allowing for cross entropy for each label prediction. However, at inference time, that autoregressive generation has to be coded. Where I'm not totally clear is where the code is that connects the models to the `GenerationMixin` for inference. It looks like that it should just work to call `.generate` from the models don't inherent from `GenerationMixin`.

Found it, the base class, `class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin, PushToHubMixin):` enables all classes to have access to generate method and they are guarded based on the config and `can_generate` which is based on whether or not it implements `prepare_inputs_for_generation`method (e.g. all models have `generate` method they can invoke, `generate` method will work if `prepare_inputs_for_generation` is implemented otherwise give an error). If only I can delete this issue.