sequences and scores dimensions are mismatch when using generate()
Author: GasolSun36
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

generation_config = GenerationConfig(
            temperature=0.7,
            top_p=1,
            top_k=40,
            num_beams=1,
            do_sample = True,
            num_return_sequences = 8,
            max_length = 109,
            return_dict_in_generate = True,
            output_scores = True
        )
        generator_outputs = generator_model.generate(
            inputs["input_ids"],
            attention_mask=inputs["attention_mask"],
            generation_config=generation_config,
            synced_gpus=True
        ) # (B*num_candidates, L)

        generated_tokens = generator_outputs.sequences # (B*num_candidates, L+1)
        
        generated_logits = torch.stack(generator_outputs.scores, dim=1)
        generated_seqs = self.generator_tokenizer.batch_decode(
                    generated_tokens, skip_special_tokens=True, clean_up_tokenization_spaces=True
                )
        # get the probability of generated tokens
        seq_len = generated_logits.size(1)
        vocab_size = generated_logits.size(-1)

        generated_probs = nn.functional.softmax(generated_logits, dim=-1)  
        new_generated_probs = generated_probs.contiguous().view(-1, vocab_size)
        generated_tokens_indices = generated_tokens.contiguous().view(-1).unsqueeze(1) 

        new_generated_probs = torch.gather(new_generated_probs, 1, generated_tokens_indices)


### Expected behavior

when i run the code to `new_generated_probs = torch.gather(new_generated_probs, 1, generated_tokens_indices)`, the error arise:

`RuntimeErrorRuntimeError: : Size does not match at dimension 0 expected index [1408, 1] to be smaller than self [1232, 250680] apart from dimension 1Size does not match at dimension 0 expected index [1968, 1] to be smaller than self [1744, 250680] apart from dimension 1`

I look carefully about the generate method, looks like the dimension of generator_outputs.sequences are different of generator_outputs.scores, cause in `transformers/src/transformers/generation/utils.py` 1363 lines:
`generation_config.max_length = generation_config.max_new_tokens + input_ids_seq_length`

however in `class SampleDecoderOnlyOutput`, it says: 
`sequences (torch.LongTensor of shape (batch_size*num_return_sequences, sequence_length)):
            The generated sequences. The second dimension (sequence_length) is either equal to max_length or shorter
            if all batches finished early due to the eos_token_id.`

and 
`scores (tuple(torch.FloatTensor) *optional*, returned when output_scores=True is passed or when config.output_scores=True):
            Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax)
            at each generation step. Tuple of torch.FloatTensor with up to max_new_tokens elements (one element for
            each generated token), with each tensor of shape (batch_size*num_return_sequences, config.vocab_size)`.
`

that means, the length of sequences is equal to max_length, the scores is equal to max_new_tokens. The difference between the two values is `input_ids_seq_length`.

However, I can't use max_length and max_new_tokens together cause the max_new_tokens are more priority.

Is there anyway to deal with it?
I use my own dataset on Bloom. But I think this problem you can use any model on any dataset to reproduct it.

Comments:
btw i think it's probably a mistake cause by `generation/utils.py` in `sample()` function, that when calculate the scores, it stops early. cause I look up my tensor dimension, it's always differ 208 dimensions (the sequences is 208 longer than the scores), that's equal to 208/8=26, which is my `input_ids_seq_length`

the sequences should be equal to scores, right?

Hi there, I am also having some issues with the shapes between sequences and scores. In my case, I am finding the length of the scores tuple is longer than the sequence length? Any idea why this would be? Seems like it's the opposite of your issue

Hey @GasolSun36 @aburns4 👋 

The behavior you see is what is expected. The docstrings are a bit outdated and in need of a retouch 🤗 In a nutshell:
1. In `generate`, the scores are exclusively related to new tokens (tokens not in the prompt)
2. The output sequence for decoder-only models (like BLOOM) includes the prompt as well
3. If you want to obtain the logits for the prompt tokens, you should do run a model forward pass with your prompt (see below). Please note that the logits always refer to the next token, so the logits with index 0 correspond to the token with index 1.
4. If you want to get the logits for the whole sequence (prompt + generated tokens), you have to concatenate these two sets of logits :)

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")

inputs = tokenizer(["The quick brown"], return_tensors="pt")
logits = model(**inputs).logits
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I have the same error 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.