Sample method doesn't work for mt5 architecture
Author: tatiana-iazykova
Pull request: no

### System Info

- `transformers` version: 4.21.3
- Platform: Linux-5.4.0-122-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

MT5 sample method @patrickvonplaten

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The following code (from https://huggingface.co/docs/transformers/v4.22.2/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.sample)

```python
from transformers import (
    MT5Tokenizer,
    MT5ForConditionalGeneration,
    LogitsProcessorList,
    MinLengthLogitsProcessor,
    TopKLogitsWarper,
    TemperatureLogitsWarper,
    StoppingCriteriaList,
    MaxLengthCriteria,
)
import torch

tokenizer = MT5Tokenizer.from_pretrained("google/mt5-base")
model = MT5ForConditionalGeneration.from_pretrained("google/mt5-base")

# set pad_token_id to eos_token_id because GPT2 does not have a EOS token

input_prompt = "Today is a beautiful day, and"
input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids

# instantiate logits processors
logits_processor = LogitsProcessorList(
    [
        MinLengthLogitsProcessor(15, eos_token_id=model.config.eos_token_id),
    ]
)
# instantiate logits processors
logits_warper = LogitsProcessorList(
    [
        TopKLogitsWarper(50),
        TemperatureLogitsWarper(0.7),
    ]
)

stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=20)])

torch.manual_seed(0)
outputs = model.sample(
    input_ids,
    logits_processor=logits_processor,
    logits_warper=logits_warper,
    stopping_criteria=stopping_criteria,
)

tokenizer.batch_decode(outputs, skip_special_tokens=True)
```

results in `ValueError: You have to specify either input_ids or inputs_embeds`

Tried to explicitly set decoder_input_ids but it didn't help

### Expected behavior

Tried to use sample method instead of generate and failed to get rid of value error

Comments:
moreover, if I try to use generate with num_beams=1 and do_sample=True as it says in documentation, I got really weird scores:


inp = batch # result of next(item(torch.utils.Dataloader))

generate_results = model.generate(
                input_ids=inp['input_ids'].to(model.device),
                attention_mask=inp['attention_mask'].to(model.device),
                output_scores=True,
                num_beams=1,
                do_sample=True,
                return_dict_in_generate=True,
            )

```
SampleEncoderDecoderOutput(sequences=tensor([[     0,    259,    264,  66689,  31018,   2793,  79143,   1334,    259,
           3205,   1264,   1285,   2776, 179124, 192823, 149529,  13647,    260,
              1,      0],
        [     0,    563,   3392,    259,  37446,   4205,  59633,    299,  19966,
            259,  20364,    484,    261,    259,  21230,    332,    287,   9983,
           9844,    260],
        [     0,  92868, 111621,   1498,    543,  68093,    259,  23886,    446,
          22708,    261,    259,   3266,    259,  31989,    411,   4338,   3695,
          18203,    388],
        [     0,   2553,   1049,   7584,   8608,   3671,    261,    892,   6888,
           3647,   7077,    456,  23480,    729,  23761,   1128,    260,      1,
              0,      0],
        [     0,   4343,    729,  79829,    261,   3553,  12799,    261,    259,
            279,  33658,    433,  30643,    308, 116992,  34741,    259,  95298,
            388,  10081],
        [     0,    259,  73394,    304,    459,  54650,    261,   1361,  24856,
            730, 169977,      1,      0,      0,      0,      0,      0,      0,
              0,      0],
        [     0,  25870,    261,    259, 109102,    261,   2477,    277,    270,
           3256,    416,    259, 185416,    521,    260,    260,   4837,    609,
            277,    263],
        [     0,    259,  74815,    688,  13986,    657,    425,    259,  46805,
           7378,    259,  30821,    877,    816,    267,      1,      0,      0,
              0,      0],
        [     0,    259,    279,    259,  74725,   2266,    259,  71145,    748,
            274,  24186,    261,  37828,    324,      1,      0,      0,      0,
              0,      0],
        [     0,    259, 176741,  11966,    425,   8790,   1344,  43078,    543,
            259,    279,    259,  98503,   1400,    259,  30821,    274,  83935,
           1633,  13637],
        [     0,    486,  10753,    263,   1432,    344,   1537,   1459,  27906,
           1537,   2985,    261,   1866,    569,   6535,    339,    259,  45628,
            281,    287],
        [     0,    336,   3031,    272,    277,    270,   1689,   4065,    288,
            342,    714,    260,      1,      0,      0,      0,      0,      0,
              0,      0]], device='cuda:0'), scores=(tensor([[-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        ...,
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf]], device='cuda:0'), tensor([[-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        ...,
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf]], device='cuda:0'), tensor([[-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        ...,
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf]], device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf, -4.1323,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -6.2682,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [-0.6975,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        ...,
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf],
        [-inf, -inf, -inf,  ..., -inf, -inf, -inf]], device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -4.8971,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf, -7.8711,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -3.7007,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [-0.7802,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -6.0860,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -3.4524,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -6.1068,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf, -6.8437,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -0.4513,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -4.2931,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,  7.0613,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -8.3437,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [14.4250,  1.6957,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf, -8.2820,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [15.0611,    -inf,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [15.2473,    -inf,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf, -0.5828,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [15.2603,  2.0967,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf, -2.9308,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [14.6447,  1.9284,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[   -inf,  4.6353,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -8.9710,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf,    -inf,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -6.0908,    -inf,  ...,    -inf,    -inf,    -inf],
        [14.7882,  1.9284,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0'), tensor([[15.3742,  2.2133,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -0.5673,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -3.3424,    -inf,  ...,    -inf,    -inf,    -inf],
        ...,
        [   -inf, -2.2742,    -inf,  ...,    -inf,    -inf,    -inf],
        [   -inf, -6.5111,    -inf,  ...,    -inf,    -inf,    -inf],
        [15.1830,  2.0473,    -inf,  ...,    -inf,    -inf,    -inf]],
       device='cuda:0')), encoder_attentions=None, encoder_hidden_states=None, decoder_attentions=None, cross_attentions=None, decoder_hidden_states=None)
```

@gante or @ArthurZucker could you take this one? :-) 

Hi @tatiana-iazykova 👋 

Regarding the issue in the first post: the problem is that the docstring example (that you based your script on) only works for decoder-only models, and an encoder-decoder model is used. For encoder-decoder models, the input needs some additional processing, that `generate()` gracefully handles [here](https://github.com/huggingface/transformers/blob/f4ef78af543a166551889da8737cc3134a7d9dd3/src/transformers/generation_utils.py#L1281). While we *could* make `model.sample()` and other generation strategies handle this sort of cases, the code will quickly become a pain to maintain (input handling would have to be added everywhere), so I'm inclined not to make any attempt to fix and to redirect towards `model.generate()` instead :) (cc @patrickvonplaten -- maybe we should add a line in the docstrings to prioritize the use of `.generate()`?)

As for the second issue -- can you share a reproducible script? 🙏 The scores would look normal if a logits processor like top_k was used, but it doesn't seem to be the case 🤔 

Noted.

As for the second issue, the input was standard like the one specified here (https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb#scrollTo=kTCFado4IrIc)

and code for generate was the following:

```python

generate_results = model.generate(
  input_ids=inp['input_ids'].to(model.device),
  attention_mask=inp['attention_mask'].to(model.device),
  output_scores=True,
  num_beams=1,
  do_sample=True,
  return_dict_in_generate=True,
)
```

@tatiana-iazykova I see -- in the example you shared, `top_k` is not passed, so it inherits the [default value](https://huggingface.co/docs/transformers/v4.23.1/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.top_k) (50). `top_k` sets the log probability of all but the K most likely tokens in `-inf`, which explains the numbers you see :)

As a counter example, you can turn `top_k` off by setting it to the size of the vocabulary:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "distilgpt2"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

inputs = tokenizer("This is a simple test", return_tensors="pt")
generate_outputs = model.generate(
  input_ids=inputs['input_ids'],
  attention_mask=inputs['attention_mask'],
  output_scores=True,
  num_beams=1,
  do_sample=True,
  return_dict_in_generate=True,
  top_k=tokenizer.vocab_size
)
print(generate_outputs)
```

I'm closing the issue as this is intended behavior, but feel free to reopen with further queries :)