Official Example - BeamSearchScorer always returns just one beam, even I specified 3
Author: Oxi84
Pull request: no

### System Info

google colab latest

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

    from transformers import (
         AutoTokenizer,
         AutoModelForSeq2SeqLM,
         LogitsProcessorList,
         MinLengthLogitsProcessor,
         BeamSearchScorer,
    )
    import torch
    
    tokenizer = AutoTokenizer.from_pretrained("t5-base")
    model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
    
    encoder_input_str = "translate English to German: How old are you?"
    encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
    
    # lets run beam search using 3 beams
    num_beams = 3
    # define decoder start token ids
    input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
    input_ids = input_ids * model.config.decoder_start_token_id
    
    # add encoder_outputs to model keyword arguments
    model_kwargs = {
         "encoder_outputs": model.get_encoder()(
             encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
         )
     }
    
    # instantiate beam scorer
    beam_scorer = BeamSearchScorer(
         batch_size=1,
         num_beams=num_beams,
         device=model.device,
     )
    
    # instantiate logits processors
    logits_processor = LogitsProcessorList(
         [
             MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
         ]
     )
    
    outputs = model.beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs)
    
    out = tokenizer.batch_decode(outputs, skip_special_tokens=True)
    
    print("out",out)
    

### Expected behavior

out is just one line, even when i put:

    num_return_sequences=2

at various places, for example:

    outputs = model.beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs, num_return_sequences=2)

    



Comments:
@ArthurZucker  @younesbelkada @Narsil

Hi @Oxi84, thanks for raising this issue!

Could you provide information about the environment packages? Just run `! transformers-cli env` in a colab cell and copy paste the output.  

cc @gante 

Sure:

- `transformers` version: 4.30.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


I have found the solution actually:

   just add:
  
   num_beam_hyps_to_keep = 5

   or number you want, then scorer just chooses 5 best hypothesis.

    beam_scorer = BeamSearchScorer(
        batch_size=1,
        do_early_stopping=True,
        num_beams=num_beams,
        num_beam_hyps_to_keep = 5,
       device=model.device
   )

I only need to make this use multiple beams. With just one beam it is too slow.

This can be done by using this:

    input_ids = torch.ones((num_beams*the_batch_size, 1), device=model.device, dtype=torch.long)

I guess something related to encoder/decoder

Hey @Oxi84 👋 

The `.generate()` method does a lot of preprocessing, input preparation, and instance initialization for you. I'd recommend using it, as we don't have the bandwidth to provide usage support for lower-level APIs :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.