LogitsProcessor - are there any examples of how to use it?
Author: Oxi84
Pull request: no

### Feature request

I could not find any more examples than this one, and this one does not work. It reports some erors, when used with t5. https://colab.research.google.com/drive/1ezT24sogpVyr2HJLOvXHzjv61JZJ1gMT?usp=sharing#scrollTo=0MJJZEylVO-x

Are there any more examples of usage.

I need to lower the scores for a specified value, for chosen tokens, but any example will do.


### Motivation

We need some usage examples of every feature of transformers, otherwise these features are not very useful for 90 percent of people.

### Your contribution

I found this example: https://colab.research.google.com/drive/1ezT24sogpVyr2HJLOvXHzjv61JZJ1gMT?usp=sharing#scrollTo=0MJJZEylVO-x

Comments:
cc @gante 

I have found one more example for T5. Only I do not know how would I add multiple inputs, not just one, when i increase the list seq1 to more elements it does not work as sizes do not match.

https://stackoverflow.com/questions/72180737/beam-search-and-generate-are-not-consistent


    seq1 = [ "summarize: beamsearch and generate does not give the same result"]
    
    
    encoding = tokenizer(
        seq1,
        padding="longest",
        max_length=128,
        truncation=True,
        return_tensors="pt",
    )
    
    encoder_input_ids, attention_mask = encoding.input_ids.to("cuda"), encoding.attention_mask.to("cuda")
    num_beams = 2
    input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
    input_ids = input_ids * model.config.decoder_start_token_id
    model_kwargs = {
        "encoder_outputs": model.get_encoder()(
            encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
        )
    }
    beam_scorer = BeamSearchScorer(
        batch_size=1,
        do_early_stopping=True,
        num_beams=num_beams,
        device=model.device,
    )
    
    outputs = model.beam_search(input_ids, beam_scorer,
                                logits_processor=None,
                                early_stopping=True,
                                no_repeat_ngram_size=4,
                                max_length=64,
                                **model_kwargs,
                                output_scores=True,
                                return_dict_in_generate=True)
    
    # beam_search result":






@Oxi84 If there are errors in the examples, could you share the errors with a full stack trace and information about the environment being run? Keep in mind that examples are not meant to be exhaustive and there may be cases they don't cover. 

Here is an example on colab - https://colab.research.google.com/drive/1TR6PWwKK4SuD7RluN_f82lOZQi_SOJnf#scrollTo=Uj4YddOGq2ee

This is a basic example, where logits_processor=None from https://stackoverflow.com/questions/72180737/beam-search-and-generate-are-not-consistent.

When seq1 = ["paraphrase: Beamsearch and generate give the same result."] , it works fine, but when 

    seq1 = ["paraphrase: Beamsearch and generate give the same result.","paraphrase: Beamsearch and generate give the same result."] 

I do not know how to make it work.  
Error is: RuntimeError: The size of tensor a (28) must match the size of tensor b (14) at non-singleton dimension 3.




    import torch
    import torch.nn as nn
    import torch.optim as optim
    import torch.nn.functional as F
    from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
    from transformers import LogitsProcessorList, MinLengthLogitsProcessor, BeamSearchScorer,MaxLengthCriteria, StoppingCriteriaList
    
    
    from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
    tokenizer = AutoTokenizer.from_pretrained("humarin/chatgpt_paraphraser_on_T5_base")
    model = AutoModelForSeq2SeqLM.from_pretrained("humarin/chatgpt_paraphraser_on_T5_base")
    
    model.resize_token_embeddings(len(tokenizer))
    model.to("cuda")
    
    seq1 = ["paraphrase: Beamsearch and generate give the same result."]
    
    
    encoding = tokenizer(
        seq1,
        padding="longest",
        max_length=128,
        truncation=True,
        return_tensors="pt",
    )
    
    encoder_input_ids, attention_mask = encoding.input_ids.to("cuda"), encoding.attention_mask.to("cuda")
    num_beams = 2
    input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
    input_ids = input_ids * model.config.decoder_start_token_id
    model_kwargs = {
        "encoder_outputs": model.get_encoder()(
            encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
        )
    }
    
    #print("input_ids",input_ids)
    #print("model_kwargs",model_kwargs)
    
    beam_scorer = BeamSearchScorer(
        batch_size=1,
        do_early_stopping=True,
        num_beams=num_beams,
        device=model.device,
    )
    
    outputs = model.beam_search(input_ids, beam_scorer,
                                logits_processor=None,
                                early_stopping=True,
                                max_length=64,
                                **model_kwargs,
                                output_scores=True,
                                return_dict_in_generate=True)
    
    # beam_search result":
    out = tokenizer.batch_decode(outputs.sequences, skip_special_tokens=True)
    
    print("out",out)
    
    #generate results:
    out = model.generate(encoder_input_ids,
                                               max_length=64,
                                         
                                               early_stopping=True,
                                               num_beams=2,
                                               do_sample=False,
                                               num_return_sequences=1)
    
    out1 = tokenizer.batch_decode(out, skip_special_tokens=True)
    
    print("out1",out1)







Hey @Oxi84 👋 

You are absolutely right, our docs and examples for generation are quite poor atm. It is my highest priority for the next 1-2 months -- stay tuned 💪 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.