Utilities for Generation

This page lists all the utility functions used by [generate()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.generate), [greedy\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.greedy_search), [contrastive\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.contrastive_search), [sample()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.sample), [beam\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.beam_search), [beam\_sample()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.beam_sample), [group\_beam\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.group_beam_search), and [constrained\_beam\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.constrained_beam_search).

Most of those are only useful if you are studying the code of the generate methods in the library.

## [](#generate-outputs)Generate Outputs

The output of [generate()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.generate) is an instance of a subclass of [ModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.utils.ModelOutput). This output is a data structure containing all the information returned by [generate()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.generate), but that can also be used as tuple or dictionary.

Here’s an example:

```
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

inputs = tokenizer("Hello, my dog is cute and ", return_tensors="pt")
generation_output = model.generate(**inputs, return_dict_in_generate=True, output_scores=True)```

The `generation_output` object is a [GreedySearchDecoderOnlyOutput](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.generation.GreedySearchDecoderOnlyOutput), as we can see in the documentation of that class below, it means it has the following attributes:

-   `sequences`: the generated sequences of tokens
-   `scores` (optional): the prediction scores of the language modelling head, for each generation step
-   `hidden_states` (optional): the hidden states of the model, for each generation step
-   `attentions` (optional): the attention weights of the model, for each generation step

Here we have the `scores` since we passed along `output_scores=True`, but we don’t have `hidden_states` and `attentions` because we didn’t pass `output_hidden_states=True` or `output_attentions=True`.

You can access each attribute as you would usually do, and if that attribute has not been returned by the model, you will get `None`. Here for instance `generation_output.scores` are all the generated prediction scores of the language modeling head, and `generation_output.attentions` is `None`.

When using our `generation_output` object as a tuple, it only keeps the attributes that don’t have `None` values. Here, for instance, it has two elements, `loss` then `logits`, so

will return the tuple `(generation_output.sequences, generation_output.scores)` for instance.

When using our `generation_output` object as a dictionary, it only keeps the attributes that don’t have `None` values. Here, for instance, it has two keys that are `sequences` and `scores`.

We document here all output types.

### [](#transformers.generation.GreedySearchDecoderOnlyOutput)GreedySearchOutput

### class transformers.generation.GreedySearchDecoderOnlyOutput

[](#transformers.generation.GreedySearchDecoderOnlyOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L83)

( sequences: LongTensor = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneattentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonehidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of decoder-only generation models using greedy search.

### class transformers.generation.GreedySearchEncoderDecoderOutput

[](#transformers.generation.GreedySearchEncoderDecoderOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L179)

( sequences: LongTensor = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneencoder\_attentions: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneencoder\_hidden\_states: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonedecoder\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonecross\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonedecoder\_hidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of encoder-decoder generation models using greedy search. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder\_attentions and the encoder\_hidden\_states attributes (respectively the decoder\_attentions and the decoder\_hidden\_states attributes)

### class transformers.generation.FlaxGreedySearchOutput

[](#transformers.generation.FlaxGreedySearchOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_utils.py#L55)

( sequences: ndarray = None )

Parameters

-   [](#transformers.generation.FlaxGreedySearchOutput.sequences)**sequences** (`jnp.ndarray` of shape `(batch_size, max_length)`) — The generated sequences.

Flax Base class for outputs of decoder-only generation models using greedy search.

“Returns a new object replacing the specified fields with new values.

### [](#transformers.generation.SampleDecoderOnlyOutput)SampleOutput

### class transformers.generation.SampleDecoderOnlyOutput

[](#transformers.generation.SampleDecoderOnlyOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L221)

( sequences: LongTensor = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneattentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonehidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of decoder-only generation models using sampling.

### class transformers.generation.SampleEncoderDecoderOutput

[](#transformers.generation.SampleEncoderDecoderOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L250)

( sequences: LongTensor = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneencoder\_attentions: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneencoder\_hidden\_states: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonedecoder\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonecross\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonedecoder\_hidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of encoder-decoder generation models using sampling. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder\_attentions and the encoder\_hidden\_states attributes (respectively the decoder\_attentions and the decoder\_hidden\_states attributes)

### class transformers.generation.FlaxSampleOutput

[](#transformers.generation.FlaxSampleOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_utils.py#L69)

( sequences: ndarray = None )

Parameters

-   [](#transformers.generation.FlaxSampleOutput.sequences)**sequences** (`jnp.ndarray` of shape `(batch_size, max_length)`) — The generated sequences.

Flax Base class for outputs of decoder-only generation models using sampling.

“Returns a new object replacing the specified fields with new values.

### [](#transformers.generation.BeamSearchDecoderOnlyOutput)BeamSearchOutput

### class transformers.generation.BeamSearchDecoderOnlyOutput

[](#transformers.generation.BeamSearchDecoderOnlyOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L293)

( sequences: LongTensor = Nonesequences\_scores: typing.Optional\[torch.FloatTensor\] = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonebeam\_indices: typing.Optional\[torch.LongTensor\] = Noneattentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonehidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of decoder-only generation models using beam search.

### class transformers.generation.BeamSearchEncoderDecoderOutput

[](#transformers.generation.BeamSearchEncoderDecoderOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L328)

( sequences: LongTensor = Nonesequences\_scores: typing.Optional\[torch.FloatTensor\] = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonebeam\_indices: typing.Optional\[torch.LongTensor\] = Noneencoder\_attentions: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneencoder\_hidden\_states: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonedecoder\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonecross\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonedecoder\_hidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of encoder-decoder generation models using beam search. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder\_attentions and the encoder\_hidden\_states attributes (respectively the decoder\_attentions and the decoder\_hidden\_states attributes)

### [](#transformers.generation.BeamSampleDecoderOnlyOutput)BeamSampleOutput

### class transformers.generation.BeamSampleDecoderOnlyOutput

[](#transformers.generation.BeamSampleDecoderOnlyOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L378)

( sequences: LongTensor = Nonesequences\_scores: typing.Optional\[torch.FloatTensor\] = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonebeam\_indices: typing.Optional\[torch.LongTensor\] = Noneattentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonehidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of decoder-only generation models using beam sample.

### class transformers.generation.BeamSampleEncoderDecoderOutput

[](#transformers.generation.BeamSampleEncoderDecoderOutput)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L413)

( sequences: LongTensor = Nonesequences\_scores: typing.Optional\[torch.FloatTensor\] = Nonescores: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonebeam\_indices: typing.Optional\[torch.LongTensor\] = Noneencoder\_attentions: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneencoder\_hidden\_states: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonedecoder\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonecross\_attentions: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = Nonedecoder\_hidden\_states: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.FloatTensor\]\]\] = None )

Base class for outputs of encoder-decoder generation models using beam sampling. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder\_attentions and the encoder\_hidden\_states attributes (respectively the decoder\_attentions and the decoder\_hidden\_states attributes)

## [](#transformers.LogitsProcessor)LogitsProcessor

A [LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) can be used to modify the prediction scores of a language model head for generation.

Abstract base class for all logit processors that can be applied during generation.

#### \_\_call\_\_

[](#transformers.LogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L54)

( input\_ids: LongTensorscores: FloatTensor ) → `torch.FloatTensor` of shape `(batch_size, config.vocab_size)`

Parameters

-   [](#transformers.LogitsProcessor.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.LogitsProcessor.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs — Additional logits processor specific kwargs.

Returns

`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`

The processed prediction scores.

Torch method for processing logits.

### class transformers.LogitsProcessorList

[](#transformers.LogitsProcessorList)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L73)

( iterable = () )

This class can be used to create a list of [LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) or [LogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsWarper) to subsequently process a `scores` input tensor. This class inherits from list and adds a specific _**call**_ method to apply each [LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) or [LogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsWarper) to the inputs.

#### \_\_call\_\_

[](#transformers.LogitsProcessorList.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L80)

( input\_ids: LongTensorscores: FloatTensor\*\*kwargs ) → `torch.FloatTensor` of shape `(batch_size, config.vocab_size)`

Parameters

-   [](#transformers.LogitsProcessorList.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.LogitsProcessorList.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs — Additional logits processor specific kwargs.

Returns

`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`

The processed prediction scores.

Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.

#### \_\_call\_\_

[](#transformers.LogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L65)

( input\_ids: LongTensorscores: FloatTensor ) → `torch.FloatTensor` of shape `(batch_size, config.vocab_size)`

Parameters

-   [](#transformers.LogitsWarper.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.LogitsWarper.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs — Additional logits processor specific kwargs.

Returns

`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`

The processed prediction scores.

Torch method for warping logits.

### class transformers.MinLengthLogitsProcessor

[](#transformers.MinLengthLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L96)

( min\_length: inteos\_token\_id: typing.Union\[int, typing.List\[int\]\] )

Parameters

-   [](#transformers.MinLengthLogitsProcessor.min_length)**min\_length** (`int`) — The minimum length below which the score of `eos_token_id` is set to `-float("Inf")`.
-   [](#transformers.MinLengthLogitsProcessor.eos_token_id)**eos\_token\_id** (`Union[int, List[int]]`) — The id of the _end-of-sequence_ token. Optionally, use a list to set multiple _end-of-sequence_ tokens.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) enforcing a min-length by setting EOS probability to 0.

#### \_\_call\_\_

[](#transformers.MinLengthLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L119)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.MinNewTokensLengthLogitsProcessor

[](#transformers.MinNewTokensLengthLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L127)

( prompt\_length\_to\_skip: intmin\_new\_tokens: inteos\_token\_id: typing.Union\[int, typing.List\[int\]\] )

Parameters

-   [](#transformers.MinNewTokensLengthLogitsProcessor.prompt_length_to_skip)**prompt\_length\_to\_skip** (`int`) — The input tokens length.
-   [](#transformers.MinNewTokensLengthLogitsProcessor.min_new_tokens)**min\_new\_tokens** (`int`) — The minimum _new_ tokens length below which the score of `eos_token_id` is set to `-float("Inf")`.
-   [](#transformers.MinNewTokensLengthLogitsProcessor.eos_token_id)**eos\_token\_id** (`Union[int, List[int]]`) — The id of the _end-of-sequence_ token. Optionally, use a list to set multiple _end-of-sequence_ tokens.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) enforcing a min-length of new tokens by setting EOS (End-Of-Sequence) token probability to 0.

#### \_\_call\_\_

[](#transformers.MinNewTokensLengthLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L157)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.TemperatureLogitsWarper

[](#transformers.TemperatureLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L166)

( temperature: float )

Parameters

-   [](#transformers.TemperatureLogitsWarper.temperature)**temperature** (`float`) — The value used to module the logits distribution.

[LogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsWarper) for temperature (exponential scaling output probability distribution).

#### \_\_call\_\_

[](#transformers.TemperatureLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L181)

( input\_ids: Tensorscores: Tensor )

### class transformers.RepetitionPenaltyLogitsProcessor

[](#transformers.RepetitionPenaltyLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L186)

( penalty: float )

Parameters

-   [](#transformers.RepetitionPenaltyLogitsProcessor.repetition_penalty)**repetition\_penalty** (`float`) — The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) enforcing an exponential penalty on repeated sequences.

#### \_\_call\_\_

[](#transformers.RepetitionPenaltyLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L202)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.TopPLogitsWarper

[](#transformers.TopPLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L240)

( top\_p: floatfilter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.TopPLogitsWarper.top_p)**top\_p** (`float`) — If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation.
-   [](#transformers.TopPLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.TopPLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[LogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsWarper) that performs top-p, i.e. restricting to top tokens summing to prob\_cut\_off <= prob\_cut\_off.

#### \_\_call\_\_

[](#transformers.TopPLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L263)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.TopKLogitsWarper

[](#transformers.TopKLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L279)

( top\_k: intfilter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.TopKLogitsWarper.top_k)**top\_k** (`int`) — The number of highest probability vocabulary tokens to keep for top-k-filtering.
-   [](#transformers.TopKLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.TopKLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[LogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsWarper) that performs top-k, i.e. restricting to the k highest probability elements.

#### \_\_call\_\_

[](#transformers.TopKLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L299)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.TypicalLogitsWarper

[](#transformers.TypicalLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L307)

( mass: float = 0.9filter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.TypicalLogitsWarper.mass)**mass** (`float`) — Value of typical\_p between 0 and 1 inclusive, defaults to 0.9.
-   [](#transformers.TypicalLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.TypicalLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[LogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsWarper) that performs typical decoding. See [Typical Decoding for Natural Language Generation](https://arxiv.org/abs/2202.00666) for more information.

#### \_\_call\_\_

[](#transformers.TypicalLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L330)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.NoRepeatNGramLogitsProcessor

[](#transformers.NoRepeatNGramLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L474)

( ngram\_size: int )

Parameters

-   [](#transformers.NoRepeatNGramLogitsProcessor.ngram_size)**ngram\_size** (`int`) — All ngrams of size `ngram_size` can only occur once.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that enforces no repetition of n-grams. See [Fairseq](https://github.com/pytorch/fairseq/blob/a07cb6f40480928c9e0548b737aadd36ee66ac76/fairseq/sequence_generator.py#L345).

#### \_\_call\_\_

[](#transformers.NoRepeatNGramLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L489)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.NoBadWordsLogitsProcessor

[](#transformers.NoBadWordsLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L541)

( bad\_words\_ids: typing.List\[typing.List\[int\]\]eos\_token\_id: typing.Union\[int, typing.List\[int\]\] )

Parameters

-   [](#transformers.NoBadWordsLogitsProcessor.bad_words_ids)**bad\_words\_ids** (`List[List[int]]`) — List of list of token ids that are not allowed to be generated. In order to get the token ids of the words that should not appear in the generated text, use `tokenizer(bad_words, add_prefix_space=True, add_special_tokens=False).input_ids`.
-   [](#transformers.NoBadWordsLogitsProcessor.eos_token_id)**eos\_token\_id** (`Union[int, List[int]]`) — The id of the _end-of-sequence_ token. Optionally, use a list to set multiple _end-of-sequence_ tokens.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that enforces that specified sequences will never be sampled.

#### \_\_call\_\_

[](#transformers.NoBadWordsLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L589)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.PrefixConstrainedLogitsProcessor

[](#transformers.PrefixConstrainedLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L675)

( prefix\_allowed\_tokens\_fn: typing.Callable\[\[int, torch.Tensor\], typing.List\[int\]\]num\_beams: int )

Parameters

-   [](#transformers.PrefixConstrainedLogitsProcessor.prefix_allowed_tokens_fn)**prefix\_allowed\_tokens\_fn** (`Callable[[int, torch.Tensor], List[int]]`) — This function constraints the beam search to allowed tokens only at each step. This function takes 2 arguments `inputs_ids` and the batch ID `batch_id`. It has to return a list with the allowed tokens for the next generation step conditioned on the previously generated tokens `inputs_ids` and the batch ID `batch_id`.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that enforces constrained generation and is useful for prefix-conditioned constrained generation. See [Autoregressive Entity Retrieval](https://arxiv.org/abs/2010.00904) for more information.

#### \_\_call\_\_

[](#transformers.PrefixConstrainedLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L692)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.HammingDiversityLogitsProcessor

[](#transformers.HammingDiversityLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L701)

( diversity\_penalty: floatnum\_beams: intnum\_beam\_groups: int )

Parameters

-   [](#transformers.HammingDiversityLogitsProcessor.diversity_penalty)**diversity\_penalty** (`float`) — This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note that `diversity_penalty` is only effective if `group beam search` is enabled.
-   [](#transformers.HammingDiversityLogitsProcessor.num_beams)**num\_beams** (`int`) — Number of beams used for group beam search. See [this paper](https://arxiv.org/pdf/1610.02424.pdf) for more details.
-   [](#transformers.HammingDiversityLogitsProcessor.num_beam_groups)**num\_beam\_groups** (`int`) — Number of groups to divide `num_beams` into in order to ensure diversity among different groups of beams. See [this paper](https://arxiv.org/pdf/1610.02424.pdf) for more details.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that enforces diverse beam search. Note that this logits processor is only effective for [PreTrainedModel.group\_beam\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.group_beam_search). See [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf) for more details.

#### \_\_call\_\_

[](#transformers.HammingDiversityLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L732)

( input\_ids: LongTensorscores: FloatTensorcurrent\_tokens: LongTensorbeam\_group\_idx: int )

### class transformers.ForcedBOSTokenLogitsProcessor

[](#transformers.ForcedBOSTokenLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L761)

( bos\_token\_id: int )

Parameters

-   [](#transformers.ForcedBOSTokenLogitsProcessor.bos_token_id)**bos\_token\_id** (`int`) — The id of the token to force as the first generated token.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that enforces the specified token as the first generated token.

#### \_\_call\_\_

[](#transformers.ForcedBOSTokenLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L773)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.ForcedEOSTokenLogitsProcessor

[](#transformers.ForcedEOSTokenLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L782)

( max\_length: inteos\_token\_id: typing.Union\[int, typing.List\[int\]\] )

Parameters

-   [](#transformers.ForcedEOSTokenLogitsProcessor.max_length)**max\_length** (`int`) — The maximum length of the sequence to be generated.
-   [](#transformers.ForcedEOSTokenLogitsProcessor.eos_token_id)**eos\_token\_id** (`Union[int, List[int]]`) — The id of the token to force as the last generated token when `max_length` is reached. Optionally, use a list to set multiple _end-of-sequence_ tokens.

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that enforces the specified token as the last generated token when `max_length` is reached.

#### \_\_call\_\_

[](#transformers.ForcedEOSTokenLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L800)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.InfNanRemoveLogitsProcessor

[](#transformers.InfNanRemoveLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L810)

( )

[LogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.LogitsProcessor) that removes all `nan` and `inf` values to avoid the generation method to fail. Note that using the logits processor should only be used if necessary since it can slow down the generation method.

#### \_\_call\_\_

[](#transformers.InfNanRemoveLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/logits_process.py#L816)

( input\_ids: LongTensorscores: FloatTensor )

### class transformers.TFLogitsProcessor

[](#transformers.TFLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L53)

( )

Abstract base class for all logit processors that can be applied during generation.

#### \_\_call\_\_

[](#transformers.TFLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L56)

( input\_ids: Tensorscores: Tensorcur\_len: int ) → `tf.Tensor` of shape `(batch_size, config.vocab_size)`

TF method for processing logits.

### class transformers.TFLogitsProcessorList

[](#transformers.TFLogitsProcessorList)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L75)

( iterable = () )

This class can be used to create a list of [TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) to subsequently process a `scores` input tensor. This class inherits from list and adds a specific _**call**_ method to apply each [TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) to the inputs.

#### \_\_call\_\_

[](#transformers.TFLogitsProcessorList.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L82)

( input\_ids: Tensorscores: Tensorcur\_len: int\*\*kwargs ) → `tf.Tensor` of shape `(batch_size, config.vocab_size)`

Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.

#### \_\_call\_\_

[](#transformers.TFLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L67)

( input\_ids: Tensorscores: Tensorcur\_len: int ) → `tf.Tensor` of shape `(batch_size, config.vocab_size)`

TF method for warping logits.

### class transformers.TFTemperatureLogitsWarper

[](#transformers.TFTemperatureLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L98)

( temperature: float )

Parameters

-   [](#transformers.TFTemperatureLogitsWarper.temperature)**temperature** (`float`) — The value used to module the logits distribution.

[TFLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsWarper) for temperature (exponential scaling output probability distribution).

#### \_\_call\_\_

[](#transformers.TFTemperatureLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L113)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFTopPLogitsWarper

[](#transformers.TFTopPLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L146)

( top\_p: floatfilter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.TFTopPLogitsWarper.top_p)**top\_p** (`float`) — If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation.
-   [](#transformers.TFTopPLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.TFTopPLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[TFLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsWarper) that performs top-p, i.e. restricting to top tokens summing to <= prob\_cut\_off.

#### \_\_call\_\_

[](#transformers.TFTopPLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L168)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFTopKLogitsWarper

[](#transformers.TFTopKLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L118)

( top\_k: intfilter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.TFTopKLogitsWarper.top_k)**top\_k** (`int`) — The number of highest probability vocabulary tokens to keep for top-k-filtering.
-   [](#transformers.TFTopKLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.TFTopKLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[TFLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsWarper) that performs top-k, i.e. restricting to the k highest probability elements.

#### \_\_call\_\_

[](#transformers.TFTopKLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L138)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFMinLengthLogitsProcessor

[](#transformers.TFMinLengthLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L200)

( min\_length: inteos\_token\_id: int )

Parameters

-   [](#transformers.TFMinLengthLogitsProcessor.min_length)**min\_length** (`int`) — The minimum length below which the score of `eos_token_id` is set to `-float("Inf")`.
-   [](#transformers.TFMinLengthLogitsProcessor.eos_token_id)**eos\_token\_id** (`int`) — The id of the _end-of-sequence_ token.

[TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) enforcing a min-length by setting EOS probability to 0.

#### \_\_call\_\_

[](#transformers.TFMinLengthLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L226)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFNoBadWordsLogitsProcessor

[](#transformers.TFNoBadWordsLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L286)

( bad\_words\_ids: typing.List\[typing.List\[int\]\]eos\_token\_id: int )

Parameters

-   [](#transformers.TFNoBadWordsLogitsProcessor.bad_words_ids)**bad\_words\_ids** (`List[List[int]]`) — List of list of token ids that are not allowed to be generated. In order to get the tokens of the words that should not appear in the generated text, use `tokenizer(bad_word, add_prefix_space=True).input_ids`.
-   [](#transformers.TFNoBadWordsLogitsProcessor.eos_token_id)**eos\_token\_id** (`int`) — The id of the _end-of-sequence_ token.

[TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) that enforces that specified sequences will never be sampled.

#### \_\_call\_\_

[](#transformers.TFNoBadWordsLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L362)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFNoRepeatNGramLogitsProcessor

[](#transformers.TFNoRepeatNGramLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L383)

( ngram\_size: int )

Parameters

-   [](#transformers.TFNoRepeatNGramLogitsProcessor.ngram_size)**ngram\_size** (`int`) — All ngrams of size `ngram_size` can only occur once.

[TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) that enforces no repetition of n-grams. See [Fairseq](https://github.com/pytorch/fairseq/blob/a07cb6f40480928c9e0548b737aadd36ee66ac76/fairseq/sequence_generator.py#L345).

#### \_\_call\_\_

[](#transformers.TFNoRepeatNGramLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L422)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFRepetitionPenaltyLogitsProcessor

[](#transformers.TFRepetitionPenaltyLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L236)

( penalty: float )

Parameters

-   [](#transformers.TFRepetitionPenaltyLogitsProcessor.repetition_penalty)**repetition\_penalty** (`float`) — The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.

[TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) enforcing an exponential penalty on repeated sequences.

#### \_\_call\_\_

[](#transformers.TFRepetitionPenaltyLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L278)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFForcedBOSTokenLogitsProcessor

[](#transformers.TFForcedBOSTokenLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L443)

( bos\_token\_id: int )

Parameters

-   [](#transformers.TFForcedBOSTokenLogitsProcessor.bos_token_id)**bos\_token\_id** (`int`) — The id of the token to force as the first generated token.

[TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) that enforces the specified token as the first generated token.

#### \_\_call\_\_

[](#transformers.TFForcedBOSTokenLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L457)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.TFForcedEOSTokenLogitsProcessor

[](#transformers.TFForcedEOSTokenLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L473)

( max\_length: inteos\_token\_id: int )

Parameters

-   [](#transformers.TFForcedEOSTokenLogitsProcessor.max_length)**max\_length** (`int`) — The maximum length of the sequence to be generated.
-   [](#transformers.TFForcedEOSTokenLogitsProcessor.eos_token_id)**eos\_token\_id** (`int`) — The id of the token to force as the last generated token when `max_length` is reached.

[TFLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.TFLogitsProcessor) that enforces the specified token as the last generated token when `max_length` is reached.

#### \_\_call\_\_

[](#transformers.TFForcedEOSTokenLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_logits_process.py#L490)

( input\_ids: Tensorscores: Tensorcur\_len: int )

### class transformers.FlaxLogitsProcessor

[](#transformers.FlaxLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L50)

( )

Abstract base class for all logit processors that can be applied during generation.

#### \_\_call\_\_

[](#transformers.FlaxLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L53)

( input\_ids: ndarrayscores: ndarray ) → `jnp.ndarray` of shape `(batch_size, config.vocab_size)`

Parameters

-   [](#transformers.FlaxLogitsProcessor.__call__.input_ids)**input\_ids** (`jnp.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.FlaxLogitsProcessor.__call__.scores)**scores** (`jnp.ndarray` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs — Additional logits processor specific kwargs.

Returns

`jnp.ndarray` of shape `(batch_size, config.vocab_size)`

The processed prediction scores.

Flax method for processing logits.

### class transformers.FlaxLogitsProcessorList

[](#transformers.FlaxLogitsProcessorList)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L72)

( iterable = () )

This class can be used to create a list of [FlaxLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsProcessor) or [FlaxLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsWarper) to subsequently process a `scores` input tensor. This class inherits from list and adds a specific _**call**_ method to apply each [FlaxLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsProcessor) or [FlaxLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsWarper) to the inputs.

#### \_\_call\_\_

[](#transformers.FlaxLogitsProcessorList.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L79)

( input\_ids: ndarrayscores: ndarraycur\_len: int\*\*kwargs ) → `jnp.ndarray` of shape `(batch_size, config.vocab_size)`

Parameters

-   [](#transformers.FlaxLogitsProcessorList.__call__.input_ids)**input\_ids** (`jnp.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.FlaxLogitsProcessorList.__call__.scores)**scores** (`jnp.ndarray` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs — Additional logits processor specific kwargs.

Returns

`jnp.ndarray` of shape `(batch_size, config.vocab_size)`

The processed prediction scores.

Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.

#### \_\_call\_\_

[](#transformers.FlaxLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L64)

( input\_ids: ndarrayscores: ndarray ) → `jnp.ndarray` of shape `(batch_size, config.vocab_size)`

Parameters

-   [](#transformers.FlaxLogitsWarper.__call__.input_ids)**input\_ids** (`jnp.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.FlaxLogitsWarper.__call__.scores)**scores** (`jnp.ndarray` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs — Additional logits processor specific kwargs.

Returns

`jnp.ndarray` of shape `(batch_size, config.vocab_size)`

The processed prediction scores.

Flax method for warping logits.

### class transformers.FlaxTemperatureLogitsWarper

[](#transformers.FlaxTemperatureLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L95)

( temperature: float )

Parameters

-   [](#transformers.FlaxTemperatureLogitsWarper.temperature)**temperature** (`float`) — The value used to module the logits distribution.

[FlaxLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsWarper) for temperature (exponential scaling output probability distribution).

#### \_\_call\_\_

[](#transformers.FlaxTemperatureLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L110)

( input\_ids: ndarrayscores: ndarraycur\_len: int )

### class transformers.FlaxTopPLogitsWarper

[](#transformers.FlaxTopPLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L115)

( top\_p: floatfilter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.FlaxTopPLogitsWarper.top_p)**top\_p** (`float`) — If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation.
-   [](#transformers.FlaxTopPLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.FlaxTopPLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[FlaxLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsWarper) that performs top-p, i.e. restricting to top tokens summing to prob\_cut\_off <= prob\_cut\_off.

#### \_\_call\_\_

[](#transformers.FlaxTopPLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L137)

( input\_ids: ndarrayscores: ndarraycur\_len: int )

### class transformers.FlaxTopKLogitsWarper

[](#transformers.FlaxTopKLogitsWarper)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L157)

( top\_k: intfilter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.FlaxTopKLogitsWarper.top_k)**top\_k** (`int`) — The number of highest probability vocabulary tokens to keep for top-k-filtering.
-   [](#transformers.FlaxTopKLogitsWarper.filter_value)**filter\_value** (`float`, _optional_, defaults to `-float("Inf")`) — All filtered values will be set to this float value.
-   [](#transformers.FlaxTopKLogitsWarper.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimum number of tokens that cannot be filtered.

[FlaxLogitsWarper](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsWarper) that performs top-k, i.e. restricting to the k highest probability elements.

#### \_\_call\_\_

[](#transformers.FlaxTopKLogitsWarper.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L177)

( input\_ids: ndarrayscores: ndarraycur\_len: int )

### class transformers.FlaxForcedBOSTokenLogitsProcessor

[](#transformers.FlaxForcedBOSTokenLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L192)

( bos\_token\_id: int )

Parameters

-   [](#transformers.FlaxForcedBOSTokenLogitsProcessor.bos_token_id)**bos\_token\_id** (`int`) — The id of the token to force as the first generated token.

[FlaxLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsProcessor) that enforces the specified token as the first generated token.

#### \_\_call\_\_

[](#transformers.FlaxForcedBOSTokenLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L204)

( input\_ids: ndarrayscores: ndarraycur\_len: int )

### class transformers.FlaxForcedEOSTokenLogitsProcessor

[](#transformers.FlaxForcedEOSTokenLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L214)

( max\_length: inteos\_token\_id: int )

Parameters

-   [](#transformers.FlaxForcedEOSTokenLogitsProcessor.max_length)**max\_length** (`int`) — The maximum length of the sequence to be generated.
-   [](#transformers.FlaxForcedEOSTokenLogitsProcessor.eos_token_id)**eos\_token\_id** (`int`) — The id of the token to force as the last generated token when `max_length` is reached.

[FlaxLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsProcessor) that enforces the specified token as the last generated token when `max_length` is reached.

#### \_\_call\_\_

[](#transformers.FlaxForcedEOSTokenLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L229)

( input\_ids: ndarrayscores: ndarraycur\_len: int )

### class transformers.FlaxMinLengthLogitsProcessor

[](#transformers.FlaxMinLengthLogitsProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L239)

( min\_length: inteos\_token\_id: int )

Parameters

-   [](#transformers.FlaxMinLengthLogitsProcessor.min_length)**min\_length** (`int`) — The minimum length below which the score of `eos_token_id` is set to `-float("Inf")`.
-   [](#transformers.FlaxMinLengthLogitsProcessor.eos_token_id)**eos\_token\_id** (`int`) — The id of the _end-of-sequence_ token.

[FlaxLogitsProcessor](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.FlaxLogitsProcessor) enforcing a min-length by setting EOS probability to 0.

#### \_\_call\_\_

[](#transformers.FlaxMinLengthLogitsProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/flax_logits_process.py#L260)

( input\_ids: ndarrayscores: ndarraycur\_len: int )

## [](#transformers.StoppingCriteria)StoppingCriteria

A [StoppingCriteria](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.StoppingCriteria) can be used to change when to stop generation (other than EOS token).

Abstract base class for all stopping criteria that can be applied during generation.

#### \_\_call\_\_

[](#transformers.StoppingCriteria.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L36)

( input\_ids: LongTensorscores: FloatTensor\*\*kwargs )

Parameters

-   [](#transformers.StoppingCriteria.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.StoppingCriteria.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax or scores for each vocabulary token after SoftMax. kwargs — Additional stopping criteria specific kwargs.

### class transformers.StoppingCriteriaList

[](#transformers.StoppingCriteriaList)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L110)

( iterable = () )

#### \_\_call\_\_

[](#transformers.StoppingCriteriaList.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L111)

( input\_ids: LongTensorscores: FloatTensor\*\*kwargs )

Parameters

-   [](#transformers.StoppingCriteriaList.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.StoppingCriteriaList.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax or scores for each vocabulary token after SoftMax. kwargs — Additional stopping criteria specific kwargs.

### class transformers.MaxLengthCriteria

[](#transformers.MaxLengthCriteria)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L41)

( max\_length: int )

Parameters

-   [](#transformers.MaxLengthCriteria.max_length)**max\_length** (`int`) — The maximum length that the output sequence can have in number of tokens.

This class can be used to stop generation whenever the full generated number of tokens exceeds `max_length`. Keep in mind for decoder-only type of transformers, this will include the initial prompted tokens.

#### \_\_call\_\_

[](#transformers.MaxLengthCriteria.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L54)

( input\_ids: LongTensorscores: FloatTensor\*\*kwargs )

Parameters

-   [](#transformers.MaxLengthCriteria.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.MaxLengthCriteria.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax or scores for each vocabulary token after SoftMax. kwargs — Additional stopping criteria specific kwargs.

### class transformers.MaxTimeCriteria

[](#transformers.MaxTimeCriteria)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L88)

( max\_time: floatinitial\_timestamp: typing.Optional\[float\] = None )

Parameters

-   [](#transformers.MaxTimeCriteria.max_time)**max\_time** (`float`) — The maximum allowed time in seconds for the generation.
-   [](#transformers.MaxTimeCriteria.initial_time)**initial\_time** (`float`, _optional_, defaults to `time.time()`) — The start of the generation allowed time.

This class can be used to stop generation whenever the full generation exceeds some amount of time. By default, the time will start being counted when you initialize this function. You can override this by passing an `initial_time`.

#### \_\_call\_\_

[](#transformers.MaxTimeCriteria.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/stopping_criteria.py#L105)

( input\_ids: LongTensorscores: FloatTensor\*\*kwargs )

Parameters

-   [](#transformers.MaxTimeCriteria.__call__.input_ids)**input\_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.
    
    Indices can be obtained using [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) for details.
    
    [What are input IDs?](../glossary#input-ids)
    
-   [](#transformers.MaxTimeCriteria.__call__.scores)**scores** (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`) — Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax or scores for each vocabulary token after SoftMax. kwargs — Additional stopping criteria specific kwargs.

## [](#transformers.Constraint)Constraints

A [Constraint](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.Constraint) can be used to force the generation to include specific tokens or sequences in the output.

Abstract base class for all constraints that can be applied during generation. It must define how the constraint can be satisfied.

All classes that inherit Constraint must follow the requirement that

```
completed = False
while not completed:
    _, completed = constraint.update(constraint.advance())```

will always terminate (halt).

#### advance

[](#transformers.Constraint.advance)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L48)

( ) → token\_ids(`torch.tensor`)

Returns

token\_ids(`torch.tensor`)

Must be a tensor of a list of indexable tokens, not some integer.

When called, returns the token that would take this constraint one step closer to being fulfilled.

#### copy

[](#transformers.Constraint.copy)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L113)

( stateful = False ) → constraint(`Constraint`)

Returns

constraint(`Constraint`)

The same constraint as the one being called from.

Creates a new instance of this constraint.

Reads in a token and returns whether it creates progress.

Returns the number of remaining steps of `advance()` in order to complete this constraint.

Resets the state of this constraint to its initialization. We would call this in cases where the fulfillment of a constraint is abrupted by an unwanted token.

Tests whether this constraint has been properly defined.

#### update

[](#transformers.Constraint.update)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L69)

( token\_id: int ) → stepped(`bool`)

Whether this constraint has become one step closer to being fulfuilled. completed(`bool`): Whether this constraint has been completely fulfilled by this token being generated. reset (`bool`): Whether this constraint has reset its progress by this token being generated.

Reads in a token and returns booleans that indicate the progress made by it. This function will update the state of this object unlikes `does_advance(self, token_id: int)`.

This isn’t to test whether a certain token will advance the progress; it’s to update its state as if it has been generated. This becomes important if token\_id != desired token (refer to else statement in PhrasalConstraint)

### class transformers.PhrasalConstraint

[](#transformers.PhrasalConstraint)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L129)

( token\_ids: typing.List\[int\] )

Parameters

-   [](#transformers.PhrasalConstraint.token_ids)**token\_ids** (`List[int]`) — The id of the token that must be generated by the output.

[Constraint](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.Constraint) enforcing that an ordered sequence of tokens is included in the output.

### class transformers.DisjunctiveConstraint

[](#transformers.DisjunctiveConstraint)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L261)

( nested\_token\_ids: typing.List\[typing.List\[int\]\] )

Parameters

-   [](#transformers.DisjunctiveConstraint.nested_token_ids)**nested\_token\_ids** (`List[List[int]]`) — a list of words, where each word is a list of ids. This constraint
-   [](#transformers.DisjunctiveConstraint.is)**is** fulfilled by generating just one from the list of words. —

A special [Constraint](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.Constraint) that is fulfilled by fulfilling just one of several constraints.

### class transformers.ConstraintListState

[](#transformers.ConstraintListState)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L350)

( constraints: typing.List\[transformers.generation.beam\_constraints.Constraint\] )

Parameters

-   [](#transformers.ConstraintListState.constraints)**constraints** (`List[Constraint]`) — A list of [Constraint](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.Constraint) objects that must be fulfilled by the beam scorer.

A class for beam scorers to track its progress through a list of constraints.

The list of tokens to generate such that we can make progress. By “list” we don’t mean the list of token that will fully fulfill a constraint.

Given constraints `c_i = {t_ij | j == # of tokens}`, If we’re not in the middle of progressing through a specific constraint `c_i`, we return:

`[t_k1 for k in indices of unfulfilled constraints]`

If we are in the middle of a constraint, then we return: `[t_ij]`, where `i` is the index of the inprogress constraint, `j` is the next step for the constraint.

Though we don’t care which constraint is fulfilled first, if we are in the progress of fulfilling a constraint, that’s the only one we’ll return.

#### reset

[](#transformers.ConstraintListState.reset)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_constraints.py#L417)

( token\_ids: typing.Optional\[typing.List\[int\]\] )

token\_ids: the tokens generated thus far to reset the state of the progress through constraints.

## [](#transformers.BeamScorer)BeamSearch

Abstract base class for all beam scorers that are used for [beam\_search()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.beam_search) and [beam\_sample()](/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.beam_sample).

#### process

[](#transformers.BeamScorer.process)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L93)

( input\_ids: LongTensornext\_scores: FloatTensornext\_tokens: LongTensornext\_indices: LongTensor\*\*kwargs ) → `UserDict`

#### finalize

[](#transformers.BeamScorer.finalize)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L105)

( input\_ids: LongTensornext\_scores: FloatTensornext\_tokens: LongTensornext\_indices: LongTensormax\_length: int\*\*kwargs ) → `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`

### class transformers.BeamSearchScorer

[](#transformers.BeamSearchScorer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L119)

( batch\_size: intnum\_beams: intdevice: devicelength\_penalty: typing.Optional\[float\] = 1.0do\_early\_stopping: typing.Union\[bool, str, NoneType\] = Falsenum\_beam\_hyps\_to\_keep: typing.Optional\[int\] = 1num\_beam\_groups: typing.Optional\[int\] = 1max\_length: typing.Optional\[int\] = None )

[BeamScorer](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.BeamScorer) implementing standard beam search decoding.

Adapted in part from [Facebook’s XLM beam search code](https://github.com/facebookresearch/XLM/blob/9e6f6814d17be4fe5b15f2e6c43eb2b2d76daeb4/src/model/transformer.py#L529).

Reference for the diverse beam search algorithm and implementation [Ashwin Kalyan’s DBS implementation](https://github.com/ashwinkalyan/dbs/blob/master/dbs/beam_utils.lua)

#### process

[](#transformers.BeamSearchScorer.process)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L205)

( input\_ids: LongTensornext\_scores: FloatTensornext\_tokens: LongTensornext\_indices: LongTensorpad\_token\_id: typing.Optional\[int\] = Noneeos\_token\_id: typing.Union\[int, typing.List\[int\], NoneType\] = Nonebeam\_indices: typing.Optional\[torch.LongTensor\] = None )

#### finalize

[](#transformers.BeamSearchScorer.finalize)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L302)

( input\_ids: LongTensorfinal\_beam\_scores: FloatTensorfinal\_beam\_tokens: LongTensorfinal\_beam\_indices: LongTensormax\_length: intpad\_token\_id: typing.Optional\[int\] = Noneeos\_token\_id: typing.Union\[int, typing.List\[int\], NoneType\] = Nonebeam\_indices: typing.Optional\[torch.LongTensor\] = None )

### class transformers.ConstrainedBeamSearchScorer

[](#transformers.ConstrainedBeamSearchScorer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L394)

( batch\_size: intnum\_beams: intconstraints: typing.List\[transformers.generation.beam\_constraints.Constraint\]device: devicelength\_penalty: typing.Optional\[float\] = 1.0do\_early\_stopping: typing.Union\[bool, str, NoneType\] = Falsenum\_beam\_hyps\_to\_keep: typing.Optional\[int\] = 1num\_beam\_groups: typing.Optional\[int\] = 1max\_length: typing.Optional\[int\] = None )

[BeamScorer](/docs/transformers/v4.30.0/en/internal/generation_utils#transformers.BeamScorer) implementing constrained beam search decoding.

#### process

[](#transformers.ConstrainedBeamSearchScorer.process)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L488)

( input\_ids: LongTensornext\_scores: FloatTensornext\_tokens: LongTensornext\_indices: LongTensorscores\_for\_all\_vocab: FloatTensorpad\_token\_id: typing.Optional\[int\] = Noneeos\_token\_id: typing.Union\[int, typing.List\[int\], NoneType\] = None ) → `UserDict`

#### finalize

[](#transformers.ConstrainedBeamSearchScorer.finalize)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/beam_search.py#L772)

( input\_ids: LongTensorfinal\_beam\_scores: FloatTensorfinal\_beam\_tokens: LongTensorfinal\_beam\_indices: LongTensormax\_length: intpad\_token\_id: typing.Optional\[int\] = Noneeos\_token\_id: typing.Union\[int, typing.List\[int\], NoneType\] = None )

## [](#transformers.top_k_top_p_filtering)Utilities

#### transformers.top\_k\_top\_p\_filtering

[](#transformers.top_k_top_p_filtering)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L4565)

( logits: FloatTensortop\_k: int = 0top\_p: float = 1.0filter\_value: float = -infmin\_tokens\_to\_keep: int = 1 )

Parameters

-   [](#transformers.top_k_top_p_filtering.top_k)**top\_k** (`int`, _optional_, defaults to 0) — If > 0, only keep the top k tokens with highest probability (top-k filtering)
-   [](#transformers.top_k_top_p_filtering.top_p)**top\_p** (`float`, _optional_, defaults to 1.0) — If < 1.0, only keep the top tokens with cumulative probability >= top\_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. ([http://arxiv.org/abs/1904.09751](http://arxiv.org/abs/1904.09751))
-   [](#transformers.top_k_top_p_filtering.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimumber of tokens we keep per batch example in the output.

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering

From: [https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317](https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317)

#### transformers.tf\_top\_k\_top\_p\_filtering

[](#transformers.tf_top_k_top_p_filtering)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/tf_utils.py#L3080)

( logitstop\_k = 0top\_p = 1.0filter\_value = -infmin\_tokens\_to\_keep = 1 )

Parameters

-   [](#transformers.tf_top_k_top_p_filtering.top_k)**top\_k** (`int`, _optional_, defaults to 0) — If > 0, only keep the top k tokens with highest probability (top-k filtering)
-   [](#transformers.tf_top_k_top_p_filtering.top_p)**top\_p** (`float`, _optional_, defaults to 1.0) — If < 1.0, only keep the top tokens with cumulative probability >= top\_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. ([http://arxiv.org/abs/1904.09751](http://arxiv.org/abs/1904.09751))
-   [](#transformers.tf_top_k_top_p_filtering.min_tokens_to_keep)**min\_tokens\_to\_keep** (`int`, _optional_, defaults to 1) — Minimumber of tokens we keep per batch example in the output.

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering

From: [https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317](https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317)

## [](#transformers.TextStreamer)Streamers

### class transformers.TextStreamer

[](#transformers.TextStreamer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/streamers.py#L38)

( tokenizer: AutoTokenizerskip\_prompt: bool = False\*\*decode\_kwargs )

Parameters

-   [](#transformers.TextStreamer.tokenizer)**tokenizer** (`AutoTokenizer`) — The tokenized used to decode the tokens.
-   [](#transformers.TextStreamer.skip_prompt)**skip\_prompt** (`bool`, _optional_, defaults to `False`) — Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
-   [](#transformers.TextStreamer.decode_kwargs)**decode\_kwargs** (`dict`, _optional_) — Additional keyword arguments to pass to the tokenizer’s `decode` method.

Simple text streamer that prints the token(s) to stdout as soon as entire words are formed.

The API for the streamer classes is still under development and may change in the future.

Examples:

```
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

>>> tok = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)

>>> 
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,```

Flushes any remaining cache and prints a newline to stdout.

#### on\_finalized\_text

[](#transformers.TextStreamer.on_finalized_text)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/streamers.py#L130)

( text: strstream\_end: bool = False )

Prints the new text to stdout. If the stream is ending, also prints a newline.

Recives tokens, decodes them, and prints them to stdout as soon as they form entire words.

### class transformers.TextIteratorStreamer

[](#transformers.TextIteratorStreamer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/streamers.py#L159)

( tokenizer: AutoTokenizerskip\_prompt: bool = Falsetimeout: typing.Optional\[float\] = None\*\*decode\_kwargs )

Parameters

-   [](#transformers.TextIteratorStreamer.tokenizer)**tokenizer** (`AutoTokenizer`) — The tokenized used to decode the tokens.
-   [](#transformers.TextIteratorStreamer.skip_prompt)**skip\_prompt** (`bool`, _optional_, defaults to `False`) — Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
-   [](#transformers.TextIteratorStreamer.timeout)**timeout** (`float`, _optional_) — The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread.
-   [](#transformers.TextIteratorStreamer.decode_kwargs)**decode\_kwargs** (`dict`, _optional_) — Additional keyword arguments to pass to the tokenizer’s `decode` method.

Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from acessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo).

The API for the streamer classes is still under development and may change in the future.

Examples:

```
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
>>> from threading import Thread

>>> tok = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextIteratorStreamer(tok)

>>> 
>>> generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20)
>>> thread = Thread(target=model.generate, kwargs=generation_kwargs)
>>> thread.start()
>>> generated_text = ""
>>> for new_text in streamer:
...     generated_text += new_text
>>> generated_text
'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,'```

#### on\_finalized\_text

[](#transformers.TextIteratorStreamer.on_finalized_text)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/streamers.py#L213)

( text: strstream\_end: bool = False )

Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.