philipp-zettl/vrom-hf-docs huggingface dataset:
- [GenerateDecoderOnlyOutput](/docs/transformers/v5.6.2/en/internal/generation_utils#transformers.generation.GenerateDecoderOnlyOutput),
- [GenerateBeamDecoderOnlyOutput](/docs/transformers/v5.6.2/en/internal/generation_utils#transformers.generation.GenerateBeamDecoderOnlyOutput)

If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
[ModelOutput](/docs/transformers/v5.6.2/en/main_classes/output#transformers.utils.ModelOutput) types are:

- [GenerateEncoderDecoderOutput](/docs/transformers/v5.6.2/en/internal/generation_utils#transformers.generation.GenerateEncoderDecoderOutput),
- [GenerateBeamEncoderDecoderOutput](/docs/transformers/v5.6.2/en/internal/generation_utils#transformers.generation.GenerateBeamEncoderDecoderOutput)
#### compute_transition_scores[[transformers.GenerationMixin.compute_transition_scores]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/utils.py#L1335)

Computes the transition scores of sequences given the generation scores (and beam indices, if beam search was
used). This is a convenient method to quickly obtain the scores of the selected tokens at generation time.

Examples:
```python
>>> from transformers import GPT2Tokenizer, AutoModelForCausalLM
>>> import numpy as np

>>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> tokenizer.pad_token_id = tokenizer.eos_token_id
>>> inputs = tokenizer(["Today is"], return_tensors="pt")

>>> # Example 1: Print the scores for each token generated with Greedy Search
>>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
>>> transition_scores = model.compute_transition_scores(
...     outputs.sequences, outputs.scores, normalize_logits=True
... )
>>> # input_length is the length of the input prompt for decoder-only models, like the GPT family, and 1 for
>>> # encoder-decoder models, like BART or T5.
>>> input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1]
>>> generated_tokens = outputs.sequences[:, input_length:]
>>> for tok, score in zip(generated_tokens[0], transition_scores[0]):
...     # | token | token string | log probability | probability
...     print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}")
|   262 |  the     | -1.414 | 24.33%
|  1110 |  day     | -2.609 | 7.36%
|   618 |  when    | -2.010 | 13.40%
|   356 |  we      | -1.859 | 15.58%
|   460 |  can     | -2.508 | 8.14%

>>> # Example 2: Reconstruct the sequence scores from Beam Search
>>> outputs = model.generate(
...     **inputs,
...     max_new_tokens=5,
...     num_beams=4,
...     num_return_sequences=4,
...     return_dict_in_generate=True,
...     output_scores=True,
... )
>>> transition_scores = model.compute_transition_scores(
...     outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False
... )
>>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores.
>>> # Tip 1: recomputing the scores is only guaranteed to match with `normalize_logits=False`. Depending on the
>>> # use case, you might want to recompute it with `normalize_logits=True`.
>>> # Tip 2: the output length does NOT include the input length
>>> output_length = np.sum(transition_scores.numpy() >> length_penalty = model.generation_config.length_penalty
>>> reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty)
>>> print(np.allclose(outputs.sequences_scores, reconstructed_scores))
True
```
**Parameters:**

sequences (`torch.LongTensor`) : The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`.

scores (`tuple(torch.FloatTensor)`) : Transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`.

beam_indices (`torch.LongTensor`, *optional*) : Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. Only required if a `num_beams>1` at generate-time.

normalize_logits (`bool`, *optional*, defaults to `False`) : Whether to normalize the logits (which, for legacy reasons, may be unnormalized).
#### transformers.ContinuousMixin[[transformers.ContinuousMixin]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L1147)

Mixin class for models to add continuous batching capabilities. Continuous batching has three entry points:
- `init_continuous_batching`, which is the actual entry point for continuous batching
- `continuous_batching_context_manager`, which itself is a wrapper around `init_continuous_batching`
- `generate_batch`, which is really a wrapper around `continuous_batching_context_manager`

They are defined in this order. Any change made to any of those three entry points should be reflected in the other
two.
continuous_batching_context_managertransformers.ContinuousMixin.continuous_batching_context_managerhttps://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L1219[{"name": "generation_config", "val": ": transformers.generation.configuration_utils.GenerationConfig | None = None"}, {"name": "block", "val": ": bool = True"}, {"name": "timeout", "val": ": float | None = None"}, {"name": "continuous_batching_config", "val": ": transformers.generation.configuration_utils.ContinuousBatchingConfig | None = None"}, {"name": "persistent_manager", "val": ": bool = False"}, {"name": "warmup_requests", "val": ": int | None = 0"}, {"name": "**deprecated_kwargs", "val": ""}]
A context manager to safely use the continuous batching manager. Arguments are similar to the ones of
`init_continuous_batching`, except for:
- block: whether to block the thread when stopping the manager. Default is True.
- timeout: maximum time to wait for the thread to stop. Default is None (no timeout).
- warmup_query_tokens: the number of expected requests for which to warmup. 0 is auto, None is no warmup.
#### generate_batch[[transformers.ContinuousMixin.generate_batch]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L1257)

Generate sequences for a batch of prompts using continuous batching.

**Parameters:**

inputs : List of input token sequences (prompts)

generation_config : Optional generation configuration

continuous_batching_config : Optional continuous batching configuration

record_timestamps : If set to true, the requests will have a timestamp for each token generated

progress_bar : If set to true, a progress bar will be displayed

persistent_manager : whether to persist the manager after the generation is finished. Default is False.

warmup : whether to pre-capture CUDA graphs before processing requests. Default is True.

- ****kwargs** : Additional generation parameters. Only max_new_tokens is used, but other deprecated arguments are extracted and passed to the continuous_batching_config object.

**Returns:**
#### init_continuous_batching[[transformers.ContinuousMixin.init_continuous_batching]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L1159)

Initialize a manager for continuous batching inference.

**Parameters:**

generation_config : An optional generation configuration, which may contain a CompileConfig object

continuous_batching_config : An optional continuous batching configuration

- ****deprecated_kwargs** : Deprecated arguments that are now passed in the continuous_batching_config. Those are: max_queue_size, q_padding_interval_size, kv_padding_interval_size, allow_block_sharing, use_async_batching, max_cached_graphs

**Returns:**

``ContinuousBatchingManager``

The manager instance to add requests and retrieve results.
#### transformers.ContinuousBatchingManager[[transformers.ContinuousBatchingManager]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L731)

Manager for handling continuous batching of generation requests. It provides a user interface for submitting
generation requests, retrieving results, and managing the background generation thread. This class should not be
created directly, but through one of the following entry points (all methods of the `ContinuousMixin` mixin):
- `init_continuous_batching`
- `continuous_batching_context_manager`
- `generate_batch`
add_requesttransformers.ContinuousBatchingManager.add_requesthttps://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L878[{"name": "input_ids", "val": ": list"}, {"name": "request_id", "val": ": str | None = None"}, {"name": "max_new_tokens", "val": ": int | None = None"}, {"name": "streaming", "val": ": bool = False"}, {"name": "record_timestamps", "val": ": bool = False"}, {"name": "eos_token_id", "val": ": int | list[int] | None = None"}, {"name": "**logit_processor_kwargs", "val": ": typing.Any"}]- **input_ids** -- Input token IDs to use as prompt
- **request_id** -- Optional custom request ID (auto-generated if None)
- **max_new_tokens** -- Maximum number of new tokens to generate
- **streaming** -- Whether to stream tokens as they're generated
- **record_timestamps** -- Whether to record timestamps for each generated token
- **eos_token_id** -- End-of-sequence token ID(s)
- **logit_processor_kwargs** -- Keyword arguments for the logits processor.0strThe request ID
Add a new generation request to the queue.
**Parameters:**

input_ids : Input token IDs to use as prompt

request_id : Optional custom request ID (auto-generated if None)

max_new_tokens : Maximum number of new tokens to generate

streaming : Whether to stream tokens as they're generated

record_timestamps : Whether to record timestamps for each generated token

eos_token_id : End-of-sequence token ID(s)

logit_processor_kwargs : Keyword arguments for the logits processor.

**Returns:**

`str`

The request ID
#### get_result[[transformers.ContinuousBatchingManager.get_result]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L970)

Retrieve one result from the output queue.

**Parameters:**

request_id : If set, only return results matching this ID (others are requeued).

timeout : Maximum time to wait for a result.

**Returns:**

`Optional[GenerationOutput]`

The result data or None if timeout.
#### register_result_handler[[transformers.ContinuousBatchingManager.register_result_handler]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L1011)

Register a callback for result delivery (streaming or non-streaming).

The callback is invoked on the event loop via `call_soon_threadsafe`
each time a result is produced for this request. For streaming requests,
this happens on every token; for non-streaming, only on completion.

The handler is automatically cleaned up when the request finishes.

**Parameters:**

request_id (*str*) : The request ID to receive outputs for.

callback (*callable*) : Called with a `GenerationOutput` for each result.
#### request_id_iter[[transformers.ContinuousBatchingManager.request_id_iter]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L998)

Iterate over results matching a specific request id (blocking).

Uses the shared output queue with requeue. For high-concurrency serving,
use `register_result_handler` instead.
#### stop[[transformers.ContinuousBatchingManager.stop]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/continuous_api.py#L823)

Signal the background thread to stop.

**Parameters:**

block : Whether to wait for the thread to stop

timeout : Maximum time to wait for the thread to stop

keep_for_next_session : Whether to cache this on the model for future use
#### transformers.generation.Scheduler[[transformers.generation.Scheduler]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/scheduler.py#L23)

Abstract base class for scheduling requests in the continuous batch processor. Schedulers manage the lifecycle of
requests from when they are added to the waiting queue to when they are scheduled for processing. Different
schedulers implement different strategies for prioritizing and batching requests.

add_waiting_requesttransformers.generation.Scheduler.add_waiting_requesthttps://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/scheduler.py#L48[{"name": "state", "val": ": RequestState"}]
Adds a request to the waiting list.
#### schedule_batch[[transformers.generation.Scheduler.schedule_batch]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/scheduler.py#L54)

Schedules requests for the next batch based on available token and cache budgets. This method selects which
requests should be processed in the current batch, considering the budgets and the scheduler's prioritization
rules. The token_budget is the maximum number of tokens that can be processed in a batch, and the cache_budget
is the maximum number of KV cache entries that can be read in a batch.
Returns the list of scheduled requests in their "FutureRequestState" form, a boolean indicating if the decode
fast path can be used, the total number of query tokens and the maximum number of kv tokens read.
#### transformers.generation.FIFOScheduler[[transformers.generation.FIFOScheduler]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/scheduler.py#L292)

This scheduler processes requests in the order they arrive, meaning decoding requests has priority over
prefilling requests. Additionally, it includes a safety margin mechanism to prevent cache exhaustion. By default,
when 80% of the cache is full, new requests will not be scheduled to prioritize decoding active requests.
#### transformers.generation.PrefillFirstScheduler[[transformers.generation.PrefillFirstScheduler]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/generation/continuous_batching/scheduler.py#L348)

Scheduler that prioritizes split prefill requests over decoding requests. This scheduler ensures that split
prefill requests (which are continuations of partially processed prompts) are completed before processing new
decoding requests.
# Model outputs

All models have outputs that are instances of subclasses of [ModelOutput](/docs/transformers/v5.6.2/en/main_classes/output#transformers.utils.ModelOutput). Those are
data structures containing all the information returned by the model, but that can also be used as tuples or
dictionaries.

Let's see how this looks in an example:

```python
from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
model = BertForSequenceClassification.from_pretrained("google-bert/bert-base-uncased")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(**inputs, labels=labels)
```
The `outputs` object is a [SequenceClassifierOutput](/docs/transformers/v5.6.2/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput), as we can see in the
documentation of that class below, it means it has an optional `loss`, a `logits`, an optional `hidden_states` and
an optional `attentions` attribute. Here we have the `loss` since we passed along `labels`, but we don't have
`hidden_states` and `attentions` because we didn't pass `output_hidden_states=True` or
`output_attentions=True`.

When passing `output_hidden_states=True` you may expect the `outputs.hidden_states[-1]` to match `outputs.last_hidden_state` exactly.
However, this is not always the case. Some models apply normalization or subsequent process to the last hidden state when it's returned.
You can access each attribute as you would usually do, and if that attribute has not been returned by the model, you
will get `None`. Here for instance `outputs.loss` is the loss computed by the model, and `outputs.attentions` is
`None`.

When considering our `outputs` object as tuple, it only considers the attributes that don't have `None` values.
Here for instance, it has two elements, `loss` then `logits`, so

```python
outputs[:2]
```

will return the tuple `(outputs.loss, outputs.logits)` for instance.

When considering our `outputs` object as dictionary, it only considers the attributes that don't have `None`
values. Here for instance, it has two keys that are `loss` and `logits`.

We document here the generic model outputs that are used by more than one model type. Specific output types are
documented on their corresponding model page.
#### transformers.utils.ModelOutput[[transformers.utils.ModelOutput]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/utils/generic.py#L366)

Base class for all model outputs as dataclass. Has a `__getitem__` that allows indexing by integer or slice (like a
tuple) or strings (like a dictionary) that will ignore the `None` attributes. Otherwise behaves like a regular
python dictionary.

You can't unpack a `ModelOutput` directly. Use the [to_tuple()](/docs/transformers/v5.6.2/en/main_classes/output#transformers.utils.ModelOutput.to_tuple) method to convert it to a tuple
before.

to_tupletransformers.utils.ModelOutput.to_tuplehttps://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/utils/generic.py#L499[]

Convert self to a tuple containing all the attributes/keys that are not `None`.
#### transformers.modeling_outputs.BaseModelOutput[[transformers.modeling_outputs.BaseModelOutput]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/modeling_outputs.py#L24)

Base class for model's outputs, with potential hidden states and attentions.

**Parameters:**

last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) : Sequence of hidden-states at the output of the last layer of the model.

hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) : Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) : Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
#### transformers.modeling_outputs.BaseModelOutputWithPooling[[transformers.modeling_outputs.BaseModelOutputWithPooling]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/modeling_outputs.py#L69)

Base class for model's outputs that also contains a pooling of the last hidden states.

**Parameters:**

last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) : Sequence of hidden-states at the output of the last layer of the model.

pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) : Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) : Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) : Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
#### transformers.modeling_outputs.BaseModelOutputWithCrossAttentions[[transformers.modeling_outputs.BaseModelOutputWithCrossAttentions]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/modeling_outputs.py#L159)

Base class for model's outputs, with potential hidden states and attentions.

**Parameters:**

last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) : Sequence of hidden-states at the output of the last layer of the model.

hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) : Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) : Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` and `config.add_cross_attention=True` is passed or when `config.output_attentions=True`) : Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.  Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads.
#### transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions[[transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/modeling_outputs.py#L192)

Base class for model's outputs that also contains a pooling of the last hidden states.

**Parameters:**

last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) : Sequence of hidden-states at the output of the last layer of the model.