philipp-zettl/vrom-hf-docs huggingface dataset:
This feature extraction pipeline can currently be loaded from [pipeline()](/docs/transformers/v5.6.2/en/main_classes/pipelines#transformers.pipeline) using the task identifier:
`"feature-extraction"`.

All models may be used for this pipeline. See a list of all models, including community-contributed models on
[huggingface.co/models](https://huggingface.co/models).

__call__transformers.FeatureExtractionPipeline.__call__https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/feature_extraction.py#L78[{"name": "*args", "val": ": str | list[str]"}, {"name": "**kwargs", "val": ": typing.Any"}]- **args** (`str` or `list[str]`) -- One or several texts (or one list of texts) to get the features of.0A nested list of `float`The features computed by the model.

Extract the features of the input(s) text.

**Parameters:**
#### transformers.ImageFeatureExtractionPipeline[[transformers.ImageFeatureExtractionPipeline]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/image_feature_extraction.py#L23)

Image feature extraction pipeline uses no model head. This pipeline extracts the hidden states from the base
transformer, which can be used as features in downstream tasks.

Example:

```python
>>> from transformers import pipeline

>>> extractor = pipeline(model="google/vit-base-patch16-224", task="image-feature-extraction")
>>> result = extractor("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", return_tensors=True)
>>> result.shape  # This is a tensor of shape [1, sequence_length, hidden_dimension] representing the input image.
torch.Size([1, 197, 768])
```

Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
This image feature extraction pipeline can currently be loaded from [pipeline()](/docs/transformers/v5.6.2/en/main_classes/pipelines#transformers.pipeline) using the task identifier:
`"image-feature-extraction"`.

All vision models may be used for this pipeline. See a list of all models, including community-contributed models on
[huggingface.co/models](https://huggingface.co/models).

__call__transformers.ImageFeatureExtractionPipeline.__call__https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/image_feature_extraction.py#L94[{"name": "*args", "val": ": typing.Union[str, ForwardRef('Image.Image'), list['Image.Image'], list[str]]"}, {"name": "**kwargs", "val": ": typing.Any"}]- **images** (`str`, `list[str]`, `PIL.Image` or `list[PIL.Image]`) --
  The pipeline handles three types of images:

  - A string containing a http link pointing to an image
  - A string containing a local path to an image
  - An image loaded in PIL directly
The pipeline accepts either a single image or a batch of images, which must then be passed as a string.
  Images in a batch must all be in the same format: all as http links, all as local paths, or all as PIL
  images.
- **timeout** (`float`, *optional*, defaults to None) --
  The maximum time in seconds to wait for fetching images from the web. If None, no timeout is used and
  the call may block forever.0A nested list of `float`The features computed by the model.

Extract the features of the input(s).

**Parameters:**

model ([PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel)) : The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from [PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel).
args_parser ([ArgumentHandler](/docs/transformers/v5.6.2/en/internal/pipelines_utils#transformers.pipelines.ArgumentHandler), *optional*) : Reference to the object in charge of parsing supplied pipeline parameters.

device (`int`, *optional*, defaults to -1) : Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, a positive will run the model on the associated CUDA device id. You can pass native `torch.device` or a `str` too

dtype (`str` or `torch.dtype`, *optional*) : Sent directly as `model_kwargs` (just a simpler shortcut) to use the available precision for this model (`torch.float16`, `torch.bfloat16`, ... or `"auto"`)

binary_output (`bool`, *optional*, defaults to `False`) : Flag indicating if the output the pipeline should happen in a serialized format (i.e., pickle) or as the raw output data e.g. text.

image_processor_kwargs (`dict`, *optional*) : Additional dictionary of keyword arguments passed along to the image processor e.g. {"size": {"height": 100, "width": 100}}
#### transformers.ImageTextToTextPipeline[[transformers.ImageTextToTextPipeline]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/image_text_to_text.py#L53)

Image-text-to-text pipeline using an `AutoModelForImageTextToText`. This pipeline generates text given an image and text.
When the underlying model is a conversational model, it can also accept one or more chats,
in which case the pipeline will operate in chat mode and will continue the chat(s) by adding its response(s).
Each chat takes the form of a list of dicts, where each dict contains "role" and "content" keys.

Unless the model you're using explicitly sets these generation parameters in its configuration files
(`generation_config.json`), the following default values will be used:
- max_new_tokens: 256

Example:
```python
>>> from transformers import pipeline

>>> pipe = pipeline("image-text-to-text", model="llava-hf/llava-interleave-qwen-0.5b-hf")
>>> messages = [
>>>     {
>>>         "role": "user",
>>>         "content": [
>>>             {
>>>                 "type": "image",
>>>                 "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
>>>             },
>>>             {"type": "text", "text": "Describe this image."},
>>>         ],
>>>     },
>>>     {
>>>         "role": "assistant",
>>>         "content": [
>>>             {"type": "text", "text": "There is a dog and"},
>>>         ],
>>>     },
>>> ]
>>> pipe(text=messages, max_new_tokens=20, return_full_text=False)
[{'input_text': [{'role': 'user',
    'content': [{'type': 'image',
    'url': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'},
    {'type': 'text', 'text': 'Describe this image.'}]},
{'role': 'assistant',
    'content': [{'type': 'text', 'text': 'There is a dog and'}]}],
'generated_text': ' a person in the image. The dog is sitting on the sand, and the person is sitting on'}]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)

This image-text to text pipeline can currently be loaded from pipeline() using the following task identifier:
"image-text-to-text".

See the list of available models on
[huggingface.co/models](https://huggingface.co/models?pipeline_tag=image-text-to-text).

__call__transformers.ImageTextToTextPipeline.__call__https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/image_text_to_text.py#L216[{"name": "images", "val": ": typing.Union[str, list[str], list[list[str]], ForwardRef('Image.Image'), list['Image.Image'], list[list['Image.Image']], list[dict], NoneType] = None"}, {"name": "text", "val": ": str | list[str] | list[dict] | None = None"}, {"name": "**kwargs", "val": ""}]- **images** (`str`, `list[str]`, `PIL.Image, `list[PIL.Image]`, `list[dict[str, Union[str, PIL.Image]]]`) --
  The pipeline handles three types of images:
The pipeline accepts either a single image or a batch of images. Finally, this pipeline also supports
  the chat format (see `text`) containing images and text in this argument.
- **text** (str, list[str], `list[dict[str, Union[str, PIL.Image]]]`) --
  The text to be used for generation. If a list of strings is passed, the length of the list should be
  the same as the number of images. Text can also follow the chat format: a list of dictionaries where
  each dictionary represents a message in a conversation. Each dictionary should have two keys: 'role'
  and 'content'. 'role' should be one of 'user', 'system' or 'assistant'. 'content' should be a list of
  dictionary containing the text of the message and the type of the message. The type of the message
  can be either 'text' or 'image'. If the type is 'image', no text is needed.
- **return_tensors** (`bool`, *optional*, defaults to `False`) --
  Returns the tensors of predictions (as token indices) in the outputs. If set to
  `True`, the decoded text is not returned.
- **return_text** (`bool`, *optional*) --
  Returns the decoded texts in the outputs.
- **return_full_text** (`bool`, *optional*, defaults to `True`) --
  If set to `False` only added text is returned, otherwise the full text is returned. Cannot be
  specified at the same time as `return_text`.
- **clean_up_tokenization_spaces** (`bool`, *optional*, defaults to `True`) --
  Whether or not to clean up the potential extra spaces in the text output.
- **continue_final_message(** `bool`, *optional*) -- This indicates that you want the model to continue the
  last message in the input chat rather than starting a new one, allowing you to "prefill" its response.
  By default this is `True` when the final message in the input chat has the `assistant` role and
  `False` otherwise, but you can manually override that behaviour by setting this flag.0A list or a list of list of `dict`Each result comes as a dictionary with the following key (cannot
return a combination of both `generated_text` and `generated_token_ids`):
- **generated_text** (`str`, present when `return_text=True`) -- The generated text.
- **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True`) -- The token
  ids of the generated text.
- **input_text** (`str`) -- The input text.

Generate a text given text and the image(s) passed as inputs.

**Parameters:**

model ([PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel)) : The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from [PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel).
processor ([ProcessorMixin](/docs/transformers/v5.6.2/en/main_classes/processors#transformers.ProcessorMixin)) : The processor that will be used by the pipeline to encode data for the model. This object inherits from [ProcessorMixin](/docs/transformers/v5.6.2/en/main_classes/processors#transformers.ProcessorMixin). Processor is a composite object that might contain `tokenizer`, `feature_extractor`, and `image_processor`.

task (`str`, defaults to `""`) : A task-identifier for the pipeline.

num_workers (`int`, *optional*, defaults to 8) : When the pipeline will use *DataLoader* (when passing a dataset, on GPU for a Pytorch model), the number of workers to be used.

batch_size (`int`, *optional*, defaults to 1) : When the pipeline will use *DataLoader* (when passing a dataset, on GPU for a Pytorch model), the size of the batch to use, for inference this is not always beneficial, please read [Batching with pipelines](https://huggingface.co/transformers/main_classes/pipelines.html#pipeline-batching) .
args_parser ([ArgumentHandler](/docs/transformers/v5.6.2/en/internal/pipelines_utils#transformers.pipelines.ArgumentHandler), *optional*) : Reference to the object in charge of parsing supplied pipeline parameters.

device (`int`, *optional*, defaults to -1) : Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, a positive will run the model on the associated CUDA device id. You can pass native `torch.device` or a `str` too

dtype (`str` or `torch.dtype`, *optional*) : Sent directly as `model_kwargs` (just a simpler shortcut) to use the available precision for this model (`torch.float16`, `torch.bfloat16`, ... or `"auto"`)

binary_output (`bool`, *optional*, defaults to `False`) : Flag indicating if the output the pipeline should happen in a serialized format (i.e., pickle) or as the raw output data e.g. text.

**Returns:**

`A list or a list of list of `dict``
Each result comes as a dictionary with the following key (cannot
return a combination of both `generated_text` and `generated_token_ids`):

- **generated_text** (`str`, present when `return_text=True`) -- The generated text.
- **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True`) -- The token
  ids of the generated text.
- **input_text** (`str`) -- The input text.
#### transformers.AnyToAnyPipeline[[transformers.AnyToAnyPipeline]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/any_to_any.py#L67)

Multimodal Generation pipeline using an `AutoModelForMultimodalLM`. This pipeline generates text given any
combination of multimodal data and text.When the underlying model is a conversational model, it can also
accept one or more chats, in which case the pipeline will operate in chat mode and will continue the
chat(s) by adding its response(s). Each chat takes the form of a list of dicts, where each dict contains
"role" and "content" keys.

Unless the model you're using explicitly sets these generation parameters in its configuration files
(`generation_config.json`), the following default values will be used:
- max_new_tokens: 256

Example:
```python
>>> from transformers import pipeline

>>> pipe = pipeline("any-to-any", model="google/gemma-3n-E4B-it")
>>> messages = [
>>>     {
>>>         "role": "user",
>>>         "content": [
>>>             {
>>>                 "type": "image",
>>>                 "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
>>>             },
>>>             {"type": "text", "text": "Describe this image."},
>>>         ],
>>>     },
>>>     {
>>>         "role": "assistant",
>>>         "content": [
>>>             {"type": "text", "text": "There is a dog and"},
>>>         ],
>>>     },
>>> ]
>>> pipe(text=messages, max_new_tokens=20, return_full_text=False)
[{'input_text': [{'role': 'user',
    'content': [{'type': 'image',
    'url': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'},
    {'type': 'text', 'text': 'Describe this image.'}]},
{'role': 'assistant',
    'content': [{'type': 'text', 'text': 'There is a dog and'}]}],
'generated_text': ' a person in the image. The dog is sitting on the sand, and the person is sitting on'}]
```
__call__transformers.AnyToAnyPipeline.__call__https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/any_to_any.py#L258[{"name": "text", "val": ": str | list[str] | list[dict]"}, {"name": "images", "val": ": typing.Union[str, list[str], list[list[str]], ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "videos", "val": ": typing.Union[str, list[str], list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "audio", "val": ": typing.Union[str, list[str], numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None"}, {"name": "**kwargs", "val": ""}]- **text** (`str`, `list[str]`, `list[dict]`) --
  The text to be used for generation. If a list of strings is passed, the length of the list should be
  the same as the number of images. Text can also follow the chat format: a list of dictionaries where
  each dictionary represents a message in a conversation. Each dictionary should have two keys: 'role'
  and 'content'. 'role' should be one of 'user', 'system' or 'assistant'. 'content' should be a list of
  dictionary containing the text of the message and the type of the message.
- **images** (`str`, `list[str]`, `ImageInput`) --
  The pipeline handles three types of images:
- A string containing a HTTP(s) link pointing to an image
  - A string containing a local path to an image
  - An image loaded in PIL directly

  The pipeline accepts either a single image or a batch of images. Finally, this pipeline also supports
  the chat format (see `text`) containing images and text in this argument.
- **videos** (`str`, `list[str]`, `VideoInput`) --
  The pipeline handles three types of videos:

  - A string containing a HTTP(s) link pointing to a video
  - A string containing a local path to a video
  - A video loaded and decoded to array format

  The pipeline accepts either a single video or a batch of videos. Finally, this pipeline also supports
  the chat format (see `text`) containing videos and text in this argument.
- **audio** (`str`, `list[str]`, `AudioInput`) --
  The pipeline handles three types of audios:

  - A string containing a HTTP(s) link pointing to an audio
  - A string containing a local path to an audio
  - An audio loaded in PIL directly
The pipeline accepts either a single audios or a batch of audios. Finally, this pipeline also supports
  the chat format (see `text`) containing audios and text in this argument.
- **return_tensors** (`bool`, *optional*, defaults to `False`) --
  Returns the tensors of predictions (as token indices) in the outputs. If set to
  `True`, the decoded text is not returned.
- **return_text** (`bool`, *optional*) --
  Returns the decoded texts in the outputs.
- **return_full_text** (`bool`, *optional*, defaults to `True`) --
  If set to `False` only added text is returned, otherwise the full text is returned. Cannot be
  specified at the same time as `return_text`.
- **clean_up_tokenization_spaces** (`bool`, *optional*, defaults to `True`) --
  Whether or not to clean up the potential extra spaces in the text output.
- **continue_final_message(** `bool`, *optional*) -- This indicates that you want the model to continue the
  last message in the input chat rather than starting a new one, allowing you to "prefill" its response.
  By default this is `True` when the final message in the input chat has the `assistant` role and
  `False` otherwise, but you can manually override that behaviour by setting this flag.0A list or a list of list of `dict`Each result comes as a dictionary with the following key (cannot
return a combination of both `generated_text` and `generated_token_ids`):
- **generated_text** (`str`, present when `return_text=True` and `generation_mode="text"`) -- The generated text.
- **generated_audio** (`np.ndarray`, present when `generation_mode="audio"`) -- The generated audio.
- **generated_image** (`PIL.Image.Image`, present when `generation_mode="image"`) -- The generated image.
- **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True` and `generation_mode="text"`) -- The token
  ids of the generated text.
- **input_text** (`str`) -- The input text.

Generate a text given text and optionally multimodal data passed as inputs.

**Parameters:**

model ([PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel)) : The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from [PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel).
Each result comes as a dictionary with the following key (cannot
return a combination of both `generated_text` and `generated_token_ids`):

- **generated_text** (`str`, present when `return_text=True` and `generation_mode="text"`) -- The generated text.
- **generated_audio** (`np.ndarray`, present when `generation_mode="audio"`) -- The generated audio.
- **generated_image** (`PIL.Image.Image`, present when `generation_mode="image"`) -- The generated image.
- **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True` and `generation_mode="text"`) -- The token
  ids of the generated text.
- **input_text** (`str`) -- The input text.
#### transformers.MaskGenerationPipeline[[transformers.MaskGenerationPipeline]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/mask_generation.py#L36)

Automatic mask generation for images using `SamForMaskGeneration`. This pipeline predicts binary masks for an
image, given an image. It is a `ChunkPipeline` because you can separate the points in a mini-batch in order to
avoid OOM issues. Use the `points_per_batch` argument to control the number of points that will be processed at the
same time. Default is `64`.

The pipeline works in 3 steps:
1. `preprocess`: A grid of 1024 points evenly separated is generated along with bounding boxes and point
   labels.
   For more details on how the points and bounding boxes are created, check the `_generate_crop_boxes`
   function. The image is also preprocessed using the `image_processor`. This function `yields` a minibatch of
   `points_per_batch`.
2. `forward`: feeds the outputs of `preprocess` to the model. The image embedding is computed only once.
   Calls both `self.model.get_image_embeddings` and makes sure that the gradients are not computed, and the
   tensors and models are on the same device.

3. `postprocess`: The most important part of the automatic mask generation happens here. Three steps
   are induced:
   - image_processor.postprocess_masks (run on each minibatch loop): takes in the raw output masks,
     resizes them according
   to the image size, and transforms there to binary masks.
   - image_processor.filter_masks (on each minibatch loop): uses both `pred_iou_thresh` and
     `stability_scores`. Also
   applies a variety of filters based on non maximum suppression to remove bad masks.
   - image_processor.postprocess_masks_for_amg applies the NSM on the mask to only keep relevant ones.

Example:
```python
>>> from transformers import pipeline

>>> generator = pipeline(model="facebook/sam-vit-base", task="mask-generation")
>>> outputs = generator(
...     "http://images.cocodataset.org/val2017/000000039769.jpg",
... )

>>> outputs = generator(
...     "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", points_per_batch=128
... )
```

Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)

This segmentation pipeline can currently be loaded from [pipeline()](/docs/transformers/v5.6.2/en/main_classes/pipelines#transformers.pipeline) using the following task identifier:
`"mask-generation"`.

See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=mask-generation).