philipp-zettl/vrom-hf-docs huggingface dataset:
extra_special_tokens (list of `str` or `tokenizers.AddedToken`, *optional*) : A list of extra model-specific special tokens. Add them here to ensure they are skipped when decoding with `skip_special_tokens` is set to True. If they are not part of the vocabulary, they will be added at the end of the vocabulary.

split_special_tokens (`bool`, *optional*, defaults to `False`) : Whether or not the special tokens should be split during the tokenization process. Passing will affect the internal state of the tokenizer. The default behavior is to not split special tokens. This means that if `` is the `bos_token`, then `tokenizer.tokenize("") = ['`]. Otherwise, if `split_special_tokens=True`, then `tokenizer.tokenize("")` will be give `['']`.

tokenizer_object (`tokenizers.Tokenizer`) : A `tokenizers.Tokenizer` object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information.
#### transformers.SentencePieceBackend[[transformers.SentencePieceBackend]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_sentencepiece.py#L45)

Base class for SentencePiece-based tokenizers that load from sentencepiece.model files.

Inherits from [PreTrainedTokenizer](/docs/transformers/v5.6.2/en/main_classes/tokenizer#transformers.PythonBackend).

Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading
pretrained tokenizers as well as adding tokens to the vocabulary.

This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the
specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...).

Class attributes (overridden by derived classes)
convert_tokens_to_stringtransformers.SentencePieceBackend.convert_tokens_to_stringhttps://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_sentencepiece.py#L232[{"name": "tokens", "val": ": list"}]
Converts a sequence of tokens (string) in a single string.

**Parameters:**

model_max_length (`int`, *optional*) : The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [from_pretrained()](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.from_pretrained), this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`).

padding_side (`str`, *optional*) : The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name.
truncation_side (`str`, *optional*) : The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name.

chat_template (`str`, *optional*) : A Jinja template string that will be used to format lists of chat messages. See https://huggingface.co/docs/transformers/chat_templating for a full description.

model_input_names (`list[string]`, *optional*) : The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name.

bos_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token representing the beginning of a sentence.

eos_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token representing the end of a sentence.

unk_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token representing an out-of-vocabulary token.
sep_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token separating two different sentences in the same input (used by BERT for instance).

pad_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation.

cls_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token representing the class of the input (used by BERT for instance).

mask_token (`str` or `tokenizers.AddedToken`, *optional*) : A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`.
extra_special_tokens (list of `str` or `tokenizers.AddedToken`, *optional*) : A list of extra model-specific special tokens. Add them here to ensure they are skipped when decoding with `skip_special_tokens` is set to True. If they are not part of the vocabulary, they will be added at the end of the vocabulary.

split_special_tokens (`bool`, *optional*, defaults to `False`) : Whether or not the special tokens should be split during the tokenization process. Passing will affect the internal state of the tokenizer. The default behavior is to not split special tokens. This means that if `` is the `bos_token`, then `tokenizer.tokenize("") = ['`]. Otherwise, if `split_special_tokens=True`, then `tokenizer.tokenize("")` will be give `['']`.
#### save_vocabulary[[transformers.SentencePieceBackend.save_vocabulary]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_sentencepiece.py#L237)

Save the sentencepiece vocabulary (copy original file) to a directory.

**Parameters:**

save_directory (`str`) : The directory in which to save the vocabulary.

filename_prefix (`str`, *optional*) : An optional prefix to add to the named of the saved files.

**Returns:**

``tuple(str)``

Paths to the files saved.
#### transformers.BatchEncoding[[transformers.BatchEncoding]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L193)

Holds the output of the [__call__()](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__),
`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus` and
`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus` methods (tokens, attention_masks, etc).

This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes
utility methods to map from word/character space to token space.
char_to_tokentransformers.BatchEncoding.char_to_tokenhttps://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L549[{"name": "batch_or_char_index", "val": ": int"}, {"name": "char_index", "val": ": int | None = None"}, {"name": "sequence_index", "val": ": int = 0"}]- **batch_or_char_index** (`int`) --
  Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of
  the word in the sequence
- **char_index** (`int`, *optional*) --
  If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the
  sequence.
- **sequence_index** (`int`, *optional*, defaults to 0) --
  If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0
  or 1) the provided character index belongs to.0`int`Index of the token, or None if the char index refers to a whitespace only token and whitespace is
trimmed with `trim_offsets=True`.
Get the index of the token in the encoded output comprising a character in the original string for a sequence
of the batch.

Can be called as:

- `self.char_to_token(char_index)` if batch size is 1
- `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words
are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized
words.

**Parameters:**

data (`dict`, *optional*) : Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.).

encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*) : If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information.
tensor_type (`Union[None, str, TensorType]`, *optional*) : You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization.

prepend_batch_axis (`bool`, *optional*, defaults to `False`) : Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). Note that this parameter has an effect if the parameter `tensor_type` is set, *otherwise has no effect*.

n_sequences (`Optional[int]`, *optional*) : You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization.

**Returns:**

``int``

Index of the token, or None if the char index refers to a whitespace only token and whitespace is
trimmed with `trim_offsets=True`.
#### char_to_word[[transformers.BatchEncoding.char_to_word]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L634)

Get the word in the original string corresponding to a character in the original string of a sequence of the
batch.

Can be called as:

- `self.char_to_word(char_index)` if batch size is 1
- `self.char_to_word(batch_index, char_index)` if batch size is greater than 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words
are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized
words.

**Parameters:**

batch_or_char_index (`int`) : Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string.

char_index (`int`, *optional*) : If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string.
#### convert_to_tensors[[transformers.BatchEncoding.convert_to_tensors]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L673)

Convert the inner content to tensors.

**Parameters:**

tensor_type (`str` or [TensorType](/docs/transformers/v5.6.2/en/internal/file_utils#transformers.TensorType), *optional*) : The type of tensors to use. If `str`, should be one of the values of the enum [TensorType](/docs/transformers/v5.6.2/en/internal/file_utils#transformers.TensorType). If `None`, no modification is done.

prepend_batch_axis (`int`, *optional*, defaults to `False`) : Whether or not to add the batch dimension during the conversion.
#### sequence_ids[[transformers.BatchEncoding.sequence_ids]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L337)

Return a list mapping the tokens to the id of their original sentences:

- `None` for special tokens added around or between sequences,
- `0` for tokens corresponding to words in the first sequence,
- `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly
  encoded.

**Parameters:**

batch_index (`int`, *optional*, defaults to 0) : The index to access in the batch.

**Returns:**

``list[Optional[int]]``

A list indicating the sequence id corresponding to each token. Special tokens added
by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding
sequence.
#### to[[transformers.BatchEncoding.to]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L767)

Send all values to device by calling `v.to(device, non_blocking=non_blocking)` (PyTorch only).

**Parameters:**

device (`str` or `torch.device`) : The device to put the tensors on.

non_blocking (`bool`) : Whether to perform the copy asynchronously.

**Returns:**

`[BatchEncoding](/docs/transformers/v5.6.2/en/main_classes/tokenizer#transformers.BatchEncoding)`

The same instance after modification.
#### token_to_chars[[transformers.BatchEncoding.token_to_chars]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L510)

Get the character span corresponding to an encoded token in a sequence of the batch.

Character spans are returned as a [CharSpan](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.CharSpan) with:

- **start** -- Index of the first character in the original string associated to the token.
- **end** -- Index of the character following the last character in the original string associated to the
  token.

Can be called as:

- `self.token_to_chars(token_index)` if batch size is 1
- `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1

**Parameters:**

batch_or_token_index (`int`) : Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence.
token_index (`int`, *optional*) : If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence.

**Returns:**

`[CharSpan](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.CharSpan)`

Span of characters in the original string, or None, if the token
(e.g. , ) doesn't correspond to any chars in the origin string.
#### token_to_sequence[[transformers.BatchEncoding.token_to_sequence]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L380)

Get the index of the sequence represented by the given token. In the general use case, this method returns `0`
for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair

Can be called as:

- `self.token_to_sequence(token_index)` if batch size is 1
- `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e.,
words are defined by the user). In this case it allows to easily associate encoded tokens with provided
tokenized words.

**Parameters:**

batch_or_token_index (`int`) : Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence.
#### token_to_word[[transformers.BatchEncoding.token_to_word]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L419)

Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch.

Can be called as:

- `self.token_to_word(token_index)` if batch size is 1
- `self.token_to_word(batch_index, token_index)` if batch size is greater than 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e.,
words are defined by the user). In this case it allows to easily associate encoded tokens with provided
tokenized words.

**Parameters:**

batch_or_token_index (`int`) : Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence.

token_index (`int`, *optional*) : If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence.

**Returns:**

``int``
#### tokens[[transformers.BatchEncoding.tokens]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L319)

Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to
integer indices) at a given batch index (only works for the output of a fast tokenizer).

**Parameters:**

batch_index (`int`, *optional*, defaults to 0) : The index to access in the batch.

**Returns:**

``list[str]``

The list of tokens at that index.
#### word_ids[[transformers.BatchEncoding.word_ids]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L361)

Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer.

**Parameters:**

batch_index (`int`, *optional*, defaults to 0) : The index to access in the batch.

**Returns:**

``list[Optional[int]]``

A list indicating the word corresponding to each token. Special tokens added by the
tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word
(several tokens will be mapped to the same word index if they are parts of that word).
#### word_to_chars[[transformers.BatchEncoding.word_to_chars]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L589)

Get the character span in the original string corresponding to given word in a sequence of the batch.

Character spans are returned as a CharSpan NamedTuple with:

- start: index of the first character in the original string
- end: index of the character following the last character in the original string

Can be called as:

- `self.word_to_chars(word_index)` if batch size is 1
- `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1

**Parameters:**

batch_or_word_index (`int`) : Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence

word_index (`int`, *optional*) : If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence.
sequence_index (`int`, *optional*, defaults to 0) : If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to.

**Returns:**

``CharSpan` or `list[CharSpan]``

Span(s) of the associated character or characters in the string. CharSpan
are NamedTuple with:

- start: index of the first character associated to the token in the original string
- end: index of the character following the last character associated to the token in the original
  string
#### word_to_tokens[[transformers.BatchEncoding.word_to_tokens]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/tokenization_utils_base.py#L457)

Get the encoded token span corresponding to a word in a sequence of the batch.

Token spans are returned as a [TokenSpan](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.TokenSpan) with:

- **start** -- Index of the first token.
- **end** -- Index of the token following the last token.

Can be called as:

- `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1
- `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to
  1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words
are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized
words.

**Parameters:**
batch_or_word_index (`int`) : Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence.

word_index (`int`, *optional*) : If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence.

sequence_index (`int`, *optional*, defaults to 0) : If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to.

**Returns:**

`([TokenSpan](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.TokenSpan), *optional*)`

Span of tokens in the encoded sequence. Returns
`None` if no tokens correspond to the word. This can happen especially when the token is a special token
that has been used to format the tokenization. For example when we add a class token at the very beginning
of the tokenization.
# Pipelines

The pipelines are a great and easy way to use models for inference. These pipelines are objects that abstract most of
the complex code from the library, offering a simple API dedicated to several tasks, including Named Entity
Recognition, Masked Language Modeling, Sentiment Analysis, Feature Extraction and Question Answering. See the
[task summary](../task_summary) for examples of use.

There are two categories of pipeline abstractions to be aware about:

- The [pipeline()](/docs/transformers/v5.6.2/en/main_classes/pipelines#transformers.pipeline) which is the most powerful object encapsulating all other pipelines.
- Task-specific pipelines are available for [audio](#audio), [computer vision](#computer-vision), [natural language processing](#natural-language-processing), and [multimodal](#multimodal) tasks.
## The pipeline abstraction[[transformers.pipeline]]

The *pipeline* abstraction is a wrapper around all the other available pipelines. It is instantiated as any other
pipeline but can provide additional quality of life.

Simple call on one item:

```python
>>> pipe = pipeline("text-classification")
>>> pipe("This restaurant is awesome")
[{'label': 'POSITIVE', 'score': 0.9998743534088135}]
```

If you want to use a specific model from the [hub](https://huggingface.co) you can ignore the task if the model on
the hub already defines it:

```python
>>> pipe = pipeline(model="FacebookAI/roberta-large-mnli")
>>> pipe("This restaurant is awesome")
[{'label': 'NEUTRAL', 'score': 0.7313136458396912}]
```

To call a pipeline on many items, you can call it with a *list*.

```python
>>> pipe = pipeline("text-classification")
>>> pipe(["This restaurant is awesome", "This restaurant is awful"])
[{'label': 'POSITIVE', 'score': 0.9998743534088135},
 {'label': 'NEGATIVE', 'score': 0.9996669292449951}]
```
To iterate over full datasets it is recommended to use a `dataset` directly. This means you don't need to allocate
the whole dataset at once, nor do you need to do batching yourself. This should work just as fast as custom loops on
GPU. If it doesn't don't hesitate to create an issue.

```python
import datasets
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset
from tqdm.auto import tqdm

pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h", device=0)
dataset = datasets.load_dataset("superb", name="asr", split="test")
# as we're not interested in the *target* part of the dataset. For sentence pair use KeyPairDataset
for out in tqdm(pipe(KeyDataset(dataset, "file"))):
    print(out)
    # {"text": "NUMBER TEN FRESH NELLY IS WAITING ON YOU GOOD NIGHT HUSBAND"}
    # {"text": ....}
    # ....
```

For ease of use, a generator is also possible:

```python
from transformers import pipeline

pipe = pipeline("text-classification")

def data():
    while True:
        # This could come from a dataset, a database, a queue or HTTP request
        # in a server
        # Caveat: because this is iterative, you cannot use `num_workers > 1` variable
        # to use multiple threads to preprocess data. You can still have 1 thread that
        # does the preprocessing while the main runs the big inference
        yield "This is a test"

for out in pipe(data()):
    print(out)
    # {"text": "NUMBER TEN FRESH NELLY IS WAITING ON YOU GOOD NIGHT HUSBAND"}
    # {"text": ....}
    # ....
```
#### transformers.pipeline[[transformers.pipeline]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/pipelines/__init__.py#L440)

Utility factory method to build a [Pipeline](/docs/transformers/v5.6.2/en/main_classes/pipelines#transformers.Pipeline).

A pipeline consists of:

- One or more components for pre-processing model inputs, such as a [tokenizer](tokenizer),
[image_processor](image_processor), [feature_extractor](feature_extractor), or [processor](processors).
- A [model](model) that generates predictions from the inputs.
- Optional post-processing steps to refine the model's output, which can also be handled by processors.