Utilities for Tokenizers

Most of those are only useful if you are studying the code of the tokenizers in the library.

### class transformers.PreTrainedTokenizerBase

[](#transformers.PreTrainedTokenizerBase)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L1494)

( \*\*kwargs )

Base class for [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) and [PreTrainedTokenizerFast](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast).

Handles shared (mostly boiler plate) methods for those two classes.

Class attributes (overridden by derived classes)

-   **vocab\_files\_names** (`Dict[str, str]`) — A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string).
-   **pretrained\_vocab\_files\_map** (`Dict[str, Dict[str, str]]`) — A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file.
-   **max\_model\_input\_sizes** (`Dict[str, Optional[int]]`) — A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or `None` if the model has no maximum input size.
-   **pretrained\_init\_configuration** (`Dict[str, Dict[str, Any]]`) — A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, a dictionary of specific arguments to pass to the `__init__` method of the tokenizer class for this pretrained model when loading the tokenizer with the [from\_pretrained()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.from_pretrained) method.
-   **model\_input\_names** (`List[str]`) — A list of inputs expected in the forward pass of the model.
-   **padding\_side** (`str`) — The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`.
-   **truncation\_side** (`str`) — The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`.

#### \_\_call\_\_

[](#transformers.PreTrainedTokenizerBase.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2487)

( text: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\]\] = Nonetext\_pair: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\], NoneType\] = Nonetext\_target: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\]\] = Nonetext\_pair\_target: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\], NoneType\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0is\_split\_into\_words: bool = Falsepad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = True\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences.

Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels.

#### batch\_decode

[](#transformers.PreTrainedTokenizerBase.batch_decode)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3445)

( sequences: typing.Union\[typing.List\[int\], typing.List\[typing.List\[int\]\], ForwardRef('np.ndarray'), ForwardRef('torch.Tensor'), ForwardRef('tf.Tensor')\]skip\_special\_tokens: bool = Falseclean\_up\_tokenization\_spaces: bool = None\*\*kwargs ) → `List[str]`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.batch_decode.sequences)**sequences** (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`) — List of tokenized input ids. Can be obtained using the `__call__` method.
-   [](#transformers.PreTrainedTokenizerBase.batch_decode.skip_special_tokens)**skip\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not to remove special tokens in the decoding.
-   [](#transformers.PreTrainedTokenizerBase.batch_decode.clean_up_tokenization_spaces)**clean\_up\_tokenization\_spaces** (`bool`, _optional_) — Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`.
-   [](#transformers.PreTrainedTokenizerBase.batch_decode.kwargs)**kwargs** (additional keyword arguments, _optional_) — Will be passed to the underlying model specific decode method.

The list of decoded sentences.

Convert a list of lists of token ids into a list of strings by calling decode.

#### batch\_encode\_plus

[](#transformers.PreTrainedTokenizerBase.batch_encode_plus)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2784)

( batch\_text\_or\_text\_pairs: typing.Union\[typing.List\[str\], typing.List\[typing.Tuple\[str, str\]\], typing.List\[typing.List\[str\]\], typing.List\[typing.Tuple\[typing.List\[str\], typing.List\[str\]\]\], typing.List\[typing.List\[int\]\], typing.List\[typing.Tuple\[typing.List\[int\], typing.List\[int\]\]\]\]add\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0is\_split\_into\_words: bool = Falsepad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = True\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Tokenize and prepare for the model a list of sequences or a list of pairs of sequences.

This method is deprecated, `__call__` should be used instead.

#### build\_inputs\_with\_special\_tokens

[](#transformers.PreTrainedTokenizerBase.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3080)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — The first tokenized sequence.
-   [](#transformers.PreTrainedTokenizerBase.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — The second tokenized sequence.

The model input with special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

This implementation does not add special tokens and this method should be overridden in a subclass.

#### clean\_up\_tokenization

[](#transformers.PreTrainedTokenizerBase.clean_up_tokenization)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3556)

( out\_string: str ) → `str`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.clean_up_tokenization.out_string)**out\_string** (`str`) — The text to clean up.

The cleaned-up string.

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms.

#### convert\_tokens\_to\_string

[](#transformers.PreTrainedTokenizerBase.convert_tokens_to_string)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3432)

( tokens: typing.List\[str\] ) → `str`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.convert_tokens_to_string.tokens)**tokens** (`List[str]`) — The token to join in a string.

The joined tokens.

Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.PreTrainedTokenizerBase.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3060)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — The first tokenized sequence.
-   [](#transformers.PreTrainedTokenizerBase.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — The second tokenized sequence.

The token type ids.

Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids)

Should be overridden in a subclass if the model has a special way of building those.

#### decode

[](#transformers.PreTrainedTokenizerBase.decode)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3479)

( token\_ids: typing.Union\[int, typing.List\[int\], ForwardRef('np.ndarray'), ForwardRef('torch.Tensor'), ForwardRef('tf.Tensor')\]skip\_special\_tokens: bool = Falseclean\_up\_tokenization\_spaces: bool = None\*\*kwargs ) → `str`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.decode.token_ids)**token\_ids** (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`) — List of tokenized input ids. Can be obtained using the `__call__` method.
-   [](#transformers.PreTrainedTokenizerBase.decode.skip_special_tokens)**skip\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not to remove special tokens in the decoding.
-   [](#transformers.PreTrainedTokenizerBase.decode.clean_up_tokenization_spaces)**clean\_up\_tokenization\_spaces** (`bool`, _optional_) — Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`.
-   [](#transformers.PreTrainedTokenizerBase.decode.kwargs)**kwargs** (additional keyword arguments, _optional_) — Will be passed to the underlying model specific decode method.

The decoded sentence.

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces.

Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.

#### encode

[](#transformers.PreTrainedTokenizerBase.encode)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2295)

( text: typing.Union\[str, typing.List\[str\], typing.List\[int\]\]text\_pair: typing.Union\[str, typing.List\[str\], typing.List\[int\], NoneType\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0return\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = None\*\*kwargs ) → `List[int]`, `torch.Tensor`, `tf.Tensor` or `np.ndarray`

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary.

Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`.

#### encode\_plus

[](#transformers.PreTrainedTokenizerBase.encode_plus)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2688)

( text: typing.Union\[str, typing.List\[str\], typing.List\[int\]\]text\_pair: typing.Union\[str, typing.List\[str\], typing.List\[int\], NoneType\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0is\_split\_into\_words: bool = Falsepad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = True\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Tokenize and prepare for the model a sequence or a pair of sequences.

This method is deprecated, `__call__` should be used instead.

#### from\_pretrained

[](#transformers.PreTrainedTokenizerBase.from_pretrained)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L1617)

( pretrained\_model\_name\_or\_path: typing.Union\[str, os.PathLike\]\*init\_inputs\*\*kwargs )

Instantiate a [PreTrainedTokenizerBase](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase) (or a derived class) from a predefined tokenizer.

Passing `use_auth_token=True` is required when you want to use a private model.

Examples:

```

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")


tokenizer = BertTokenizer.from_pretrained("dbmdz/bert-base-german-cased")


tokenizer = BertTokenizer.from_pretrained("./test/saved_model/")


tokenizer = BertTokenizer.from_pretrained("./test/saved_model/my_vocab.txt")


tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", unk_token="<unk>")


assert tokenizer.unk_token == "<unk>"```

#### get\_special\_tokens\_mask

[](#transformers.PreTrainedTokenizerBase.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3525)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → A list of integers in the range \[0, 1\]

Parameters

-   [](#transformers.PreTrainedTokenizerBase.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of ids of the first sequence.
-   [](#transformers.PreTrainedTokenizerBase.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — List of ids of the second sequence.
-   [](#transformers.PreTrainedTokenizerBase.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

Returns

A list of integers in the range \[0, 1\]

1 for a special token, 0 for a sequence token.

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.

Returns the vocabulary as a dictionary of token to index.

`tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab.

#### pad

[](#transformers.PreTrainedTokenizerBase.pad)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2887)

( encoded\_inputs: typing.Union\[transformers.tokenization\_utils\_base.BatchEncoding, typing.List\[transformers.tokenization\_utils\_base.BatchEncoding\], typing.Dict\[str, typing.List\[int\]\], typing.Dict\[str, typing.List\[typing.List\[int\]\]\], typing.List\[typing.Dict\[str, typing.List\[int\]\]\]\]padding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Truemax\_length: typing.Optional\[int\] = Nonepad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Noneverbose: bool = True )

Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch.

Padding side (left/right) padding token ids are defined at the tokenizer level (with `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`).

Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.

If the `encoded_inputs` passed are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however.

#### prepare\_for\_model

[](#transformers.PreTrainedTokenizerBase.prepare_for_model)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3100)

( ids: typing.List\[int\]pair\_ids: typing.Optional\[typing.List\[int\]\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0pad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = Trueprepend\_batch\_axis: bool = False\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for _pair\_ids_ different than `None` and _truncation\_strategy = longest\_first_ or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error.

#### prepare\_seq2seq\_batch

[](#transformers.PreTrainedTokenizerBase.prepare_seq2seq_batch)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3656)

( src\_texts: typing.List\[str\]tgt\_texts: typing.Optional\[typing.List\[str\]\] = Nonemax\_length: typing.Optional\[int\] = Nonemax\_target\_length: typing.Optional\[int\] = Nonepadding: str = 'longest'return\_tensors: str = Nonetruncation: bool = True\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Prepare model inputs for translation. For best performance, translate one sentence at a time.

#### push\_to\_hub

[](#transformers.PreTrainedTokenizerBase.push_to_hub)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/utils/hub.py#L734)

( repo\_id: struse\_temp\_dir: typing.Optional\[bool\] = Nonecommit\_message: typing.Optional\[str\] = Noneprivate: typing.Optional\[bool\] = Noneuse\_auth\_token: typing.Union\[bool, str, NoneType\] = Nonemax\_shard\_size: typing.Union\[int, str, NoneType\] = '10GB'create\_pr: bool = Falsesafe\_serialization: bool = False\*\*deprecated\_kwargs )

Upload the tokenizer files to the 🤗 Model Hub while synchronizing a local clone of the repo in `repo_path_or_name`.

Examples:

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")


tokenizer.push_to_hub("my-finetuned-bert")


tokenizer.push_to_hub("huggingface/my-finetuned-bert")```

#### register\_for\_auto\_class

[](#transformers.PreTrainedTokenizerBase.register_for_auto_class)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3630)

( auto\_class = 'AutoTokenizer' )

Parameters

-   [](#transformers.PreTrainedTokenizerBase.register_for_auto_class.auto_class)**auto\_class** (`str` or `type`, _optional_, defaults to `"AutoTokenizer"`) — The auto class to register this new tokenizer with.

Register this class with a given auto class. This should only be used for custom tokenizers as the ones in the library are already mapped with `AutoTokenizer`.

This API is experimental and may have some slight breaking changes in the next releases.

#### save\_pretrained

[](#transformers.PreTrainedTokenizerBase.save_pretrained)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2078)

( save\_directory: typing.Union\[str, os.PathLike\]legacy\_format: typing.Optional\[bool\] = Nonefilename\_prefix: typing.Optional\[str\] = Nonepush\_to\_hub: bool = False\*\*kwargs ) → A tuple of `str`

Save the full tokenizer state.

This method make sure the full tokenizer can then be re-loaded using the `~tokenization_utils_base.PreTrainedTokenizer.from_pretrained` class method..

Warning,None This won’t save modifications you may have applied to the tokenizer after the instantiation (for instance, modifying `tokenizer.do_lower_case` after creation).

#### save\_vocabulary

[](#transformers.PreTrainedTokenizerBase.save_vocabulary)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2257)

( save\_directory: strfilename\_prefix: typing.Optional\[str\] = None ) → `Tuple(str)`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.save_vocabulary.save_directory)**save\_directory** (`str`) — The directory in which to save the vocabulary.
-   [](#transformers.PreTrainedTokenizerBase.save_vocabulary.filename_prefix)**filename\_prefix** (`str`, _optional_) — An optional prefix to add to the named of the saved files.

Paths to the files saved.

Save only the vocabulary of the tokenizer (vocabulary + added tokens).

This method won’t save the configuration and special token mappings of the tokenizer. Use `_save_pretrained()` to save the whole state of the tokenizer.

#### tokenize

[](#transformers.PreTrainedTokenizerBase.tokenize)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L2275)

( text: strpair: typing.Optional\[str\] = Noneadd\_special\_tokens: bool = False\*\*kwargs ) → `List[str]`

Parameters

-   [](#transformers.PreTrainedTokenizerBase.tokenize.text)**text** (`str`) — The sequence to be encoded.
-   [](#transformers.PreTrainedTokenizerBase.tokenize.pair)**pair** (`str`, _optional_) — A second sequence to be encoded with the first.
-   [](#transformers.PreTrainedTokenizerBase.tokenize.add_special_tokens)**add\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not to add the special tokens associated with the corresponding model.
-   [](#transformers.PreTrainedTokenizerBase.tokenize.kwargs)**kwargs** (additional keyword arguments, _optional_) — Will be passed to the underlying model specific encode method. See details in [**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__)

The list of tokens.

Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`.

#### truncate\_sequences

[](#transformers.PreTrainedTokenizerBase.truncate_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3236)

( ids: typing.List\[int\]pair\_ids: typing.Optional\[typing.List\[int\]\] = Nonenum\_tokens\_to\_remove: int = 0truncation\_strategy: typing.Union\[str, transformers.tokenization\_utils\_base.TruncationStrategy\] = 'longest\_first'stride: int = 0 ) → `Tuple[List[int], List[int], List[int]]`

Truncates a sequence pair in-place following the strategy.

### class transformers.SpecialTokensMixin

[](#transformers.SpecialTokensMixin)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L778)

( verbose = True\*\*kwargs )

A mixin derived by [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) and [PreTrainedTokenizerFast](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) to handle specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access these special tokens in a model-independent manner and allow to set and update the special tokens.

#### add\_special\_tokens

[](#transformers.SpecialTokensMixin.add_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L858)

( special\_tokens\_dict: typing.Dict\[str, typing.Union\[str, tokenizers.AddedToken\]\]replace\_additional\_special\_tokens = True ) → `int`

Parameters

-   [](#transformers.SpecialTokensMixin.add_special_tokens.special_tokens_dict)**special\_tokens\_dict** (dictionary _str_ to _str_ or `tokenizers.AddedToken`) — Keys should be in the list of predefined special attributes: \[`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`\].
    
    Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them).
    
-   [](#transformers.SpecialTokensMixin.add_special_tokens.replace_additional_special_tokens)**replace\_additional\_special\_tokens** (`bool`, _optional_,, defaults to `True`) — If `True`, the existing list of additional special tokens will be replaced by the one specified in `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is updated. In the former case, the tokens will NOT be removed from the tokenizer’s full vocabulary - they are only being flagged as non-special tokens.

Number of tokens added to the vocabulary.

Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary).

Note,None When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer.

In order to do that, please use the [resize\_token\_embeddings()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.resize_token_embeddings) method.

Using `add_special_tokens` will ensure your special tokens can be used in several ways:

-   Special tokens are carefully handled by the tokenizer (they are never split).
-   You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts.

When possible, special tokens are already registered for provided pretrained models (for instance [BertTokenizer](/docs/transformers/v4.30.0/en/model_doc/bert#transformers.BertTokenizer) `cls_token` is already registered to be :obj_’\[CLS\]’_ and XLM’s one is also registered to be `'
'`).

Examples:

```
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2Model.from_pretrained("gpt2")

special_tokens_dict = {"cls_token": "<CLS>"}

num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
print("We have added", num_added_toks, "tokens")

model.resize_token_embeddings(len(tokenizer))

assert tokenizer.cls_token == "<CLS>"```

#### add\_tokens

[](#transformers.SpecialTokensMixin.add_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L952)

( new\_tokens: typing.Union\[str, tokenizers.AddedToken, typing.List\[typing.Union\[str, tokenizers.AddedToken\]\]\]special\_tokens: bool = False ) → `int`

Parameters

-   [](#transformers.SpecialTokensMixin.add_tokens.new_tokens)**new\_tokens** (`str`, `tokenizers.AddedToken` or a list of _str_ or `tokenizers.AddedToken`) — Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string token to let you personalize its behavior: whether this token should only match against a single word, whether this token should strip all potential whitespaces on the left side, whether this token should strip all potential whitespaces on the right side, etc.
-   [](#transformers.SpecialTokensMixin.add_tokens.special_tokens)**special\_tokens** (`bool`, _optional_, defaults to `False`) — Can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or \[MASK\] are usually not lower-cased for instance).
    
    See details for `tokenizers.AddedToken` in HuggingFace tokenizers library.
    

Number of tokens added to the vocabulary.

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary and and will be isolated before the tokenization algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore not treated in the same way.

Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer.

In order to do that, please use the [resize\_token\_embeddings()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.resize_token_embeddings) method.

Examples:

```
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")

num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"])
print("We have added", num_added_toks, "tokens")

model.resize_token_embeddings(len(tokenizer))```

#### sanitize\_special\_tokens

[](#transformers.SpecialTokensMixin.sanitize_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L846)

( ) → `int`

The number of tokens added in the vocabulary during the operation.

Make sure that all the special tokens attributes of the tokenizer (`tokenizer.mask_token`, `tokenizer.cls_token`, etc.) are in the vocabulary.

Add the missing ones to the vocabulary if needed.

### class transformers.tokenization\_utils\_base.TruncationStrategy

[](#transformers.tokenization_utils_base.TruncationStrategy)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L126)

( valuenames = Nonemodule = Nonequalname = Nonetype = Nonestart = 1 )

Possible values for the `truncation` argument in [PreTrainedTokenizerBase.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__). Useful for tab-completion in an IDE.

### class transformers.CharSpan

[](#transformers.CharSpan)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L138)

( start: intend: int )

Parameters

-   [](#transformers.CharSpan.start)**start** (`int`) — Index of the first character in the original string.
-   [](#transformers.CharSpan.end)**end** (`int`) — Index of the character following the last character in the original string.

Character span in the original string.

### class transformers.TokenSpan

[](#transformers.TokenSpan)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L151)

( start: intend: int )

Parameters

-   [](#transformers.TokenSpan.start)**start** (`int`) — Index of the first token in the span.
-   [](#transformers.TokenSpan.end)**end** (`int`) — Index of the token following the last token in the span.

Token span in an encoded string (list of tokens).