# Utilities for Tokenizers

This page lists all the utility functions used by the tokenizers, mainly the class

[`~tokenization_utils_base.PreTrainedTokenizerBase`] that implements the common methods between

[`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] and the mixin

[`~tokenization_utils_base.SpecialTokensMixin`].

Most of those are only useful if you are studying the code of the tokenizers in the library.

## PreTrainedTokenizerBase

[[autodoc]] tokenization_utils_base.PreTrainedTokenizerBase

    - __call__

    - all

## SpecialTokensMixin

[[autodoc]] tokenization_utils_base.SpecialTokensMixin

## Enums and namedtuples

[[autodoc]] tokenization_utils_base.TruncationStrategy

[[autodoc]] tokenization_utils_base.CharSpan

[[autodoc]] tokenization_utils_base.TokenSpan