PreTrainedTokenizer requires self.get_vocab() which is no longer implemented
Author: eugenekwaNeuromics
Pull request: no
Labels: name: bug

### System Info

Hi, there is an issue with the PreTrainedTokenizer class in the current transformers v4.56.1. Line 1516 of `tokenization_utils_base.py` curently returns a `NotImplementedError()` for the `PreTrainedTokenizerBase.get_vocab(self)` function. However, PreTrainedTokenizer still requires the `.get_vocab` function at two places in `tokenization_utils.py`:

- Line 546 for `PreTrainedTokenizer._add_tokens(self, new_tokens, special_tokens)`
`current_vocab = self.get_vocab().copy()`

- Line 510 for `PreTrainedTokenizer._update_total_vocab_size(self)`
`self.total_vocab_size = len(self.get_vocab())`

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Bug encountered when attempting to use the `CharacterTokenizer` class for [standalone_hyenadna.py](https://github.com/HazyResearch/hyena-dna/blob/main/standalone_hyenadna.py). The code requires the `PreTrainedTokenizer` class from  `transformers.tokenization_utils`. Problem encountered upon `super().__init__(...)`, traceback thereafter:

.../lib/python3.12/site-packages/transformers/tokenization_utils.py:438 in PreTrainedTokenizer.__init__(self, **kwargs)
.../lib/python3.12/site-packages/transformers/tokenization_utils.py:546 in PreTrainedTokenizer._add_tokens(self, new_tokens, special_tokens)
.../lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1516 in PreTrainedTokenizerBase.get_vocab(self)

### Expected behavior

PreTrainedTokenizerBase no longer implements .get_vocab, but the PreTrainedTokenizer still requires it.

Comments:
Hey! Could you share more details, I don't think we changed this anytime recently! 

Hi -- yes had a look through previous versions of the transformers package (was hoping to monkey patch by just adding in old code), and it seems PreTrainedTokenizerBase never had a working .get_vocab(). The earliest appearance of that function that I could find was Commit [3a40cdf](https://github.com/huggingface/transformers/commit/3a40cdf58d8b7233e561d1bf9e76f2ed02d9c4b7) on 23 October 2020, and even then it was raising a non-implemented error. [Commit 16da877 just the day before did not have the .get_vocab() function.]

```   
 def get_vocab(self) -> Dict[str, int]:
        """
        Returns the vocabulary as a dictionary of token to index.

        :obj:`tokenizer.get_vocab()[token]` is equivalent to :obj:`tokenizer.convert_tokens_to_ids(token)` when
        :obj:`token` is in the vocab.

        Returns:
            :obj:`Dict[str, int]`: The vocabulary.
        """
        raise NotImplementedError()
```

Interestingly, PreTrainedTokenizer (that inherits from PreTrainedTokenizerBase) never relied on the .get_vocab() function until commit 2da8853 on Sep 19, 2023. [The .get_vocab in PreTrainedTokenizerBase was still a NotImplementedError at that point, so it is unclear how PreTrainedTokenizer was meant to work with this function.]
 

Okay, closing then as it's not a bug!