XLMRobertaTokenizer.__init__ passes dict to Unigram(vocab=...) expecting a Sequence
Author: saattrupdan
Pull request: no

## System Info

```
transformers version: 5.12.1
tokenizers version: 0.22.2
platform: macOS (also reproduced on Linux)
python version: 3.12 / 3.14
```

## Who can help?

@ArthurZucker @itazap

## Information

- [x] My own modified scripts

## Tasks

- [x] My own task or dataset (give details below)

## Reproduction

PR #44452 fixed `KeyError: 0` in `convert_to_native_format` for dict vocabs (issue #44451). However, loading `vesteinn/ScandiBERT` still fails — through a **different code path**. The slow tokenizer constructor `XLMRobertaTokenizer.__init__` passes `self._vocab` (a dict) directly to `Unigram(vocab=...)`, which expects a `Sequence` (list).

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")
```

Result:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../transformers/models/auto/tokenization_auto.py", line 862, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File ".../transformers/tokenization_utils_base.py", line 1744, in from_pretrained
    return cls._from_pretrained(...)
  File ".../transformers/tokenization_utils_base.py", line 1934, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File ".../transformers/models/xlm_roberta/tokenization_xlm_roberta.py", line 81, in __init__
    self._tokenizer = Tokenizer(Unigram(vocab=self._vocab, unk_id=3, byte_fallback=False))
TypeError: argument 'vocab': 'dict' object cannot be converted to 'Sequence'
```

Root cause in `src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py`:

```python
# self._vocab is loaded as a dict from the sentencepiece model file
self._tokenizer = Tokenizer(Unigram(vocab=self._vocab, unk_id=3, byte_fallback=False))
```

The `Unigram` class from the `tokenizers` library expects `vocab` to be a `list[tuple[str, float]]`, not a `dict`.

Suggested fix — convert the dict to a list of tuples before passing to `Unigram`:

```python
if isinstance(self._vocab, dict):
    self._vocab = list(self._vocab.items())
self._tokenizer = Tokenizer(Unigram(vocab=self._vocab, unk_id=3, byte_fallback=False))
```

This is reproducible on the latest release (5.12.1). Discovered while running [EuroEval](https://github.com/EuroEval/EuroEval) evaluations — see [EuroEval issue #1826](https://github.com/EuroEval/EuroEval/issues/1826).

## Expected behavior

`AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")` should load the tokenizer successfully, as it did in transformers 4.57.1. PR #44452 fixed the `convert_to_native_format` code path but this separate path through `XLMRobertaTokenizer.__init__` remains broken.

Comments:
Thank you for the clear repro and root cause, @saattrupdan. I reproduced it on CPU with vesteinn/ScandiBERT on 5.12.1.

I would like to take this. My plan is to convert self._vocab from a dict to a list of (token, score) tuples before passing it to Unigram in XLMRobertaTokenizer.init, keeping the id order the same. I will also add a test that loads a dict vocab tokenizer and checks that it builds without error. Since #44452 only fixed one code path and this is a different one, I will also check if the other similar tokenizers have the same problem and mention them in the PR. I will open the PR soon and link it here.


Picking this up — @ChetanyaRathi mentioned opening a PR a few days ago but I don't see one linked yet. I'll implement the dict→list vocab conversion in `XLMRobertaTokenizer.__init__` (and check similar Unigram tokenizers), add a regression test, and open a PR shortly. Happy to coordinate if you're already working on it.

```python
>>> from transformers import AutoTokenizer, TokenizersBackend
>>> tokenizer = TokenizersBackend.from_pretrained("vesteinn/ScandiBERT")
>>> tokenizer._tokenizer.model
BPE(dropout=None, unk_token="<unk>", continuing_subword_prefix=None, end_of_word_suffix=None, fuse_unk=True, byte_fallback=False, ignore_merges=False, vocab={"
":0, "<pad>":1, "
":2, "<unk>":3, ...
>>> tokenizer = TokenizersBackend.from_pretrained("FacebookAI/xlm-roberta-base")
>>> tokenizer._tokenizer.model
Unigram(unk_id=3, vocab=[("
", 0), ("<pad>", 0), ("
", 0), ("<unk>", 0),  ...
```

`vesteinn/ScandiBERT` has a BPE model tokenizer, not Unigram like an `xlm-roberta` does, so it is not a `xlm-roberta` tokenizer. You can load it with `TokenizersBackend` instead of `AutoTokenizer`, as above!

As well, feel free to open a PR on  https://huggingface.co/vesteinn/ScandiBERT/discussions to update the tokenizer_class in the `tokenizer_config.json` to `PreTrainedTokenizerFast, so that it will load with `TokenizersBackend` in v4 and v5 🤗 

> >>> from transformers import AutoTokenizer, TokenizersBackend
> >>> tokenizer = TokenizersBackend.from_pretrained("vesteinn/ScandiBERT")
> >>> tokenizer._tokenizer.model
> BPE(dropout=None, unk_token="<unk>", continuing_subword_prefix=None, end_of_word_suffix=None, fuse_unk=True, byte_fallback=False, ignore_merges=False, vocab={"
":0, "<pad>":1, "
":2, "<unk>":3, ...
> >>> tokenizer = TokenizersBackend.from_pretrained("FacebookAI/xlm-roberta-base")
> >>> tokenizer._tokenizer.model
> Unigram(unk_id=3, vocab=[("
", 0), ("<pad>", 0), ("
", 0), ("<unk>", 0),  ...
> 
> `vesteinn/ScandiBERT` has a BPE model tokenizer, not Unigram like an `xlm-roberta` does, so it is not a `xlm-roberta` tokenizer. You can load it with `TokenizersBackend` instead of `AutoTokenizer`, as above!
> 
> As well, feel free to open a PR on https://huggingface.co/vesteinn/ScandiBERT/discussions to update the tokenizer_class in the `tokenizer_config.json` to `PreTrainedTokenizerFast, so that it will load with `TokenizersBackend` in v4 and v5 🤗

Aha, thanks for that diagnosis! I've opened a PR now :)

Closing this issue, as I assume that there's nothing to be done on the `transformers` end in that case (other than perhaps a better error message?).