fix(tokenizers): accept dict Unigram vocabs in slow tokenizer init
Author: Taranum01
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47075)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47075)
<!-- ci-dashboard-badge:end -->

## Summary
- Add `_normalize_unigram_vocab()` to convert `tokenizer.json` `{token: id}` dict vocabs into the `list[tuple[str, float]]` shape required by the Rust `Unigram` backend
- Apply the normalization in `convert_to_native_format`, `XLMRobertaTokenizer.__init__`, and `XGLMTokenizer.__init__`
- Fixes `AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")`, which still failed after #44452 because that PR only covered a different code path

Fixes #47020

## Root cause
`vesteinn/ScandiBERT`'s `tokenizer.json` stores its Unigram vocab as `{token: id}`. `XLMRobertaTokenizer.__init__` passed that dict straight into `Unigram(vocab=...)`, which expects `(piece, score)` pairs and raised:
```
TypeError: argument 'vocab': 'dict' object cannot be converted to 'Sequence'
```

## Test plan
- [x] Issue repro: `AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")` succeeds and encodes text
- [x] `test_unigram_dict_vocab_does_not_crash_init` for XLM-RoBERTa and XGLM
- [x] `test_from_pretrained_scandibert_with_dict_vocab_in_tokenizer_json`
- [x] Full `tests/models/xlm_roberta/test_tokenization_xlm_roberta.py` — 56 passed, 2 skipped


Made with [Cursor](https://cursor.com)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: xglm, xlm_roberta

Quick update on the failing `pr-ci / tests_non_model / tests_non_model [shard 0/8]` job:

That's a pre-existing CI-image bug, **not caused by this PR**. Details:

- The shard fails with 6 tests in `tests/cli/*` and a stack like:
  ```
  AttributeError: 'HFCliTyperGroup' object has no attribute '_add_completion'
    at /usr/local/lib/python3.10/site-packages/typer/main.py:1174
  ```
- The image installed `typer==0.26.8` (newest). `HFCliTyperGroup` subclasses typer's `TyperGroup` but doesn't override `_add_completion`, which newer typer calls during `typer.main.get_command` setup.
- HuggingFace has already filed this upstream as #47059 ("tests/cli fail with huggingface_hub 1.22.0: 'HFCliTyperGroup' object has no attribute '_add_completion'"). It is being hit on completely unrelated PRs as well (e.g. #47084 RoMa, #47082 keypoint matcher — same shard, same six tests, same error).
- This PR touches only the 5 files already in the diff (xlm_roberta + xglm tokenizer wiring + the helper in `tokenization_utils_tokenizers.py`); it doesn't import or call typer, `HFCliTyperGroup`, or anything in `tests/cli/`.

So `tests_non_model [shard 0/8]` is expected to fail on any PR until #47059 is fixed upstream; nothing to do in this branch.

All other shards dispatched fine. Re-running with `/ci` after the upstream fix should be enough.

Update on the test results:

- ✅ `tests_tokenization [shard 7/8]` now passes (was the only red test touching our diff).
- The remaining `tests_non_model [shard 0/8]` failure is the pre-existing `HFCliTyperGroup._add_completion` issue, unrelated to this PR (see PR-thread comment above and upstream #47059).
- `Check code quality`, repo consistency, security, anti-slop, DCO: all green.

What changed in commit `aeb3985` to fix the shard 7 red:

The test `test_from_pretrained_scandibert_with_dict_vocab_in_tokenizer_json` calls `AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")`. Even when the snapshot is fully cached, `huggingface_hub` needs to write `.lock`/metadata files on the way out — and `tests_tokenization`'s CI image mounts `/mnt/cache` read-only, so the call raises:

```
OSError: [Errno 30] Read-only file system: '/mnt/cache/hub/models--vesteinn--ScandiBERT'
```

That's a CI-environment constraint, not something the fix in #47020 introduced. Two options were viable:

1. Delete the integration test and rely on the in-memory unit test `test_unigram_dict_vocab_does_not_crash_init` that asserts the dict-vocab code path itself doesn't crash — which is what #47020 was actually about.
2. Keep the integration test as @slow (project convention via `RUN_SLOW=1`), so it runs for maintainers locally but doesn't trip the read-only cache in PR CI.

I kept the test by going with option **2** (decorated `@slow` + a docstring explaining the read-only FS constraint). The test still passes when run with `RUN_SLOW=1` (verified locally on a writable FS in 1.18s), so the original repro is preserved for maintainers. The default `pr-ci/tests_tokenization` shards will now skip it as expected.

Net diff: `+61 / -8` across 5 files.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47075)
**Latest run:** [28738055735:2](https://github.com/huggingface/transformers/actions/runs/28738055735)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,301 | **Failures:** 9 | **Duration:** 26h 43m

<!-- ci-dashboard-recap:end -->

Update — final CI recap on commit `aeb3985`:

The matrix has stabilized. Here's the situation:

**Pre-existing CI image / random-flake failures (not from this PR):**

- `tests_non_model [shard 0/8]` → upstream #47059 (typer 0.26.8 / HFCliTyperGroup._add_completion); documented earlier in this thread. Same failure hits `#47084`, `#47082`, `#47074`, `#47073`, `#47072`, etc.
- `tests_torch [shard 0/8]` → `tests/models/bigbird_pegasus/test_modeling_bigbird_pegasus.py::BigBirdPegasusModelTest::test_training_gradient_checkpointing_use_reentrant_false` (`AssertionError` on gradient set).
- `tests_torch [shard 3/8]` → `tests/models/longformer/test_modeling_longformer.py::LongformerModelTest::test_training_gradient_checkpointing_use_reentrant_false` (`AssertionError: … has no gradient`).
- `tests_generate [shard 4/8]` → `tests/models/olmo/test_modeling_olmo.py::OlmoModelTest::test_generate_with_static_cache` (`Generate outputs are not similar enough atol=1e-05`).

None of those test files (`bigbird_pegasus/*`, `longformer/*`, `olmo/*`) overlap with this PR's diff — the patch touches only `xlm_roberta/`, `xglm/`, and `tokenization_utils_tokenizers.py`. None of `#47084`, `#47082`, etc. touch them either; the failures are scattered flake from the random-order shard bucketing + torch gradient-checkpointing strictness + static-cache numerics. They are unrelated to `_normalize_unigram_vocab` and `XLMRobertaTokenizer.__init__`.

**Green checks (~80 of them):** `Check code quality`, `Check repository consistency`, security gate, anti-slop, DCO, `tests_tokenization` (all 8 shards, including the one we fixed earlier in this thread), `tests_custom_tokenizers`, `tests_repo_utils`, `tests_processors`, `tests_exotic_models`, `tests_training_ci`, `tests_peft_integration`, `tests_tensor_parallel_ci`, `pipelines_torch`, `examples_torch`, etc.

`PR CI status` is the umbrella check that flips red when any shard fails, so it inherits the same upstream flake as `tests_non_model`.

**Recommendation for review:** the changes are review-ready. If a maintainer kicks off another run with `/ci` after the upstream #47059 fix lands, the four pre-existing flakes are also highly likely to clear.