noamaanMulla-03/transformers-issues huggingface dataset:
fix: bump kernels max version for 0.16 compatibility
Author: webtecnica
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47414)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47414)
<!-- ci-dashboard-badge:end -->

## What this PR does

Fixes compatibility with `kernels` library version 0.16.0.

The `kernels` library released version 0.16.0, but transformers had a strict exclusive upper bound of `<0.16.0` in both the version check and the dependency constraint. This caused the following error when loading models with `use_kernels=True`:

```
ValueError: Kernels are not available. Please install a compatible version (0.15.2 <= version < 0.16.0), e.g. `pip install kernels==0.15.2`
```

## Changes

- `src/transformers/utils/import_utils.py`: Bump `KERNELS_MAX_VERSION` from `"0.16.0"` to `"0.17.0"`
- `src/transformers/dependency_versions_table.py`: Update constraint from `"kernels>=0.15.2,<0.16"` to `"kernels>=0.15.2,<0.17"`

## Who can review?

@vasqu

Fixes #47312

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47414)
**Latest run:** [29689195613:2](https://github.com/huggingface/transformers/actions/runs/29689195613)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 172,161 | **Failures:** 1 | **Duration:** 16h 31m

<!-- ci-dashboard-recap:end -->
fix: handle list input for special_tokens (#47110)
Author: webtecnica
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47416)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47416)
<!-- ci-dashboard-badge:end -->

## What this PR does

Fixes #47110

Adds a TypeError guard in `add_special_tokens` to handle the case when a list is passed as `special_tokens_dict` instead of a dict. Previously, passing a list would raise a cryptic `'list' object has no attribute 'keys'` (or `ValueError: dictionary update sequence element has length...`). Now it raises a clear TypeError with guidance on the expected format.

## Who can review?

@ArthurZucker @itazap

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47416)
**Latest run:** [29689682453:2](https://github.com/huggingface/transformers/actions/runs/29689682453)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 172,139 | **Failures:** 2 | **Duration:** 14h 32m

<!-- ci-dashboard-recap:end -->
fix: restore fast tokenizer thread-safety (#47085)
Author: webtecnica
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47417)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47417)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Fixes #47085: Fast (Rust-backed) tokenizers are not thread-safe anymore.

## Root cause

`set_truncation_and_padding()` calls `&mut self` methods on the underlying `tokenizers::Tokenizer`, while `encode_batch()` and `decode()` call `&self` methods that release the GIL. When multiple threads share a single `PreTrainedTokenizerFast` instance and one thread is mutating truncation/padding settings while another is encoding, the Rust borrow checker raises `RuntimeError: Already borrowed`.

This fires on nearly every call under concurrent load — ~99.7% error rate with 16 threads in the reporter's reproduction script.

## Fix

Added a per-instance `threading.RLock` that serializes the critical section:

1. **`_encode_plus`** — the entire block from `set_truncation_and_padding` through `encode_batch` is now wrapped in `with self._lock`
2. **`_decode`** — the `self._tokenizer.decode()` call is wrapped in `with self._lock`
3. **Pickle compatibility** — `__getstate__` drops the lock (not picklable), `__setstate__` recreates it on unpickle

The lock is initialized in `__init__` after the tokenizer is validated but before any Rust-side mutation.

## Tests

Extended `ReduceMutableBorrowTests` with 7 new tests:
- `test_concurrent_mixed_settings` — 16 threads × 50 iterations with varying truncation/padding (original repro)
- `test_concurrent_encode_with_decode` — interleaved encoding and decoding on the same tokenizer
- `test_batch_encode_concurrent` — concurrent batch encoding with varying settings
- `test_bpe_concurrent_mixed_settings` — BPE tokenizer with 16 threads
- `test_multiple_tokenizer_instances` — multiple tokenizer instances should not interfere
- `test_stress_many_threads_many_iterations` — 64 threads × 100 iterations
- `test_pickle_round_trip` / `test_deepcopy_round_trip` / `test_pickle_does_not_break_thread_safety` — serialization round-trips preserve thread safety

All tests use offline tokenizers (WordLevel + BPE) built in `setUpClass`, no network calls.

## Before submitting

- [ ] This PR fixes an issue (Fixes: #47085)
- [x] I have ensured code style is correct by running `make style`
- [x] I have included appropriate tests
- [x] I have not modified any generated/modular files

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47417)
**Latest run:** [29689783001](https://github.com/huggingface/transformers/actions/runs/29689783001)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->