Wav2Vec2PhonemeCTCTokenizer.phonemize keeps the previous language's espeak backend after switching back (stale phonemizer_lang)
Author: BlueCollarChris
Pull request: no
Labels: name: bug

### System Info

transformers 4.57.6, phonemizer 3.3.0, espeak-ng via espeakng-loader 0.2.4,
Python 3.10, Linux.



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("facebook/wav2vec2-xlsr-53-espeak-cv-ft")

a = tok.phonemize("the quick brown fox", phonemizer_lang="en-us")
tok.phonemize("buenos dias amigo", phonemizer_lang="es")
b = tok.phonemize("the quick brown fox", phonemizer_lang="en-us")

print(a)  # ð ə | k w ɪ k | b ɹ aʊ n | f ɑː k s
print(b)  # t e | k i k | β ɾ o w n | f o k s   <-- Spanish phonemization
assert a == b  # fails
```

After one call with a different `phonemizer_lang`, every subsequent call
that passes the ORIGINAL language silently returns phonemes produced by
the OTHER language's espeak voice, for the lifetime of the tokenizer.

### Cause

In `tokenization_wav2vec2_phoneme.py`, `phonemize()` re-creates the
backend when the requested language differs from `self.phonemizer_lang`,
but **never updates `self.phonemizer_lang`**:

```python
if phonemizer_lang is not None and phonemizer_lang != self.phonemizer_lang:
    self.init_backend(phonemizer_lang)   # backend is now `es`
# self.phonemizer_lang still holds the checkpoint default ("en-us")
```

So on the next call with `phonemizer_lang="en-us"`, the comparison
`"en-us" != self.phonemizer_lang` is False, `init_backend` is skipped, and
the cached Spanish backend phonemizes the English text. Any pipeline that
interleaves languages through one tokenizer instance (e.g. forced
alignment over multilingual audio) silently gets wrong-language G2P for
everything after the first switch.

### Expected behavior

Either `init_backend()` or the `phonemize()` branch should record the
active language, e.g.:

```python
if phonemizer_lang is not None and phonemizer_lang != self.phonemizer_lang:
    self.phonemizer_lang = phonemizer_lang
    self.init_backend(phonemizer_lang)
```

(`prepare_for_tokenization` has the same stale-comparison pattern and
should be checked alongside.)

### Workaround

Until fixed, callers can keep the tokenizer's notion of the active
language in sync manually before each call:

```python
tok.phonemizer_lang = lang
tok.init_backend(lang)
phonemes = tok.phonemize(text, phonemizer_lang=lang)
```

or hold one `phonemizer.backend.EspeakBackend` instance per language
outside the tokenizer and bypass its internal caching entirely.


Comments:
Confirmed the bug — `phonemize()` at `tokenization_wav2vec2_phoneme.py:257-260` skips `self.phonemizer_lang = phonemizer_lang` in the re-init branch, so any later call with the original `phonemizer_lang` arg compares equal to the now-stale field and the backend never resets. (`prepare_for_tokenization` at L226-228 already updates the field correctly, so only `phonemize` needs the assignment.)

Per the transformers contribution guidelines I wanted to coordinate before opening anything — happy to take this if you'd prefer not to. Planned scope:

- Add `self.phonemizer_lang = phonemizer_lang` inside the re-init branch in `phonemize()`.
- Add a regression test that interleaves two languages through one tokenizer instance and asserts the original-language output matches the first call (the exact pattern in your repro).

If you've already started, no worries — happy to defer. Let me know which way works.


Opened PR #46616 with a fix.

The diagnosis in the issue is exactly right. In `phonemize()` (line 257-258), `init_backend` is called with the new language but `self.phonemizer_lang` is never updated, so the next call with the original language skips re-init and silently uses the cached wrong backend.

The fix is one line — `self.phonemizer_lang = phonemizer_lang` inside the re-init branch — mirroring the pattern `prepare_for_tokenization` already uses correctly (L226-228).

Also added a regression test (`test_phonemize_lang_switch_roundtrip`) that exercises the exact interleaving pattern from the repro.

cc @eustlb @ebezzam

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Opened a PR for this: #47412

The fix is one line: `self.phonemizer_lang = phonemizer_lang` after `self.init_backend(phonemizer_lang)` in `phonemize()`.

Thanks for the detailed bug report — the root cause analysis and workaround made the fix straightforward.