SpeechT5 cannot read numbers
Author: jeromemassot
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### System Info

transformers == 4.29.0
environment = Colab
Python == 3.10.11
tensorflow == 2.12.0
torch == 2.0.1+cu118
torchaudio == 2.0.2+cu118

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Init a Transformer agent
2. Init a text which contains numbers. For example text = "More than 10 people have been killed by Covid."
3. Call the agent for a text-to-speech (SpeechT5). For example, audio_translated = agent.run("Read out loud the text", text=text)
4. Play the generated audio

The audio blanks all the numbers/digits.

I am suspecting SpeechT5 to behave wrongly as the code generated by the agent seems to be correct.

Good luck :)

### Expected behavior

The audio file should contain numbers/digits indicated in the text.

Comments:
The SpeechT5 tokenizer does not understand numerals. For this to work, the input text should be normalized to "More than ten people ...", with the number spelled out. My guess is that the agent doesn't do this.

Thanks for the reply. It should not be too difficult to ask the LLM to process the text in order to replace all numbers by their litteral equivalents. I will see the agent code to propose a fix.

Since the mapping from numbers -> words is deterministic, we could add this as a pre-processing step in the SpeechT5 tokenizer? E.g. the number "150" always gets mapped to "one-hundred and fifty". IMO this is a pretty easy way of guaranteeing that we have our text formatted as the model expects

We do the opposite in Whisper, where we normalise all spoken words to numbers ("one-hundred and fifty" -> "150"), see
https://github.com/huggingface/transformers/blob/fe34486f129d47abc0dddb39a22b24cdbe52dec8/src/transformers/models/whisper/english_normalizer.py#L110

May be just asking in the prompt of the agent that a pre-processing of input text is needed and should be done by the LLM itself before to be sent to Speech T5. The OpenAI LLM used by the agent by default can do the job, pretty sure about it.

That would fix the issue of SpeechT5 in the context of using transformers agents, but not as a standalone model! E.g. if we wanted to use SpeechT5 according to the example docs or blog post: https://huggingface.co/blog/speecht5#text-to-speech, then passing a numeric value (e.g. "150") still wouldn't work

My thinking is that we can add a normalisation argument to the `processor`/`tokenizer` that handles this for us:
```python
inputs = processor(text="This is a number, 150", return_tensors="pt", normalize=True)
```

Which under the hood converts the text from:
```
This is a number, 150
```
to
```
This is a number, one-hundred and fifty
```

Once we have this, it's trivial to update the transformers agents pipeline to switch on normalisation by default

Yeap I agree that the agent-only solution is not the optimal one. The processor is certainly a better way to fix it.

Keeping this open since I think it would be a valuable addition (especially since SpeechT5 is being used in transformers agents) - would you like to have a go at adding such a pre-processing function @jeromemassot? Happy to help you with the integration!

@sanchit-gandhi you predicted well. I think that it is important to include this pre-processing to bypass the issue of numbers recognition. 🙏

Perfect - do you want to open a PR for this? We can reverse the logic that we use in the Whisper normaliser, where we go from written to numeric (see https://github.com/huggingface/transformers/issues/23480#issuecomment-1557526685)

I sadly won't have time to undertake this PR myself, but maintain that it's a worthwhile update to the library. If anyone in the community would like to pick-up this issue and submit a PR I'd be more than happy to guide you through the integration process and answer any questions. Much of what's needed to be done is outlined in this comment thread already!

Hi @sanchit-gandhi, if no one else has taken this up, I would love to fix this issue in a new PR!

Hey @heytanay - thanks for jumping on here, it's all yours! Feel free to open a PR and tag me - happy to assist with the integration! Think the details of how we can do this are more or less detailed in this thread, but let me know if you have any questions

Thanks, @sanchit-gandhi, I have started working on it. 

Quick question:
Do we need to reverse the entire `EnglishNumberNormalizer` class from the [`models/whisper/english_normalizer.py`](https://github.com/huggingface/transformers/blob/fe34486f129d47abc0dddb39a22b24cdbe52dec8/src/transformers/models/whisper/english_normalizer.py#L94) that you referenced earlier, or can some cases be skipped?



Awesome - thanks for the update @heytanay! I don't think it's strictly necessary to include all the cases there - let's try and cover the main ones though so that the normaliser is fairly generalisable. In the PR description you can include a list of all the cases you kept, and the ones you discarded, and we can decide together whether we think it's exhaustive enough! Having had a quick scan, I think it's worth keeping at least all the number formatting from the normaliser: https://github.com/huggingface/transformers/blob/fe34486f129d47abc0dddb39a22b24cdbe52dec8/src/transformers/models/whisper/english_normalizer.py#L108-L167

I have created a draft PR @sanchit-gandhi: #25447