[Whisper] Multilingual Tokeniser uses wrong normaliser
Author: sanchit-gandhi
Pull request: no

### System Info

- `transformers` version: 4.26.0.dev0
- Platform: macOS-12.5-arm64-arm-64bit
- Python version: 3.8.9
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.5.1 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5

### Who can help?

@sanchit-gandhi @ArthurZucker (cc @Vaibhavs10 for info)

### Reproduction

All of the English-only and multilingual Whisper models have `normalizer.json` files in their model repository's on the HF Hub, e.g. for Whisper tiny: https://huggingface.co/openai/whisper-tiny/blob/main/normalizer.json

This means that when we load the tokenisers for any of these models from pre-trained, we default to using the "English Text Normaliser" specified by this `normalizer.json` file:
https://github.com/huggingface/transformers/blob/7319850902ba9b2a44c36ccddd044f98abd9b597/src/transformers/models/whisper/tokenization_whisper.py#L300-L302
and then:
https://github.com/huggingface/transformers/blob/7319850902ba9b2a44c36ccddd044f98abd9b597/src/transformers/models/whisper/tokenization_whisper.py#L488

However, this English text normaliser should **only** be used for English-only models. A separate, "Basic Text Normaliser" should be used in the multilingual setting. This is in accordance with the official Whisper implementation and paper (see Appendix C on page 21 of the [Whisper paper](https://cdn.openai.com/papers/whisper.pdf)).

In short, the English normaliser is **too stringent** for multilingual languages, removing diacritics and other linguistic features that change the meaning of the words:
```python
from transformers import WhisperTokenizer

tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny")

input_str = "Ça va?"
norm_str = tokenizer._normalize(input_str)

print("Un-normalised:", input_str)
print("Normalised:   ", norm_str)
```
**Print Output**:
```
Un-normalised: Ça va?
Normalised:    ca va
```

We should not ever apply normalisation to the ASR output that changes the meaning of the words. This is not allowed when evaluating systems and is undesirable behaviour.

In the Whisper fine-tuning event, we've seen participants using the multilingual models, fine-tuning on multilingual languages, and normalising according to the default normaliser, thus giving erroneous and invalid evaluation results that have had to be repeated.





### Expected behavior

The basic text normaliser does not remove such diacritics, but does lower case and strip punctuation (as intended):
```python
from transformers.models.whisper.english_normalizer import BasicTextNormalizer

normalizer = BasicTextNormalizer()
basic_norm_str = normalizer(input_str)

print("Un-normalised:   ", input_str)
print("Basic normalised:", basic_norm_str)
```
**Print Output:**
```
Un-normalised:    Ça va?
Basic normalised: ça va 
```

We should revert to using the `BasicTextNormalizer` for the multilingual models. We can do this by:
1. Removing the English `normalizer.json` files from the multilingual Whisper models on the HF Hub
2. Defaulting to using the `BasicTextNormalizer` when the normaliser file is `None`

Comments:
I see, that’s indeed correct. Adding it to my whisper todo, this will probably make the tests fail again 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.