Speech recognition with CTC runs not reproducible 
Author: bhavitvyamalik
Pull request: no

### System Info

- `transformers` version: 4.30.1
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No



### Who can help?

@sg

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm using official `run_speech_recognition_ctc.py` on a single GPU. I ran this for pretrained `hubert` twice with same seed but every time I get different WER on test set. 

### Expected behavior

Should return same WER when running with same seed.

Comments:
Hi @bhavitvyamalik, 

If running the `run_speech_recognition_ctc.py` script, could you share the command being run, including all argument settings? 

Is the script being run on a single or multiple GPUs? 

cc @sanchit-gandhi 

The script is being run on single GPU. I'm training on Multilingual Librispeech dataset (English version).

```
DOMAINS="mls_en"
PYTHON_FILE=${PROJECT_ROOT}/"dom_finetune/run_speech_recognition_ctc.py"

CUDA_VISIBLE_DEVICES=0 python ${PYTHON_FILE} \
        --model_name_or_path="facebook/hubert-base-ls960" \
        --domains=${DOMAINS} \
        --num_train_epochs="20" \
        --per_device_train_batch_size="4" \
        --per_device_eval_batch_size="8" \
        --gradient_accumulation_steps="2" \
        --preprocessing_num_workers="16" \
        --learning_rate="3e-5" \
        --lr_scheduler_type="constant" \
        --logging_steps="25" \
        --evaluation_strategy="epoch" 
        --save_strategy="epoch" \
        --load_best_model_at_end=true \
        --metric_for_best_model="wer" \
        --greater_is_better=false \
        --text_column_name="transcription" \
        --length_column_name="input_length" \
        --layerdrop="0.0" \
        --save_total_limit="1" \
        --freeze_feature_encoder \
        --chars_to_ignore , ? . ! \
        --output_dir "/disk/scratch1/" \
        --group_by_length \
        --do_train --do_eval --do_predict
```

@bhavitvyamalik Thanks for the additional information. Could you also share the WER results seen after different runs? i.e. how different are they typically? 

WER: 0.6475 and 0.651 using same seed 42. The loss numbers remain very similar initially but after a point (roughly after 2nd epoch) they start differing at 3rd decimal place

Hey @bhavitvyamalik - could you also share the script you're using to fine-tune? Since it differs from the original example script, it's not possible to say whether the non-determinism comes from the 🤗 Trainer, or the data pre-processing. I see that the data arguments in your script differ from those in the example, so would be interested in checking what data pre-processing strategy is employed!

It would also be super helpful to have the dataset as well so that we can run it locally as well for reproducibility

Hi @sanchit-gandhi, I'm using similar data pre-processing given in the official script. Line 412-421 is the only change I've done to the official script to use `audiofolder` functionality of `datasets`. I'm using 10h English data of MLS for training and full dev, test data for validation and testing respectively.

Here is the link to the script: https://gist.github.com/bhavitvyamalik/948d6ca9f42e6c4d70fb8a2f037b4c88

I will upload the dataset in a while to dataset hub. Thank you!

Link to dataset: https://huggingface.co/datasets/bhavitvyamalik/mls_english_10h


Thanks @bhavitvyamalik - running two runs concurrently now:
1. Run 1: https://wandb.ai/sanchit-gandhi/huggingface/runs/0auf9oue?workspace=user-sanchit-gandhi
2. Run 2: https://wandb.ai/sanchit-gandhi/huggingface/runs/jsbzkm4o?workspace=user-sanchit-gandhi

The runs are indeed not identical, e.g. comparing the eval loss:

![Screenshot 2023-07-13 at 09 26 40](https://github.com/huggingface/transformers/assets/93869735/5f7603d8-df90-4c22-b786-f9fa7f9dcb8b)

This is pretty strange behaviour considering we fix the same seed in both cases and use the same training arguments.

cc'ing @muellerzr and @pacman100 here - for context, we're fine-tuning a CTC model for ASR using the examples script [run_speech_recognition_ctc.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py) and the arguments:

<details>
<summary> run_mls.sh </summary>

```bash
#!/usr/bin/env bash

python run_speech_recognition_ctc.py \
        --model_name_or_path="facebook/hubert-base-ls960" \
        --dataset_name "bhavitvyamalik/mls_english_10h" \
        --num_train_epochs="20" \
        --per_device_train_batch_size="4" \
        --per_device_eval_batch_size="8" \
        --gradient_accumulation_steps="2" \
        --preprocessing_num_workers="16" \
        --train_split_name "train" \
        --eval_split_name "eval" \
        --learning_rate="3e-5" \
        --lr_scheduler_type="constant" \
        --logging_steps="25" \
        --evaluation_strategy="epoch" \
        --save_strategy="epoch" \
        --load_best_model_at_end True \
        --metric_for_best_model="wer" \
        --greater_is_better False \
        --text_column_name="transcription" \
        --length_column_name="input_length" \
        --layerdrop="0.0" \
        --save_total_limit="1" \
        --freeze_feature_encoder \
        --chars_to_ignore , ? . ! \
        --output_dir "./" \
        --group_by_length \
        --overwrite_output_dir \
        --do_train \
        --do_eval
```

</details>

However, the training runs are not reproducible, even when we use the same seed. The runs do not give the same eval loss and eval WER between training runs (see above plot). The training loss also diverges after approx 800 training steps (see [logs](https://wandb.ai/sanchit-gandhi/huggingface?workspace=user-sanchit-gandhi)).

Wondering whether there's any non-determinism that we can try and investigate with the new `accelerate` powered trainer? Or whether we put this down to numerical differences?

Alright after leaving the runs to continue for the full length of training, we see that the run 1 and run 2 are to within 0.01 of each other on pretty much all metrics: https://wandb.ai/sanchit-gandhi/huggingface?workspace=user-sanchit-gandhi

So I think we can conclude the seed is set correctly (the differences would be much larger if this wasn't the case). So probably what we're seeing is the effect of numerical differences accumulated over many thousands of ops? I still would have thought the two runs would be exactly the same since I've run them on the same hardware, same env, same seed etc.

Would be interested in hearing whether you agree here both!

@sanchit-gandhi Thanks for digging into this 🕵️‍♂️ ! 

Yes, I agree, it looks there's just some small numerical differences creeping in. Given how tricky these things are to investigate and how small the differences are, it's not something I think is worth investigating further. 

If someone from the community is interested and wants to dig into this more, then we will still welcome links to relevant write-ups or results in this issue. 

Doing a quick run with Transformers v4.27.4 to see whether the Trainer was reproducible to within 0.01 prior to the `accelerate` integration: https://wandb.ai/sanchit-gandhi/huggingface?workspace=user-sanchit-gandhi

If the behaviour is the same as it is on `main` with the `accelerate` back-end, I think we can safely conclude this is an accumulation of numerical errors.

We're looking into it on the accelerate side for fixing. Thanks for the flag

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Update: looks like the differences are approx the same for `transformers==main` as they are `transformers==4.27.4`(see https://wandb.ai/sanchit-gandhi/huggingface?workspace=user-sanchit-gandhi) => we can safely conclude this is an accumulation of numerical errors

(click image below to get zoomed in version)
![Screenshot 2023-08-09 at 18 18 40](https://github.com/huggingface/transformers/assets/93869735/133d3ecf-5188-4825-b279-364730e6c68c)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.