Fine-tuning wav2vec2 model: eval_loss & eval_wer keep increasing
Author: lgq-liao
Pull request: no

### System Info

- transformers version: 4.22.0.dev0
- Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Both have same issue
- $ pip freeze |grep datasets
      datasets==2.4.0

### Who can help?

@patrickvonplaten
@anton-l
@sanchit-gandhi
@OllieBroadhurst

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. I am fine-tuning the pre-trained model : **facebook/wav2vec2-large-robust-ft-libri-960h** by using customized dataset (3k hours Singapore English dataset). 
2. I split the large dataset into multiple small batches, each batch contained about 100k+ samples. 
3. Load the previous tuned output with next batch of dataset to run the training script at [ctc_finetune.py](https://drive.google.com/file/d/1NogO0G8-RtLGaisfcmBrXh6ESKZbWfZK/view?usp=sharing)
   - First time loaded pre-trained model **facebook/wav2vec2-large-robust-ft-libri-960h** and frist batch of dataset 
4. After that run the evaluation on the tuned model with fixed eval_dataset
5. Loop step 3&4 until end of the last batch dataset. 


- Except the customized dataset, my training script [ctc_finetune.py](https://drive.google.com/file/d/1NogO0G8-RtLGaisfcmBrXh6ESKZbWfZK/view?usp=sharing) which is almost same as original one
- Train and eval log at [train_eval.log](https://drive.google.com/file/d/1nsuWis0r6inuyF80aZDn7FaXActfMAUJ/view?usp=share_link)
- I upload all my scripts and training logs in the link at https://drive.google.com/drive/folders/1M5xE4L_HBxBQynWyl6f1c-tJtat027d1?usp=share_link
- I cannot figure out where was wrong. I am not sure whether it's libs bug or my configuration mistake.   
-  Appricate if someone can help me take look it.

### Expected behavior

**After 20 EPOCH for each train and eval, here is the output as follows:**


dataset batch | epoch | train_samples | train_loss | eval_loss | eval_wer | eval_samples(Fixed for all the eval)
-- | -- | -- | -- | -- | -- | --
1 | 20 | 104398 | 27.24 | 268.3798 | 0.5587 | 76946
2 | 20 | 104211 | 32.74 | 389.0578 | 0.6787 | 76946
3 | 20 | 104223 | 27.24 | 436.5064 | 0.7194 | 76946
4 | 20 | 104174 | 24.86 | 469.7542 | 0.7437 | 76946
5 | 20 | 104018 | 23.01 | 484.5408 | 0.7627 | 76946
6 | 20 | 104158 | 21.79 | 508.0651 | 0.7728 | 76946
7 | 20 | 104166 | 21.21 | 503.9046 | 0.7799 | 76946
8 | 20 | 104280 | 20.44 | 516.5866 | 0.7919 | 76946
9 | 20 | 104111 | 19.6 | 508.3685 | 0.7888 | 76946
10 | 20 | 104073 | 19.31 | 525.6768 | 0.7914 | 76946
11 | 20 | 104144 | 19.04 | 534.6445 | 0.7979 | 76946
12 | 20 | 104298 | 18.8 | 525.5178 | 0.7936 | 76946
13 | 20 | 104230 | 18.62 | 520.3677 | 0.7952 | 76946
14 | 20 | 104053 | 17.95 | 526.2173 | 0.8025 | 76946

- I expected the **eval_loss** and **eval_wer** should decrease gradually.
- The above result was performed train and eval separately. 
- I had tried to run the evaluatoin during the training , the result was similar, which the **eval_loss** and **eval_wer** **increased** unexceptedly as following table:  


epoch | train_loss | eval_loss | eval_wer
-- | -- | -- | --
1 | 138.01 | 58.21 | 0.21
2 | 114.08 | 75.16 | 0.27
3 | 98.49 | 86.66 | 0.31
4 | 85.66 | 95.94 | 0.33
5 | 76.16 | 105.48 | 0.34
6 | 66.63 | 107.92 | 0.34
7 | 60.2 | 118.42 | 0.37
8 | 55.34 | 121.54 | 0.37
9 | 49.62 | 128.89 | 0.37
10 | 43.41 | 137.79 | 0.39
11 | 40.14 | 133.79 | 0.38
12 | 36.3 | 143.32 | 0.4
13 | 33.48 | 144.25 | 0.38
14 | 30.32 | 152.04 | 0.4
15 | 27.34 | 158.57 | 0.4
16 | 24.78 | 153.13 | 0.38
17 | 22.8 | 159.58 | 0.39
18 | 21.36 | 165.38 | 0.39
19 | 20.27 | 167.18 | 0.38
20 | 20.34 | 167.7 | 0.38



Comments:
It's hard to work out the exact issue but every time you run `ctc_finetune.py` you're resetting the learning rate and performing warmup all over again. This means that your training parameters are going to be all over the show so you only really want to run a training script once.

With a dataset that large, you're probably better off [sharding](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) the datasets and then saving the shards to disk in sizes of around 1GB.

```python
num_shards = 50  # or whatever
ds = ... # some large dataset

for shard_idx in range(num_shards)):
    shard = ds.shard(num_shards, shard_idx, contiguous=True)
    shard.save_to_disk(f"shard_{shard_idx}")
```

Then load the dataset by concatenating the shards. Something like:

```python
from datasets import load_from_disk, concatenate_datasets

ds = concatenate_datasets([load_from_disk(shard_fp) for shard_fp in shard_paths])
```

Then running the custom training script again after making these changes or following the guide [here](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2#prepare-data-tokenizer-feature-extractor) and tweaking the notebook.

From a performance perspective, you're probably more likely to get better results with the [XLS-R weights](https://huggingface.co/facebook/wav2vec2-xls-r-300m) so try that first.

Also the [forums](https://discuss.huggingface.co/) are probably a better source of help for this than here.

Also note that CTC training is not very robust, I'd recommend to try out a bunch of different hyper-paremeters. The most important one to get right is usually the learning rate and dropout 

Also cc @sanchit-gandhi 

> ```python
> ds = concatenate_datasets([load_from_disk(shard_fp) for shard_fp in shard_paths])
> ```

Thanks **OllieBroadhurst** and **Patrickvonplaten** for the guide. 

- I had tried to create all the dataset into one ds file, which had no issue to create such ds in my end, expected it was extremely slow (about few days). 
- With the large ds file, then I run the training script above, somehow it  hung and stucked forever at the loading of ds. That's why I split the dataset into small batch. 
- I haven't tried the **shading** soultion, just curiously, by concatenate_datasets, will it be the same effect as creating the one ds file? 
  

As a rule of thumb, using datasets of 1GB or less shouldn't cause problems. Sharding just chunks the dataset up into smaller datasets, not too different to what you were doing before. Replace the line of code where you save your large dataset with the code that splits it into shards and saves each shard. No need to save the whole thing as one large dataset.

The `concatenate_datasets` should work fine. The contents of the dataset wont be loaded into memory, it's only done so while it's being iterated over during training.



Thank you **OllieBroadhurst** for the prompt response.
- Yes I concatenated all the small ds in to one as you suggested.  The process was pretty fast, only took seconds to reach
the line of **data_collator = DataCollatorCTCWithPadding(processor=processor)**, and it was stucking there or the **Trainer initialization** for a long time.
- It took long time to go through this portion of code as follows:
`
//Instantiate custom data collator
    data_collator = DataCollatorCTCWithPadding(processor=processor)

    //Initialize Trainer
    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        tokenizer=feature_extractor,
    )
` 
- It seems it only use single thread to execute the data process in somewhere. Does it possible to speed up the data process ? (eg, by multiple threads)  


This is almost certainly the trainer which is pretty complex so it's hard to tell why. You can check if it's the dataset by using the top few rows, like `train_dataset.select(range(100))` and seeing if a smaller model works. Also check your memory usage maybe.

I'm not 100% what you mean by "data process"?

> This is almost certainly the trainer which is pretty complex so it's hard to tell why. You can check if it's the dataset by using the top few rows, like `train_dataset.select(range(100))` and seeing if a smaller model works. Also check your memory usage maybe.
> 
> I'm not 100% what you mean by "data process"?

- It only takes 7GB out  of 128GB total memory. 
- And 2 out of 32 cores taken,  2GB out of 48GB used for each GPU(total 2) .
-  Distributed launch the training with local_rank=2 as follows : 
    python -m torch.distributed.launch **--nproc_per_node=2** ctc_finetune.py --train --eval -lr 2
- ~20 hours passed, so far, it is still stucking in same steps mentioned above. 
- If I use small dataset, it works. For the size of 100K samples dataset, it will take about **30**minutes to pass through the above steps.
-  I jus felt that the stucking might be caused by the itration of the dataset elements in the training code somewhether. However I did change any code from the libs. No sure if it caused by my configurations
- I had put my model output configs at [Json_Configs](https://drive.google.com/drive/folders/1p19ohigIUxlGyLei72ZfABXVrI611h3I?usp=share_link).  Appreciate if you can have a look on it.  

Start by eliminating any multi-device influence. Try `python -m torch.distributed.launch ctc_finetune.py --train --eval -lr 2`. If that doesn't work, try `python -m ctc_finetune.py --train --eval`.

Thanks OllieBroadhurst
- I will try out your suggestion and let you know the result.
- For the stucking issue, after 21hours, it finally went through the stucking point above, and started the training iteration as follow:
    1%|▎                           | 42500/**4338800** [7:42:00<**590:50:23**,  2.02it/s]Saving model checkpoint to /mnt/workspace/output/train_eval/checkpoint-42500
Configuration saved in /mnt/workspace/output/train_eval/checkpoint-42500/config.json

{'loss': 286.0071, 'learning_rate': 4.9099999999999994e-05, 'epoch': 0.0}
{'loss': 114.5247, 'learning_rate': 0.0002977528714424097, 'epoch': 0.16}
{'loss': 111.7823, 'learning_rate': 0.0002977182757507265, 'epoch': 0.17}
{'loss': 112.0589, 'learning_rate': 0.00029754529729231053, 'epoch': 0.18}
{'loss': 110.4228, 'learning_rate': 0.0002973033350246782, 'epoch': 0.19}
{'loss': 109.3662, 'learning_rate': 0.00029723414364131185, 'epoch': 0.2}

- Currently the estimated training is about 590 hours, I need increase the **train_batch_size** to reduce the time (the GPU load was only taken 13G/48G for each GPU). 
- Is there other way to reduce the training time, other than train_batch_size ?  

You probably won't need the full 590 hours. You would most likely stop when the `eval_loss` starts plateauing. Try the XLS-R model if you aren't already, the loss should converge quicker. Feel free to increase `train_batch_size` but also consider the `fp16` argument which will help things a lot.

You might also want to increase the steps between evaluation which can take a lot of time depending on your eval dataset size.

> Start by eliminating any multi-device influence. Try `python -m torch.distributed.launch ctc_finetune.py --train --eval -lr 2`. If that doesn't work, try `python -m ctc_finetune.py --train --eval`.

Hello **OllieBroadhurst** , thanks for the guiding. 
- I had tried `python -m torch.distributed.launch ctc_finetune.py --train --eval -lr 2` as your suggestion. it ended with same stucking as previously. 
- I traced the caller and identified the stucked point as follow:
   1. `processor = AutoProcessor.from_pretrained(training_args.output_dir)`
   2. `data_collator = DataCollatorCTCWithPadding(processor=processor)`
   3. `trainer = Trainer(****)`
   4. `train_result = trainer.train(resume_from_checkpoint=checkpoint)`
       `-> find_executable_batch_size -> self._inner_training_loop`
      ` ->train_dataloader = self.get_train_dataloader()`   ----- **this was the stuck point, looks it was stucking at DataLoader()** 
- I suspected it might casued by the line of **self._remove_unused_columns(train_dataset, description="training")** which inside of the caller of `get_train_dataloader()` as my dataset has one unused column **input_length**, however, I set the r**emove_unused_columns=False** in TrainingArguments,  and rerun it, the stucking still there. 
- And ideal about the stucking cause ?




> You probably won't need the full 590 hours. You would most likely stop when the `eval_loss` starts plateauing. Try the XLS-R model if you aren't already, the loss should converge quicker. Feel free to increase `train_batch_size` but also consider the `fp16` argument which will help things a lot.
> 
> You might also want to increase the steps between evaluation which can take a lot of time depending on your eval dataset size.

- Yes, fp16 set to True in my script
-  And I increased the `train_batch_size` and `eval_batch_size` to maximize my training PC load.    
- For the model fine-tuning with customized Singapre English dataset, what base model should I started with ? 
- XLS-R, wav2vec2-large-robust-ft-libri-960h or wav2vec2-large-robust, what's the different if I pick up one of them as my base model?   

I would set `group_by_length` to `False` if you haven't already. This can take a very, very long time for large datasets like yours.

`wav2vec2-large-robust` hasn't been fine-tuned, `wav2vec2-large-robust-ft-libri-960h` has been fine-tuned on English (Librispeech). `XLS-R` has been pretrained on 436 000 hours of audio from multiple language. It means that you'll get the best "head start" using those weights.

> I would set `group_by_length` to `False` if you haven't already. This can take a very, very long time for large datasets like yours.
> 
> `wav2vec2-large-robust` hasn't been fine-tuned, `wav2vec2-large-robust-ft-libri-960h` has been fine-tuned on English (Librispeech). `XLS-R` has been pretrained on 436 000 hours of audio from multiple language. It means that you'll get the best "head start" using those weights.

- Thanks so much for the issue troubleshooting. The `group_by_length= True` for my settings currently. I think it could be the cause. Will disable it and give a try.
- Can I used fine tuned model such as `wav2vec2-large-robust-ft-libri-960h` to continue fine-tuning with my own dataset ?


You can go with whatever weights/architecture you like! `wav2vec2-large-robust-ft-libri-960h` is really great for English but I haven't tried it on other languages yet, feel free to give it a shot.

Thank you **OllieBroadhurst** for the advice. 
- Yup, my intention is to do the **incremental** training, in another word, to accumulate the multiple runs of training result to the existing model.  
- In order to achieve the goal above, I am not sure whether it's possible to fine tune the new dataset with the fine-tuned model 
- The answer seems you already gave to me at your first reply : **you're resetting the learning rate and performing warmup all over again**.
- So my question is how can I achieve the **incremental** like training?  
     - In my case, should I prepare all the datasets in one go ? Means start with **wav2vec2-large-robust +  libri-960h dataset + my custermized dataset**


Really great point from @OllieBroadhurst about the learning rate reset! Related to this, it's worth making sure you're reloading your optimiser states if you're resuming training from a checkpoint to avoid the momentum (optimiser) terms being reset each time.

Regarding incremental training, is this purely to save disk space (i.e. download smaller chunks of the data at a time)? Maybe you could try running training using [streaming mode](https://huggingface.co/blog/audio-datasets#streaming-mode-the-silver-bullet)? This way, you won't ever have to download anymore than 1 batch worth of data to your device at a time. You can check out the script [run_speech_recognition_ctc_streaming.py](https://github.com/huggingface/transformers/blob/main/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_streaming.py). It works in much the same way as the non-streaming script, so should look quite familiar! This will enable you to train your model in one go. 

The main changes in using this script revolve around the fact that we don't know the **size** of our dataset a-priori, so we can't specify the number of training epochs. Instead, we have to specify a number of train steps. You can work out how many train steps you require: 1 epoch is `floor( total num samples / train batch size )` samples. 

Hello **Sanchit-gandhi**, thanks for the elaboration. 
- For the incremental training, what I am looking for is **to have a methodology to add on the addition new dataset** to existing pre-trained / fine-tuned model. 
- More specific, for example I want to fine-tune the model **wav2vec2-large-robust-ft-libri-960h** with my own dataset. Here are two options in my mind to run the fine-tuning:
     1. Load the model  wav2vec2-large-robust-ft-libri-960h + my dataset. 
     2. Load the original model **wav2vec2-large-robust** + **libri-960h dataset** + my dataset
- For the **option 1**, my concern is the **learning rate reset** issue
- For the **option 2**, my concern is, since we already have fine-tuned weight `wav2vec2-large-robust-ft-libri-960h`, it's no point to re-run it again. 
- Hope my explanation is clear. Do you have other option to recommend?

Hey @lgq-liao, thanks for the clarification. Indeed if you load the checkpoint [`wav2vec2-large-robust-ft-libri-960h`](https://huggingface.co/facebook/wav2vec2-large-robust-ft-libri-960h) there is little point in fine-tuning on LS 960h. Here, it only really makes sense to fine-tune on your own dataset.

If you're take the fine-tuned checkpoint `wav2vec2-large-robust-ft-libri-960h` and train it further on your additional dataset, you don't need to worry too much about matching the learning rates. I would just treat it as a new fine-tuning run and apply a standard learning rate procedure here. Since your dataset is likely out-of-domain with LS 960h, in effect it's like starting a new training run.


@sanchit-gandhi : got you. Thanks for the guiding. 
@OllieBroadhurst : The dataset loading stuck issue goes away after I set `group_by_length=False ` :+1: 
 

The training was running 140+ hours and it looks good before the **EPOCH 12.36**. 
Would you please tell me what was wrong after the EPOCH 12.36? 
Here is the log as follows: 
```
{'loss': 142.6154, 'learning_rate': 0.00028539729505169863, 'epoch': 0.0}
{'loss': 116.9755, 'learning_rate': 0.0002705168020679468, 'epoch': 1.0}
{'eval_loss': 80.37191009521484, 'eval_wer': 0.07217616514530197, 'eval_runtime': 325.7806, 'eval_samples_per_second': 236.19, 'eval_steps_per_second': 7.382, 'epoch': 1.0}
						.
{'loss': 105.4153, 'learning_rate': 0.0002554974150664697, 'epoch': 2.0}
{'eval_loss': 73.21361541748047, 'eval_wer': 0.06538271103380858, 'eval_runtime': 316.0957, 'eval_samples_per_second': 243.426, 'eval_steps_per_second': 7.608, 'epoch': 2.0}
						.
{'loss': 96.2782, 'learning_rate': 0.0002404780280649926, 'epoch': 3.0}
{'eval_loss': 66.80726623535156, 'eval_wer': 0.06313430751456828, 'eval_runtime': 316.3667, 'eval_samples_per_second': 243.218, 'eval_steps_per_second': 7.602, 'epoch': 3.0}
						.
{'loss': 89.7653, 'learning_rate': 0.00022545891802067946, 'epoch': 4.0}
{'eval_loss': 61.151939392089844, 'eval_wer': 0.06073101679637412, 'eval_runtime': 315.8389, 'eval_samples_per_second': 243.624, 'eval_steps_per_second': 7.615, 'epoch': 4.0}
						.
{'loss': 83.1802, 'learning_rate': 0.00021043939254062035, 'epoch': 5.0}
{'eval_loss': 57.170997619628906, 'eval_wer': 0.0541305368999708, 'eval_runtime': 316.7452, 'eval_samples_per_second': 242.927, 'eval_steps_per_second': 7.593, 'epoch': 5.0}
						.
{'loss': 74.4893, 'learning_rate': 0.00019541972858197931, 'epoch': 6.0}
{'eval_loss': 53.97002410888672, 'eval_wer': 0.056226592354666295, 'eval_runtime': 316.7992, 'eval_samples_per_second': 242.886, 'eval_steps_per_second': 7.592, 'epoch': 6.0}
						.
{'loss': 73.0806, 'learning_rate': 0.0001804002031019202, 'epoch': 7.0}
{'eval_loss': 49.42509841918945, 'eval_wer': 0.04158086508309317, 'eval_runtime': 316.5823, 'eval_samples_per_second': 243.052, 'eval_steps_per_second': 7.597, 'epoch': 7.0}
						.
{'loss': 65.0651, 'learning_rate': 0.0001653810930576071, 'epoch': 8.0}
{'eval_loss': 44.11850357055664, 'eval_wer': 0.040122132365076744, 'eval_runtime': 316.0374, 'eval_samples_per_second': 243.471, 'eval_steps_per_second': 7.61, 'epoch': 8.0}
						.
{'loss': 58.1436, 'learning_rate': 0.00015036170605613, 'epoch': 9.0}
{'eval_loss': 40.9358024597168, 'eval_wer': 0.04316401538715452, 'eval_runtime': 315.3451, 'eval_samples_per_second': 244.006, 'eval_steps_per_second': 7.627, 'epoch': 9.0}						
						.
{'loss': 54.2643, 'learning_rate': 0.00013534231905465286, 'epoch': 10.0}
'eval_loss': 36.22915267944336, 'eval_wer': 0.03583353434814072, 'eval_runtime': 316.2894, 'eval_samples_per_second': 243.277, 'eval_steps_per_second': 7.604, 'epoch': 10.0}
						.
{'loss': 47.4946, 'learning_rate': 0.00012032320901033972, 'epoch': 11.0}
{'eval_loss': 32.77891159057617, 'eval_wer': 0.030601647898231492, 'eval_runtime': 316.6529, 'eval_samples_per_second': 242.998, 'eval_steps_per_second': 7.595, 'epoch': 11.0}
						.
{'loss': 45.0098, 'learning_rate': 0.00010530368353028065, 'epoch': 12.0}
{'eval_loss': 28.909757614135742, 'eval_wer': 0.029566950626531415, 'eval_runtime': 316.2782, 'eval_samples_per_second': 243.286, 'eval_steps_per_second': 7.604, 'epoch': 12.0}
						.
{'loss': 42.008, 'learning_rate': 0.0001001817762186115, 'epoch': 12.34}
{'loss': 40.349, 'learning_rate': 0.00010011267540620383, 'epoch': 12.34}

> {'loss': 41.2924, 'learning_rate': 9.983682607090103e-05, 'epoch': 12.36}

{'loss': 41.5023, 'learning_rate': 9.94907680945347e-05, 'epoch': 12.39}
						.
{'loss': 39.5593, 'learning_rate': 9.042360598227473e-05, 'epoch': 12.99}
{'loss': 40.342, 'learning_rate': 9.035436669128508e-05, 'epoch': 12.99}						
{'loss': 38.5417, 'learning_rate': 9.028512740029543e-05, 'epoch': 13.0}
{'eval_loss': 25.61539649963379, 'eval_wer': 0.03266723374001803, 'eval_runtime': 316.6491, 'eval_samples_per_second': 243.001, 'eval_steps_per_second': 7.595, 'epoch': 13.0}
						.
{'loss': 36.2269, 'learning_rate': 9.021588810930574e-05, 'epoch': 13.0}
{'loss': 37.1206, 'learning_rate': 9.014664881831609e-05, 'epoch': 13.01}
{'loss': 36.0957, 'learning_rate': 9.007754800590842e-05, 'epoch': 13.01}
{'loss': 38.5678, 'learning_rate': 9.000830871491875e-05, 'epoch': 13.02}
  					.
{'loss': 42.091, 'learning_rate': 8.717143648449039e-05, 'epoch': 13.21}
{'loss': 42.3242, 'learning_rate': 8.696427252584932e-05, 'epoch': 13.22}
{'loss': 130.8234, 'learning_rate': 8.689517171344164e-05, 'epoch': 13.22}
{'loss': 585.9929, 'learning_rate': 8.684545790251107e-05, 'epoch': 13.23}

> {'loss': 0.0, 'learning_rate': 8.67762186115214e-05, 'epoch': 13.23}

{'loss': 0.0, 'learning_rate': 8.670697932053174e-05, 'epoch': 13.24}
{'loss': 0.0, 'learning_rate': 8.663774002954209e-05, 'epoch': 13.24}
{'loss': 0.0, 'learning_rate': 8.656850073855243e-05, 'epoch': 13.25}
```


This is odd. It seems like logging changed from once per epoch to many times per epoch - probably based on the number of steps? In my mind this can only be the case if you reran training from an old checkpoint while changing `TrainingArguments`.

The `0.0` loss means that your training became unstable. It's hard to tell why because things seemed to be converging nicely until then. If you _did_ run training again and used new data, then check that none of your target transcripts are longer than the output sequence length of the model (~120 characters I think?) and that there aren't any missing values.

Thanks @OllieBroadhurst . 
> probably based on the number of steps?
- Yes, the `logging_steps=500` in my configuration, it could be the cause of multiple times logging
 > In my mind this can only be the case if you reran training from an old checkpoint while changing TrainingArguments.
- True, in fact, I had changed the `group_by_length=False` once and resumed the running from the exisiting checkpoint. 
- Let me add more new dataset and start a fresh run to see whether the issue goes aways

> The 0.0 loss means that your training became unstable. It's hard to tell why because things seemed to be converging nicely until then. If you did run training again and used new data, then check that none of your target transcripts are longer than the output sequence length of the model (~120 characters I think?) and that there aren't any missing values.
- Currently, I filtered the dataset with maximum length is 6 seconds by the `vectorized_datasets.filter`. May I know how can I map the length in seconds to characters?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The final re-run result looks good as expected.  I'd like to close this issue. Thank you all for the help.