Wav2Vec2ForCTC fine-tuning best practices
Author: stefan-falk
Pull request: no

I have started to train models based on [this tutorial](https://huggingface.co/blog/fine-tune-wav2vec2-english) (thanks to @patrickvonplaten) and so far everything works. 

> Note: The model I am fine-tuning here is the [`facebook/wav2vec-base`](https://huggingface.co/facebook/wav2vec2-base) model as I am targeting mobile devices.

However, there are still a few details that I am missing here. First off, I noticed that most experiments stagnate at around 30% WER after which the model does not seem to improve anymore. The data is a 2000 hour English dataset on which I get around 10% WER when using my own (streaming) Transducer implementation. I would not expect Wav2Vec2 to perform worse than that.

I am not sure whether the tutorial tells the full story or is just a basic example so I have a few questions on how to improve the model. I'm asking these questions because at this point I am just not sure what I have to write myself and what is already provided by Huggingface and can be dealt with by simply changing a config-object.

### Q: Should we use subwords instead of characters in the vocabulary?

Subwords appear to work great for Transformer-models but I am not sure what the best practice is here with a CTC model.

### Q: Can we/should we apply augmentation techniques (SpecAugment, time-stretching, etc.) during fine-tuning

Since the model is pre-trained, I am not sure whether it's a good idea to apply certain augmentation-techniques to the inputs during fine-tuning. Is there a recommendation how we should do this?

### Q: Should inputs bucketed together in batches by sequence length?

The tutorial appears to batch together audio-samples as they come. However, sometimes bucketing these examples by sequence length may improve results or at least accelerate the experiment. Is there a way we can bucket samples or is this somehow unnecessary?

Comments:
Hey @stefan-falk, 

To answer your questions above:

> Q: Should we use subwords instead of characters in the vocabulary?
> Subwords appear to work great for Transformer-models but I am not sure what the best practice is here with a CTC model.

You can try, but I don't recommend to do it. The input to output ration in Wav2Vec2 is made to work very well with characters.

> Q: Can we/should we apply augmentation techniques (SpecAugment, time-stretching, etc.) during fine-tuning
> Since the model is pre-trained, I am not sure whether it's a good idea to apply certain augmentation-techniques to the inputs during fine-tuning. Is there a recommendation how we should do this?

In my experience SpecAugment and other techniques help quite a bit to improve performance 

> Q: Should inputs bucketed together in batches by sequence length?
> The tutorial appears to batch together audio-samples as they come. However, sometimes bucketing these examples by sequence length may improve results or at least accelerate the experiment. Is there a way we can bucket samples or is this somehow unnecessary?

Yes, you can use `--group_by_length` as is done in all examples usually. See:
https://huggingface.co/patrickvonplaten/wav2vec2-base-timit-fine-tuned/blob/main/run.sh#L24 for https://huggingface.co/patrickvonplaten/wav2vec2-base-timit-fine-tuned

@patrickvonplaten thanks a lot! :) 

> You can try, but I don't recommend to do it. The input to output ration in Wav2Vec2 is made to work very well with characters.

Okay, I'll stick to that. Just a curios question in addition.. when it comes to Chinese we have a pretty large vocabulary. Do you know how we can deal with that or are we just going to have to use a ~5k vocab?

> In my experience SpecAugment and other techniques help quite a bit to improve performance

Definitely. My problem here is that since I am using the pre-trained model `facebook/wav2vec-base` I don't know whether or not I can deviate here. I don't know in detail how this model was exposed to augmentation - there's a chance that deviating from their methods could degrade performance I guess?

> Yes, you can use --group_by_length as is done in all examples usually.

Ah, thank you. I am not running `run_speech_recognition_ctc.py` though as I was following your tutorial and I just realized that `group_by_length` was already set to `True`. I just don't understand then how this works in combination with `per_device_train_batch_size`.

The batch_size simply corresponds to the number of audio inputs (not #number of audio inputs * #number of loss tokens) so with ``--group_by_length`` we group audio inputs of similar length (= #number of loss tokens) together.

Regarding Chinese, that's a good question actually! Maybe @voidful has an idea here?

> Regarding Chinese, that's a good question actually! Maybe @voidful has an idea here?

If you are working in large vocabulary size model like Chinese, you often encounter the problem of homophones. the acoustic model actually lacks the ability to deal with this. Therefore, it is recommended to use the language model to adjust the results, which will greatly improve.

@patrickvonplaten I see. Thanks for that info. But does that mean that the number of samples (audio samples) stays constant no matter how long the input is? So I get 20x 5 seconds as well as 20x 20 seconds in a batch?

The way Tensorflow bucketing works is we can actually vary the number of samples in a batch. With the Transducer I have buckets which are 300ms apart and the longer the maximum length is, the fewer samples will be in a bucket.

In that case I am actually not setting "number of audio samples" per se. Instead, I pass the number of frames (features) I want to see in each batch. So.. the number of frames and therefore the total duration in each batch stays constant.

I've just tried to set `group_by_length=True` and added the field `length` to each example:

```python
yield dict(input_values=input_values, labels=labels, length=input_values.size)
```

However, the data collator still receives samples which vary quite a lot in duration (1sec to 13sec e.g.). This doesn't seem to have an effect - am I missing something out? 

@voidful I see, thank you. Right now I kind of have memory issues with Chinese because of the larger vocabulary 😓 

@patrickvonplaten Since I've already created this issue here: It's just a minor thing but in the tutorial we're setting the `ctc_loss_function` to `"mean"`:

```python
model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-base", 
    ctc_loss_reduction="mean", 
    pad_token_id=processor.tokenizer.pad_token_id,
)
```

Now, in version 4.12.5 we cannot set it like this (same for `pad_token_id`). Instead this argument appears to have moved to the `Wav2Vec2Config` where its default value is `"sum"` however.

Could this have any effect on training/performance whether we set sum or mean here? I'm just looking for ways to improve the training.

And one last thing: Regarding augmentation, _how_ could I implement augmentation e.g. SpecAugment? The `Wav2Vec2FeatureExtractor` expects 1-dimensional raw audio data and the model appears to extract features from that signal. I'm sorry if this is a rather basic question that I should know the answer to but I couldn't make out where to do this except in a preprocessing step on the raw audio.

1. We are still using "mean" as the reduction method as can be seen here: https://github.com/huggingface/transformers/blob/6beae766eeaa0a5656d2a69df849041bfacf4851/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L125
2. A type of SpecAugment is done automatically inside the model as shown here: https://github.com/huggingface/transformers/blob/6beae766eeaa0a5656d2a69df849041bfacf4851/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1284 . By masking time frames with learned vectors you are essentially augmenting the audio data

And regarding the bucketing - yeah the audio lengths might very well vary, but this is usually not a problem. The idea of `group_by_length` is to reduce padding by grouping similar length-ed audio samples together to avoid padding as much as possible.

> @patrickvonplaten I see. Thanks for that info. But does that mean that the number of samples (audio samples) stays constant no matter how long the input is? So I get 20x 5 seconds as well as 20x 20 seconds in a batch?
> 
> The way Tensorflow bucketing works is we can actually vary the number of samples in a batch. With the Transducer I have buckets which are 300ms apart and the longer the maximum length is, the fewer samples will be in a bucket.
> 
> In that case I am actually not setting "number of audio samples" per se. Instead, I pass the number of frames (features) I want to see in each batch. So.. the number of frames and therefore the total duration in each batch stays constant.
> 
> I've just tried to set `group_by_length=True` and added the field `length` to each example:
> 
> ```python
> yield dict(input_values=input_values, labels=labels, length=input_values.size)
> ```
> 
> However, the data collator still receives samples which vary quite a lot in duration (1sec to 13sec e.g.). This doesn't seem to have an effect - am I missing something out?
> 
> @voidful I see, thank you. Right now I kind of have memory issues with Chinese because of the larger vocabulary 😓

Is that happen on evaluation stage? I actually trained on 56 language on 2 3090 and around 200GB memory machine before. I am using a large vocabulary on all language, and all seems to be fine.

> @patrickvonplaten Since I've already created this issue here: It's just a minor thing but in the tutorial we're setting the `ctc_loss_function` to `"mean"`:
> 
> ```python
> model = Wav2Vec2ForCTC.from_pretrained(
>     "facebook/wav2vec2-base", 
>     ctc_loss_reduction="mean", 
>     pad_token_id=processor.tokenizer.pad_token_id,
> )
> ```
> 
> Now, in version 4.12.5 we cannot set it like this (same for `pad_token_id`). Instead this argument appears to have moved to the `Wav2Vec2Config` where its default value is `"sum"` however.
> 
> Could this have any effect on training/performance whether we set sum or mean here? I'm just looking for ways to improve the training.
> 
> And one last thing: Regarding augmentation, _how_ could I implement augmentation e.g. SpecAugment? The `Wav2Vec2FeatureExtractor` expects 1-dimensional raw audio data and the model appears to extract features from that signal. I'm sorry if this is a rather basic question that I should know the answer to but I couldn't make out where to do this except in a preprocessing step on the raw audio.

Here is some tips regarding to the trainer with large vocabulary size:
https://discuss.huggingface.co/t/problems-and-solution-on-trainer/11498

taking argmax of ctc result on evaluation stage can largely reduce the memory usage XD


@patrickvonplaten 

> We are still using "mean" as the reduction method as can be seen here:

I see, but it looks like the `Wav2Vec2Config`, based on the tutorial, sets the loss-reduction. I've tested it and it looks like I have to set it here. However, the result in the end does not seem to vary. Looks like it does not matter whether we use sum or mean here.

> A type of SpecAugment is done automatically inside the model as shown here

I see. Thank you. Here I wonder how much we can vary from the original pre-training. I see that per default `mask_feature_prob` is set to `0.0` in `Wav2Vec2Config`. Cranking that up might hurt the quality since the encoder is frozen? 

But: My question was a bit more general. I was wondering how/where I could implement other methods like time-warping (see [SpecAugment](https://arxiv.org/pdf/1904.08779.pdf)) or noise-mixing etc and whether or not this is a good idea due to the frozen encoder. 

> And regarding the bucketing - yeah the audio lengths might very well vary, but this is usually not a problem. The idea of group_by_length is to reduce padding by grouping similar length-ed audio samples together to avoid padding as much as possible.

In my batches they varied too much which told me something is not working. It looks like bucketing/grouping does not work if the dataset provided is a `torch.utils.data.IterableDataset`. There's a [branch](https://github.com/huggingface/transformers/blob/c4d1fd77fa52d72b66207c6002d3ec13cc36dca8/src/transformers/trainer.py#L646:L662) which appears to prevent this in such a case. 

This here is never reached:
https://github.com/huggingface/transformers/blob/c4d1fd77fa52d72b66207c6002d3ec13cc36dca8/src/transformers/trainer.py#L576

A warning after 

https://github.com/huggingface/transformers/blob/c4d1fd77fa52d72b66207c6002d3ec13cc36dca8/src/transformers/trainer.py#L646

could be an option here.

---

@voidful 

> Is that happen on evaluation stage? I actually trained on 56 language on 2 3090 and around 200GB memory machine before. I am using a large vocabulary on all language, and all seems to be fine.

I have 4x 1080 Ti (each 11 GB VRAM). A dirty-generated vocabulary for Chines has around 3500 different symbols and I am not able to train this model. Not even with `batch_size = 1`.

Thank you for the link. I'll take a look :)

Hey @stefan-falk,

Sorry I think we're getting a bit too many issues / questions thrown together here. Could you try to open a new issue for the `group_by_length` and maybe cc @sgugger and/or @stas00 ? 

Regarding the config, I mean you can freely choose whatever parameter you would like, but in the official examples we set it to `"mean"` as stated above - see: https://github.com/huggingface/transformers/blob/6beae766eeaa0a5656d2a69df849041bfacf4851/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L536

I don't understand:

> Cranking that up might hurt the quality since the encoder is frozen?

Using a value > 0.0 means that we are masking time frames. It's similar to dropout and will help the model generalize better. 

Finally, if you're looking for custom implementations of data augmentation, please use the forum instead: https://discuss.huggingface.co/

Grouping cannot work with an iterable dataset as we can't select elements by indices, this is not a bug, just something that isn't possible :-)

Also note that this whole thread should be on the [forums](https://discuss.huggingface.co/) in my opinion as it's not reporting a bug or discussing a new feature, but asking general questions about training. The wider community would benefit from having it there :-)

This issue got a bit out of hand, I agree. Sorry about that. 😓 

@patrickvonplaten Regarding `group_by_length` I think I got it working. It wasn't just apparent to me that I need to implement `datasets.Dataset` here. 

Regarding augmentation: My fear is, since it's a pre-trained model, that the quality might get worse if I don't use the same parameters during fine-tuning. Since the model was never trained on masked frequencies, this might throw the encoder off and delivers worse results. I know I can test this via experiments but I thought maybe I am lucky and you guys can warn me about not doing that :)

@sgugger well, given a buffer, it is possible to implement it without the need of indices. I don't know for sure but I think this is how Tensorflow's [`bucket_by_sequence_length`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#bucket_by_sequence_length) is implemented. 

Regarding the forums, I was not very lucky there getting answers, which is why I tried it on github:

- https://discuss.huggingface.co/t/constantly-running-out-of-memory-fine-tuning-wav2vec2/12976
- https://discuss.huggingface.co/t/need-help-training-speech2text-from-scratch/12306
- https://discuss.huggingface.co/t/is-there-a-complete-speech2text-example/12234

Anyway, thanks a lot guys! I've learned a thing or two here and I'll try to see how far I can get with that knowledge. I'll keep you updated.

From my side, we can close this issue. :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.