Fine-tuned Robust Wav2Vec 2.0 models
Author: Nithin-Holla
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description
The pretrained Robust Wav2Vec 2.0 model is already available on the Hugging Face model hub (https://huggingface.co/facebook/wav2vec2-large-robust). Facebook also released two fine-tuned models -- one which is fine-tuned on Librispeech and another which is fine-tuned on Switchboard. Would be good to have these fine-tuned models on the hub as well.

## Open source status

* [x] the model weights are available on fairseq: https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/README.md


@patrickvonplaten 

Comments:
Adding them now :-) 

BTW @Nithin-Holla, 

It could be a cool project to fine-tune the `wav2vec2-large-robust` checkpoint on the [AMI dataset](https://groups.inf.ed.ac.uk/ami/corpus/datasets.shtml) since the model was not pretrained on AMI and AMI is a conversational dataset instead of a read-out corpus. Would be pretty interesting to see what performance can be achieved there (also compared to "non-robust" Wav2Vec2. 

If you'd be interested in such a project, let me know I'd be more than happy to help you there :-)

@patrickvonplaten Awesome, thanks for the speedy addition!

Sure, I'd be interested in working on fine-tuning on the AMI dataset :)

Awesome, I'll send you a mail :-) 

BTW, there is still a problem with the just published checkpoints: https://github.com/pytorch/fairseq/issues/3799

Hi @Nithin-Holla @patrickvonplaten Wondering if this thread went anywhere?  I'm attempting to finetune Hubert-large on the AMI dataset, would be interested to see where you guys got to and share results.

Yeah, we didn't manage to get good results yet sadly with AMI - it's mainly due AMI not being chunked by default.

Thanks @patrickvonplaten. I think It would be cool to make a robust spoken-to-written-language engine.  I'm thinking we could supplement the AMI corpus, eg. youtube, or even make our own spoken wikipedia/similar.  If you have started a working group on this, let me know. Feel free to send me an email. :)

From my experiments, even though WER isn't "great," I see the finetuned model picking features of meeting conversation, which encourages me to see possibility here.  I think the AMI data mimics adult spoken language better (than just reading of text, aka librispeech), and _together_ is like how humans learn language (from hearing -> reading + writing).  



ps. I had chunked AMI to 10-second chunks, using actually the scripts you already published on HG.

Hey @i-am-neo,

We've run quite some experiments here: https://huggingface.co/ami-wav2vec2, but didn't get super good results so far compared to other research papers.
I think it's mainly due the way we've chunked AMI. Think we need to align ourselves with how other people chunked the data.

Here is how the data should be chunked (didn't manage to take a look into this yet - sadly):

> The first question we have is whether IHM corresponds to all 4 individual headset audio files are used, *i.e.* the  Individual headsets 120M four individual WAV headsets data or whether it corresponds to the single headset mix, *i.e.*, the Headset mix 30M single wav file (on https://groups.inf.ed.ac.uk/ami/download/)

> The Kaldi recipe that we wrote for the paper(s) uses separate channels for each headset (see: [here](https://github.com/kaldi-asr/kaldi/blob/master/egs/ami/s5/local/ami_download.sh)), though I would be surprised if the mix-headset variant resulted in statistically different WERs, saving tons of bandwidth (of course, one never knows until you the experiment). The segmentations we use for separate individual headsets are of course compatible with the mixed channel waveform.

> The second question that we have is how the long audio clips are exactly chunked. Sadly we couldn't really find an "official" pre-processing script and t[here](https://github.com/kaldi-asr/kaldi/blob/master/egs/ami/s5/local/ami_xml2text.sh) is very little exact information on how the data is chunked.  Do you guys have an official preprocessing script that you could maybe share? 

> The script here starts with the original AMI annotations in XML format. The AMI data comes with manual segmentations and timings for interpunction signs, and this is how we broke the long utterances. One caveat here, for end2end models you probably do this anyways as decoding graphs are much smaller than in HMM days, thus decoding time for these will not be an issue. That script can either query textual annotations from the AMI-provided JAVA tool, or download exported version in textual form. In the Kaldi recipe we did not want to make a dependency on JAVA thus we follow from exported files by default, though you can grasp from the script how to do so if you wish.


Hi @patrickvonplaten, thanks. I got similar loss and wer results with two flavors of hubert (3 epochs, max 10-sec chunks, single-headset).
Were your experiment results run with the 20-sec max chunks? Which version of the dataset?

I suspect there's more to the results than how we chunked.  One can find in the single-headset version -
1. two speakers speaking over each other
2. and text like `i d i don't thi i don't think that it would be a a structural weakness` and `yes that's wi uh this will definitely`.  
Examples [here](https://colab.research.google.com/drive/1vSAabtxv_4HHi3mEdht4B6ud60djtj1z?usp=sharing).
Some of the timings also seem a bit off, though I have to find time to look into it further.

What are your thoughts?

I agree with chunking by `interpunction` - it seems more "natural" to me, though I grouped short phrases into a minimum of 6 words if possible.

We used the processing as described here: https://huggingface.co/datasets/ami#dataset-preprocessing

Think we should apply the official Kaldi preprocessing though. Agree the target text is quite noisy 

Hi @patrickvonplaten
Yes, the preprocess steps I had used for training was almost verbatim from [https://huggingface.co/datasets/ami#dataset-preprocessing](https://huggingface.co/datasets/ami#dataset-preprocessing), except for setting `MAX_LENGTH_IN_SECONDS = 10.0`.  

I've taken a closer look at the segment timestamps, and think they are actually inaccurate, likely due to 

> The AMI data comes with _manual_ segmentations and timings for interpunction signs, and this is how we broke the long utterances

(italics mine). Have a look at some examples [here](https://colab.research.google.com/drive/1hUrREy7kV1HuUqJlNvzJqbr_KwZZBX2Y?usp=sharing).

It looks to me that cleanly labeled audio is freely mixed in with inaccurately-labeled ones, enough that we can't use AMI by itself without someone going in to clean the dataset manually.  Still, I think it a helpful exercise to find other datasets similar to AMI to train a robust transcription model, either by supplementing AMI, or replacing it. You?

Agree that it'd be very important to test Wav2Vec2 on more "real-world" / "robust" data. Sadly, I don't know any other dataset besides AMI that could fit that use-case. Do you have any ideas?

I'm thinking Youtube.

We can try training first on some English Youtube public AMI-like videos.  (I suspect we'd need to hook up a "spoken" version of a language model for decoding as well, but this is probably the easier part of the task).  If the WER/whatever-metric-we-choose from training looks promising, we could take a subset of [AudioSet](https://research.google.com/audioset///index.html) (those identified as human speech) and build out a larger dataset.

To me, Hubert-large seems a more robust acoustic model than Wav2Vec2 to start with. Let me know what you think.