Why are certain models with a higher WER (on the eval set) performing better or as good as  models with a lower WER – when tested on the test set?
Author: drishtishrrma
Pull request: no

Hello @patrickvonplaten and @anton-l  !

I had two weird observations while evaluating my models, one of which showed that the model with the higher WER clearly outperformed the one with the lower WER (So strange!). In another scenario, when evaluated on the test set, the model with higher WER (on the eval set) fared "almost" as good as the model with low WER. I'm curious as to what the likely causes of these observations are. 

I compared two models (Model A against model B) in each case. So, in a nutshell, I’ll be mentioning two models in each case for the sake of comparison.

**Outline of the issue:**

Case 1: Model A and Model B — trained for Guarani language (gn) —- CV8 dataset — model with higher  WER on eval set clearly outperformed the model with lower WER, when tested on the test set.

Case 2: Different Model A and Model B — trained for Assamese language (as) —- CV8 dataset —- model with higher WER performed nearly as good as model with low WER, when tested on the test set.


**Detailed Description:** 

**Case 1:** 
Models A and B were trained for the Guarani (**gn**) – Common Voice 8 dataset. 

**It should be noted that the library versions of Datasets and Transformers utilized while training both models were different.** **However, those models were tested in literally the same environment setting ie. the same version of Datasets and Transformers**. And it turns out that when tested on the test set, the model with a higher WER on the eval set outperformed the model with a lower WER.

Case 1: Model A: [https://huggingface.co/lgris/wav2vec2-xls-r-300m-gn-cv8](url)

Case 1: Model B: [https://huggingface.co/DrishtiSharma/wav2vec2-large-xls-r-300m-gn-k1](url)

**Comparison in Tabular form**: 
![image](https://user-images.githubusercontent.com/82262606/154126093-d6bd46c7-adfe-496c-a791-f2d597a8c7c2.png)


**Case 2**: In this case, different model A and model B were trained for Assamese language (as) – Common Voice 8 dataset. 

It's worth noting that **the Datasets version used in both the cases were same ie. Datasets 1.18.3. However the Transformers version was different.** And it turns out that when tested on the test set, the model with a higher WER performed nearly as good as the model with a lower WER. 

Case 2: Model A: [https://huggingface.co/DrishtiSharma/wav2vec2-large-xls-r-300m-as-g1 ](url)

Case 2: Model B: [https://huggingface.co/infinitejoy/wav2vec2-large-xls-r-300m-assamese-cv8 ](url)

**Comparison in Tabular form**: 


![image](https://user-images.githubusercontent.com/82262606/154129659-b9a49041-3366-4b4d-a52f-2670a971b914.png)


I'm curious as to what could be the underlying causes of such observations. I'd be glad if I get your inputs in this regard.
Currently, I'm a super novice in the ASR field. I literally calculated the WER for the very first time in this sprint itself, so please pardon me if I'm witnessing these observations due to some mistake on my part.

Thanks in advance... 


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

Hi @LysandreJik!

I opened this issue since it was asked.. A bug in Datasets library was reported a few days back due to which we gotta new version now. I was actually referring to that bug when I was comparing the Datasets version utilised in different cases. This issue was opened in an attempt to detect the underlying causes of the models' anomalous behaviour.

But I'm guessing I didn't elucidate properly. Perhaps, I'd have used the default format to explain the issue, however I  thought it's easier this way since there were too many variables.. :/

@LysandreJik :

Anyway, if you still believe that the aforementioned issue/post is forum- worthy, I have no objection in taking it down and publishing it on forums.
Multi-channel communication is turning chaotic for me as well. And I've a strong intuition that I've almost found the underlying causes of anomalous behaviour of certain models, albeit I'm still refining my points and I'm almost there... I might get my points evaluated on the forum itself! No worries! 

Lastly, I apologize if the above issue is too naive/inappropriate for the platform. I've transitioned from the Patent Research and Analytics domain, and haven't utilized GitHub a lot more. However, I hope to do better in future. 

Thanks!


Ah, sorry, if it was asked by a member of the team then please ignore my message above. Thank you for doing the effort of opening an issue, even if I mentioned the forum would be a better place we definitely appreciate you trying to solve the problem. The issue is neither too naive nor inappropriate, it was just more likely to get an answer on the forum :)

Hey @drishtishrrma! I've re-evaluated the models over the past week and haven't found any significant model or config-related differences that would explain these weird coincidences :slightly_smiling_face: 
However, note that the Guarani and Assamese languages have **very** little data in the Common Voice dataset (40m and 1h of validated data respectively). This means that the validation and testing metrics can be highly uncorrelated (e.g. there are just 293 train, 159 val and 93 test sentences for Guarani). In these low-resource conditions even changing the random seed before training can sometimes drastically change the final model.
Hopefully that makes sense, and thank you for the evaluation report! Let me know if you have any questions :slightly_smiling_face: 

@LysandreJik : Yeah, I got your POV. Thanks for your advice, I appreciate it... 

Hi @anton-l! 

Thank you so much for your time and efforts, I appreciate it. 

You've made some really excellent points there! Thank you so much! 

I almost figured this out yesterday, but after re-reading your reply, I realized my analogy was slightly different in comparison to your ideas. It'd be fantastic if you could give me your thoughts on it, especially if you think it's a possibility. I  was envisioning the aforementioned issue as being very analogous to a real-life scenario. 


**My Analogy**: Assuming a student is preparing for a competitive exam in which he will be tested on five subjects (analogous to entire training dataset): General Science, Social Science, Maths, English, and Computer Science **(syllabus of all the subjects collectively is analogous to entire training dataset)**. Further assuming that the candidate excelled at Maths, General Science, and Social Science but struggled to learn/perform well/generalize in English and Computer Science (**struggled to generalize for a portion of entire training dataset**). **His overall score was good (let's compare this to WER now) due to high contribution of marks obtained in the subjects he learnt/generalized well during studying (training).** However, his overall good score (analogous to WER) doesn't tell a true story--- his overall good score (WER) conceals how poorly he performed in the remaining two subjects. **(analogous to a portion of the entire training dataset)** ---> And what if he's put into an environment where he's to deal with English and Computer Science only (his weak-links), this environment exposes his weak-links outrightly. Regardless of how good his overall score was previously, the WER will turn out to be poor if tested in an  environment where the model (person) has to deal with its/his weak-links alone. 

And this behaviour is more pronounced in a low-resource scenario since the model is less robust in such scenario because it was trained on very little data, and additionally it didn't even generalize well for a certain type of distribution during the training process.

Although the two models were trained on the same data, but due to different batch_sizes perhaps one generalized better!? They say, " small batch_size (s) achieve better generalization, perhaps it has something to do with it.
 "random_seed" has a major role to play, in general; but since it was 42 in all the cases, the effect became uniformized. The default value is also 42, so it's a little less likely that it was tweaked before training. (At least, I didn't tweak at all; can't speak for the other two models that were trained by others.)

These are some random thoughts, please feel free to correct. I'd be wrong. 

Thanks! 


@anton-l:

Small batch_size achieves better generalization, this claim needs to be tested a little. Recently I came across a couple tweets touting the benefits of larger batch_size. I'll be putting it to test soon. 

**To simplify things and avoid making claims I haven't personally tested, I'd like to generalize the statement I stated above as follows:**
Because different batch size (s) were utilized, it's likely that one of the models generalized better for the limited data available.

**In short, let say,  if a model generalized well for a portion of the training data which closely matches the environment in which we're gonna be testing, then obviously it'll perform better. On the contrary, it might not perform well if subjected to envio for which it didn't generalize well during the training.**

And it's very likely that if the aforementioned models are exposed to a different kind of environment (distribution), the outcome could be the polar opposite of what we've seen so far. This behaviour/effect is more pronounced in the low-resource scenario, imo.