SlowGuess/NAACL_2022_OCR huggingface dataset:
rate. This observation further verifies that the FAt-Net embedding can help in reducing the language dependency of baselines and increase their generalizability on out-of-domain test sets. We observe an absolute improvement of  $2.64\%$  on the Dutch (Non-English) test set after integrating FAtNet-v1 with VGG-M. We observe an absolute improvement of  $3.65\%$  on the Portuguese (Non-English) set after integrating RawNet-2 with FAtNet-v2. Thus, the highest absolute improvements observed in the language-specific test sets were in Non-English test sets (Dutch and Portuguese).
5.3 Linguistic study with augmentation
To get more linguistic insights, we study the performance of standalone FAtNet models using test-time augmentation (TTA) to feed input data to the models. We call the input strategy described in section 4.3 as  $\mathbf{S}_0$ . In TTA Strategy, each input audio recording in the test set is either repeated several times or clipped to make its duration equal to 30-second (Nagrani et al., 2020). We further clip them into 3-second chunks. We create a batch of all possible pairs. So, we get a batch of 100 pairs. Finally, we feed the entire batch to the model (as shown for  $\mathbf{S}_0$ ). For FAtNet-v1, we average out the probabilities in the end. For FAtNet-v2, we compute the average of 100 sets of obtained embedding's to get the
Model	FAtNet Train set	Test Set	EER (%)	Rel. Imp. (%)
VGG-M	-	Voxforge	9.190	-
VGG-M ⊕ FAtNet-v1	VoxCeleb-1 dev	Voxforge	7.665	+16.594%
VGG-M ⊕ FAtNet-v1	VoxCeleb-2 dev	Voxforge	7.618	+17.106%
VGG-M	-	Aishell-1	9.999	-
VGG-M ⊕ FAtNet-v1	VoxCeleb-1 dev	Aishell-1	9.139	+8.601%
VGG-M ⊕ FAtNet-v1	VoxCeleb-2 dev	Aishell-1	6.866	+31.333%
RawNet-2	-	Voxforge	7.012	-
RawNet-2 ⊕ FAtNet-v2	VoxCeleb-2 dev	Voxforge	5.341	+23.831%
RawNet-2	-	Aishell-1	6.202	-
RawNet-2 ⊕ FAtNet-v2	VoxCeleb-2 dev	Aishell-1	3.832	+38.213%
Table 1: Table showing the relative improvements in the performance of VGG-M and RawNet-2 baselines after integration with FAtNet embeddings.
Figure 5: Figure showing the stability of the proposed FAtNet models on out-of-domain test sets. We achieved a better performance using the  $S_{1}$  test-time augmentation strategy as compared to  $S_{0}$ .
Figure 6: Figure showing that baselines' performance degraded on Non-English test sets without domain adaptation. On the contrary, the performance of FAtNet models improved on those sets without adaptation.
final tensor for embedding $_1$ . We do the same for embedding $_2$ . Then, we compute the cosine similarity between final embedding $_1$  and embedding $_2$ . We call this input strategy  $S_1$ .
Observations: Figure 5 shows that we achieved a better performance using  $S_{1}$  as compared to  $S_{0}$ . Interestingly, FAtNet-v2 achieved better performance on out-of-domain (Non-English) test sets than FAtNet-v1. It is reasonable as FAtNet-v2 contains two 4-head attention blocks, whereas FAtNet-v1 contains only a single 8-head attention block. It indicates that an intelligent selection of frame-level features individually from each audio clip enhances the language robustness of the model for out-of-domain sets.
5.4 Qualitative comparison with the baselines
In this experiment, we compare the generalization capabilities of our proposed networks with the base
lines. We observe from Figure 6 that the baseline models performed relatively well on the VoxCeleb-1 test set (mostly English) and LibriSpeech (English) test set. However, the performance of baselines degraded on the other two out-of-domain multilingual test sets, namely, Aishell-1 (Mandarin) and Voxforge (Non-English).
On the contrary, we observe that the performance of FAtNet models improved for out-of-domain multilingual test sets. FAtNet models generalized well to out-of-domain test sets without domain adaptation. It illustrates the language dependency in VGG-M and RawNet-2 baselines. The relatively poor performance of FAtNet models on VoxCeleb-1 and LibriSpeech test sets may be due to other variability issues in these datasets. VoxCeleb-1 is a dataset collected in noisy unconstrained conditions. LibriSpeech corpus (derived from audiobooks) consists of high prosodic variations. It suggests that
frame-attentive networks are generalizable but not robust to noise or prosodic variations.
Intending to investigate which component of FAtNet models makes them language robust, we did an ablation study. FAtNet models contain two parts: the TDNN module and the attention block. We trained a simple TDNN-based model on the VoxCeleb-2 dev set. The architecture of this model remains the same as FAtNet models, except that this TDNN-based model does not contain the attention block. It is observed from Figure 7 and Figure 8 that our proposed FAtNet models outperformed the TDNN model on most of the test sets. The plots suggest that frame-level features make the model language robust. An intelligent selection of features using attention can help in enhancing performance. Thus, our proposed FAtNet models consisting of this TDNN model and attention blocks are language robust.
Interestingly, the TDNN model outperformed FAtNet-v1 and FAtNet-v2 on Aishell-1 and LibriSpeech. FAtNet-v1 outperformed TDNN on LibriSpeech, but FAtNet-v2 could not show that FAtNet-v1 can better handle prosodic variations than FAtNet-v2. It is reasonable as FAtNet-v1 contains a single attention block that selects intelligent features after concatenating the frame-level features of both the audio clips. Thus, the number of dimensions of each frame sent to the FAtNet-v1 attention block is higher than FAtNet-v2. However, FAtNet-v2 performed better on out-of-domain test sets than TDNN, whereas FAtNet-v1 performed poorly on the Aishell-1 test set compared to TDNN. The presence of two separate attention blocks (specific for each audio clip) in FAtNet-v2 makes it more language robust than FAtNet-v1.
This work investigates the cost-effective idea of integrating the lightweight frame-attentive embedding with heavier and stronger baselines to mitigate the linguistic bias in such baselines without adaptation. After comprehensive experimentation on twelve languages and ablation studies, we observed that the proposed method showed significant and consistent improvements in reducing the linguistic bias in the baselines. Some final considerations:
Model complexity: Table 2 illustrates that the FAtNet models consist of fewer parameters as com
Figure 7: Comparing the performance of TDNN model with FAtNet-v1. We used the S0 strategy (FAtNet-v1 version) for both these models.
Figure 8: Comparing the performance of TDNN model with FAtNet-v2. We used the S0 strategy (FAtNet-v2 version) for both these models.
pared to the baselines. Thus, the proposed frame-attentive networks take less time to train. The VGG-M and the RawNet-2 occupy 71.7MB and 53.6MB of disk space. Our proposed FAtNet-v1 and FAtNet-v2 occupy 41MB and 32.6MB of space. Thus, FAtNet models are lighter than the baselines.
Model	#Parameters
VGG-M	17909219
RawNet-2	13379378
FAtNet-v1	10226690
FAtNet-v2	8127490
Table 2: Table showing details about the number of parameters in the proposed networks and the baselines.
Cost-effectiveness: Mitigating the linguistic bias without adaptation is crucial for enhancing the usability of the model across the globe. However, it is also an extremely challenging problem that requires complex decision-making. Consequently, it requires more parameters in the network and some
additional overhead. Below we discuss the overhead of some of the popular methods to mitigate the problem of linguistic bias:
1. Fine-tuning the pre-trained model individually for each language could help mitigate this issue. However, considering that there are approximately 7,000 languages in the world (Huang et al., 2021), it is costly to adapt the models individually for each existing language.
2. Training heavy and highly complex models on extensive datasets could help enhance the model's generalizability to out-of-domain testing. However, this approach requires significant computational overhead and storage space.
FAtNet models have lower complexity than the baseline models. We purposefully integrated our proposed lightweight embeddings with heavier and stronger baselines. We observed significant improvements after integrating the lightweight FAtNet embeddings with the baselines. Hence, with a very little overhead, FAtNet embeddings may help enhance the generalizability of baselines. Therefore, as compared to the overhead of the above two approaches, our proposed approach cost-effectively enhances the usability of baseline models across the globe.
In this paper, we introduced the cost-effective idea of utilizing a lightweight frame-level embedding for reducing linguistic bias in existing speaker verification systems without the overhead of domain adaptation. We also explored applying attention to individual frames to focus on relevant frame-level discriminative information. For an in-depth analysis of our proposed theoretical hypothesis, we proposed two variants of frame-attentive networks: FAtNet-v1 and FAtNet-v2. We investigated the effect of their integration with the baselines for twelve languages. Empirical results showed consistent improvements in the performance of baselines on out-of-domain test sets without domain adaptation after integration with the FAtNet embedding. Qualitative comparison with baselines suggested that the proposed models are comparatively more generalizable. Additionally, we did an ablation study of our proposed networks. It turns out
that frame-level embedding captures less linguistic information from speech than utterance-level embedding. An intelligent selection of features from frames can further improve the performance of speaker verification models.
Our analysis points to vital problems for future work. For instance, it may be worthwhile to explore standalone domain-invariant architectures. This work focuses on the scenario where the input speech recordings within a trial pair are in the same language. However, it may be helpful to explore this idea further for bilingual speakers where the input speech recordings in the trial pair are in different languages.
R. Auckenthaler, M. J. Carey, and J. S. D. Mason. 2001. Language dependency in text-independent speaker verification. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), volume 1, pages 441-444 vol.1.
Yonatan Belinkov, Ahmed Ali, and James Glass. 2019. Analyzing Phonetic and Graphemic Representations in End-to-End Automatic Speech Recognition. In Proc. Interspeech 2019, pages 81-85.
Vincent Brignatz, Jarod Duret, Driss Matrouf, and Mickael Rouvier. 2021. Language adaptation for speaker recognition systems using contrastive learning. In Speech and Computer, pages 91-99, Cham. Springer International Publishing.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline.
Zhengyang Chen, Shuai Wang, and Yanmin Qian. 2020. Adversarial Domain Adaptation for Speaker Verification Using Partially Shared Network. In Proc. Interspeech, pages 3017-3021.
Zhengyang Chen, Shuai Wang, and Yanmin Qian. 2021. Self-supervised learning based domain adaptation for robust speaker verification. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5834-5838.
Roza Chojnacka, Jason Pelecanos, Quan Wang, and Ignacio Lopez-Moreno. 2021. Speakerstew: Scaling to many languages with a triaged multilingual text-dependent and text-independent speaker verification system. CoRR, abs/2104.02125.
Grzegorz Chrupa, Bertrand Higy, and Afra Alishahi. 2020. Analyzing analytical methods: The case of phonology in neural models of spoken language. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4146-4156, Online. Association for Computational Linguistics.
J. S. Chung, A. Nagrani, and A. Zisserman. 2018. Voxceleb2: Deep speaker recognition. In INTER-SPEECH.
Zhifu Gao, Yan Song, Ian McLoughlin, Wu Guo, and Lirong Dai. 2018. An improved deep embedding learning method for short duration speaker verification. In Proc. Interspeech 2018, pages 3578-3582.
Peter Guzewich, Stephen Zahorian, Xiao Chen, and Hao Zhang. 2018. Cross-corpora convolutional deep neural network dereverberation preprocessing for speaker verification and speech enhancement. In Proc. Interspeech 2018, pages 1329-1333.
J. H. L. Hansen and T. Hasan. 2015. Speaker recognition by machines and humans: A tutorial review. IEEE Signal Processing Magazine, 32(6):74-99.
William N. Havard, Jean-Pierre Chevrot, and Laurent Besacier. 2019. Word recognition, competition, and activation in a model of visually grounded speech. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 339-348, Hong Kong, China. Association for Computational Linguistics.
Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann. 2021. Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2443-2459, Online. Association for Computational Linguistics.
Jee-Weon Jung, Hee-Soo Heo, Ju-Ho Kim, Hye-Jin Shim, and Ha-Jin Yu. 2019. Rawnet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification. pages 1268-1272.
Jee-weon Jung, Seung-bin Kim, Hye-jin Shim, Ju-ho Kim, and Ha-Jin Yu. 2020. Improved rawnet with feature map scaling for text-independent speaker verification using raw waveforms. Proc. Interspeech 2020, pages 3583-3587.
Elie Khoury, Laurent Shafey, and Sébastien Marcel. 2014. Spear: An open source toolbox for speaker recognition based on bob. pages 1655-1659.
Na Li, Deyi Tuo, Dan Su, Zhifeng Li, and Dong Yu. 2018. Deep discriminative embeddings for duration robust speaker verification. In Proc. Interspeech 2018, pages 2262-2266.
J. Lyons. Mel frequency cepstral coefficient (mfcc) tutorial. http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstralcoefficients-mfccs/.
Ayoub Malek. 2020. Signal framing. https://superkogito.github.io/blog/ SignalFraming.html.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python.
Ion Moshnoi. All you need is attention — computer vision edition. https://medium.com/@moshnoi2000/all-you-need-is-attention-computer-vision-edition-dbe7538330a4. 04/19/2018.
A. Nagrani, J. S. Chung, and A. Zisserman. 2017. Voxceleb: a large-scale speaker identification dataset. In INTERSPEECH.
Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. 2020. Voxceleb: Large-scale speaker verification in the wild. Computer Speech & Language, 60:101027.
Koji Okabe, Takafumi Koshinaka, and Koichi Shinoda. 2018. Attentive statistics pooling for deep speaker embedding. In Proc. Interspeech 2018, pages 2252-2256.
Kudashev Oleg, Sergey Novoselov, Timur Pekhovsky, Konstantin Simonchik, and Galina Lavrentyeva. 2016. Usage of dnn in speaker recognition: Advantages and problems. In Advances in Neural Networks - ISNN 2016, volume 9719, pages 82-91.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W.
Johan Rohdin, Themos Stafylakis, Anna Silnova, Hossein Zeinali, Lukas Burget, and Oldrich Plchot. 2019. Speaker verification using end-to-end adversarial language adaptation. In ICASSP 2019, pages 6006-6010.
Cory Shain and Micha Elsner. 2020. Acquiring language from speech by learning to remember and predict. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 195-214, Online. Association for Computational Linguistics.
David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018. X-vectors: Robust dnn embeddings for speaker recognition. pages 5329-5333.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Sanjeev Khudanpur Vijayaditya Peddinti, Daniel Povey. 2015. A time delay neural network architecture for efficient modeling of long temporal contexts. In *Interspeech*, pages 3214-3218.
Voxforge.org. Free and open source speech recognition engines (on linux, windows and mac). http://www.voxforge.org/. Accessed 06/25/2014.
Yanfeng Wu, Chenkai Guo, Hongcan Gao, Xiaolei Hou, and Jing Xu. 2020. Vector-Based Attentive Pooling for Text-Independent Speaker Verification. In Proc. Interspeech 2020, pages 936–940.
Wei Xia, Jing Huang, and John H. L. Hansen. 2019. Cross-lingual text-independent speaker verification using unsupervised adversarial discriminative domain adaptation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pages 5816-5820. IEEE.
Ya-Qi Yu, Lei Fan, and Wu-Jun Li. 2019. Ensemble additive margin softmax for speaker verification. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6046-6050.
F. Zhao, H. Li, and X. Zhang. 2019. A robust text-independent speaker verification method based on speech separation and deep speaker. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6101-6105.
Yingke Zhu, Tom Ko, David Snyder, Brian Mak, and Daniel Povey. 2018. Self-attentive speaker embeddings for text-independent speaker verification. In Proc. Interspeech 2018, pages 3573–3577.
A Training Setup
We cropped the silent parts of speech recordings and clipped them into chunks of 3-seconds. We generated MFCCs of 80-dimensions as acoustic inputs to the model using the Librosa library (McFee et al., 2015). We randomly generated separate training sets consisting of 5,25,000 and 23,88,000 trial pairs for models trained on VoxCeleb-1 dev and VoxCeleb-2 dev sets, respectively. The training set consists of an equal number of positives and negatives. We shuffled the training examples before each epoch. The batch size was 128. We trained the proposed models under the joint supervision of softmax loss and center loss. Center loss helps in reducing intra-speaker variations, whereas the softmax loss helps in increasing the
inter-speaker variations.(Li et al., 2018). We use Adam and RMSProp for FAtNet and center loss, respectively (Li et al., 2018; Paszke et al., 2017). For training on VoxCeleb-1 dev, the learning rate was 0.005 and 0.2 for Adam (for FAtNet) and RMSprop (for center loss), respectively. The learning rates were reduced after every ten epochs using step_lr scheduler with the gamma value of 0.5 and 0.3 for Adam and RMSprop, respectively. For training on VoxCeleb-2 dev, the learning rate was as low as 0.0005 for both Adam (for FAtNet) and RMSprop (for center loss), respectively. It is due to more steps being performed in one epoch while training on VoxCeleb-2 dev. We computed the total loss as follows: total_loss := softmax_loss + 0.01 * center_loss (Li et al., 2018). We used GeForce GTX 1080 GPU.
B Effect of training set size on model performance
Figure 9: Figure showing the effect of training set size on the performance of FAtNet-v1. Observation: The FAtNet-v1 model trained on the VoxCeleb-2 dev set performs better than the model trained on the VoxCeleb-1 dev set for most of the test sets. It indicates that increasing the size of the training set can help in improving the performance of speaker verification models.
In this experiment, we study the effect of training set size on the model performance. It can help visualize the improvements in increasing the number of speakers in the training set. For this experiment, we trained separate models of FAtNet-v1 on VoxCeleb-1 dev and VoxCeleb-2 dev respectively. As shown in Figure 9, the model trained on the VoxCeleb-2 dev set outperformed that trained on the VoxCeleb-1 dev set for most of the test set. It is reasonable as the VoxCeleb-2 dev set is more diverse and multilingual than the VoxCeleb-1 dev set. For the LibriSpeech test set, the model trained on
the VoxCeleb-1 dev set performed slightly better than that trained on the VoxCeleb-2 dev set. It is reasonable due to the high proportion of English speech recordings in VoxCeleb-1 and LibriSpeech being an English speech corpus. Thus, we used the FAtNet models trained on the VoxCeleb-2 dev set for experiments (unless explicitly specified otherwise).
C Choice of Input Features
We compared the performance of 80-dimensional MFCCs with 300-dimensional spectrogram (Nagrani et al., 2017) as inputs to FAtNet-v1. As illustrated in Table 3, we achieved better performance using the MFCC features.
Test Set	MFCC	Spectrogram
VoxCeleb-1	14.682	20.668
LibriSpeech	13.527	20.919
Voxforge	10.694	20.274
Aishell-1	9.521	21.143
D Hyper-parameter details for the stacked TDNN
Table 4 illustrates the hyper-parameter detail for the four stacked TDNN layers in the proposed frame-attentive networks. We set the dropout and stride to 0.1 and 1, respectively, for all four layers.
Table 3: Table showing the EER(%) on training FAtNetv1 using MFCC and Spectrogram features, respectively.
Hyper-parameter	#1	#2	#3	#4
Input dimension	80	512	512	512
Output dimension	512	512	512	512
Context-size	3	5	3	1
Batch-norm	False	False	True	True
Table 4: Hyper-parameter detail for the stacked TDNN layers in FAtNet models.