SlowGuess/NAACL_2022_OCR huggingface dataset:
Divya V Sharma
IIIT-Delhi
divyas@iiitd.ac.in
Arun Balaji Buduru
arunb@iiitd.ac.in
Linguistic bias in Deep Neural Network (DNN) based Natural Language Processing (NLP) systems is a critical problem that needs attention. The problem further intensifies in the case of security systems, such as speaker verification, where fairness is essential. Speaker verification systems are intelligent systems that determine if two speech recordings belong to the same speaker. Such human-oriented security systems should be usable by diverse people speaking varied languages. Thus, a speaker verification system trained on speech in one language should generalize when tested for other languages. However, DNN-based models are often language-dependent. Previous works explore domain adaptation to fine-tune the pre-trained model for out-of-domain languages. Fine-tuning the model individually for each existing language is expensive. Hence, it limits the usability of the system. This paper proposes the cost-effective idea of integrating a lightweight embedding with existing speaker verification systems to mitigate linguistic bias without adaptation. This work is motivated by the theoretical hypothesis that attentive-frames could help generate language-agnostic embeddings. For scientific validation of this hypothesis, we propose two frame-attentive networks and investigate the effect of their integration with baselines for twelve languages. Empirical results suggest that frame-attentive embedding can cost-effectively reduce linguistic bias and enhance the usability of baselines.
Mitigating the linguistic bias in Deep Neural Network (DNN) based models is one of the critical challenges in Natural Language Processing (NLP). The linguistic bias, specifically in the security systems, such as speaker verification models, is a far more critical problem requiring much research. Speaker verification systems are biometric authentication systems that use speech signals to authenticate a speaker. These systems use the fact that every
speaker has unique traits in their voice (Hansen and Hasan, 2015). Such systems have real-world applications in e-commerce, forensics, law, business, and access control mechanisms (Hansen and Hasan, 2015). These systems can be text-dependent or text-independent (Hansen and Hasan, 2015). Text-independent speaker verification systems are more user-friendly than text-dependent systems. These systems authenticate a speaker without any constraint on the content of speech.
However, speaker verification models often tend to be language-dependent (Auckenthaler et al., 2001). The reason is that a robust speaker verification system would require memory to analyze the sequential speech data and capture relevant discriminatory information. Memory helps in remembering past information. Remembering the past and predicting the future can contribute to linguistic content in the embedding (Shain and Elsner, 2020). Therefore, the generated embedding for speaker verification may contain linguistic detail.
Language-dependent speaker verification models perform relatively well on test sets containing speech recordings in the same language as the training set. However, the performance of these systems degrades on test sets containing speech recordings in different languages. The majority of the publicly available speech datasets are in English. It is a tedious task to get labeled datasets for various low-resource languages. Most of the previous works use domain adaptation to improve the performance of speaker verification models only for a limited set of languages (Rohdin et al., 2019; Xia et al., 2019; Chen et al., 2020). It is also costly to fine-tune a pre-trained speaker verification model individually for each existing language. Further, studies show that the linguistic content in the embedding increase with the temporal scope of representations (Chrupaña et al., 2020).
Our proposed work is based on the theoretical hypothesis that frame-level features contain less lin
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1247 - 1258
guistic information due to the low-temporal scope of frames. Thus, frame-level features may help generate a language-agnostic embedding. Furthermore, an intelligent selection of frame-level features may help in enhancing the model's generalizability to out-of-domain testing. We aim to address the problem of language dependency in text-independent speaker verification systems cost-effectively, without the overhead of domain adaptation. We propose an idea that incorporates a lightweight embedding with existing speaker verification systems which may help in improving the generalizability of these systems to out-of-domain testing. To scientifically validate the theoretical hypothesis, we present and investigate two variants of frame-attentive networks: FAtNet-v1 and FAtNet-v2. Our proposed models accept two speech recordings as input and determine if they belong to the same speaker. The speakers in the trial pair may be unknown. We qualitatively compare the generalization ability of our proposed models with two strong baselines on four publicly available data sets. We perform quantitative experiments on 12 languages to assess the integration of our proposed FAtNet embeddings with the baselines on publicly available out-of-domain test sets without domain adaptation. We have released the code<sup>1</sup> to encourage more research on this problem.
1. Investigate cognitive ideas such as attention, residual connection for memory, and learning parameters to generate language-agnostic embeddings.
2. To validate the theoretical hypothesis scientifically, propose two novel frame-attentive networks: FAtNet-v1 and FAtNet-v2.
3. Perform qualitative and quantitative experiments for twelve languages using two strong baselines and four publicly available datasets.
Language dependency in speaker verification: The current state-of-the-art explores deep neural networks (DNN) to solve speaker recognition problems (Hansen and Hasan, 2015; Li et al., 2018; Jung et al., 2020, 2019; Nagrani et al., 2017; Snyder et al., 2018; Nagrani et al., 2020; Guzewich et al., 2018; Zhao et al., 2019; Gao et al., 2018). However, most DNN-based feature extractors are
language-dependent (Oleg et al., 2016). Language dependency can make the system less usable as users may belong to different geographic locations and speak varied languages. It is challenging to get labeled datasets for various low-resource languages (Brignatz et al., 2021). Moreover, when tested on multilingual datasets and features, the models that show consistent behavior may be helpful in other applications (such as code-switching) through information sharing (Belinkov et al., 2019). We know that there are approximately 7,000 languages in the world (Huang et al., 2021). One of the critical challenges in Natural Language Processing (NLP) is to develop techniques to overcome this linguistic bias and enhance the usability of the model across the globe (Huang et al., 2021).
Recent works: Transfer learning is a solution to address the problem of domain mismatch. However, it is challenging to get labeled datasets for various low-resource languages (Brignatz et al., 2021). Recent works investigate adversarial domain adaptation techniques for solving cross-lingual speaker verification problems (Rohdin et al., 2019; Xia et al., 2019; Chen et al., 2020; Brignatz et al., 2021). However, most of these approaches can improve the performance of speaker verification models for a limited set of languages as these approaches require an additional overhead of domain adaptation.
To the best of our knowledge, (Chojnacka et al., 2021) is the closest work related to our problem statement where the authors attempt to reduce linguistic bias in speaker verification without domain adaptation. In (Chojnacka et al., 2021), the authors suggest that training a speaker verification model in multiple languages can increase its generalizability to out-of-domain languages. However, they trained the model on an extensive training set consisting of 1,96,000 speakers and 2,06,18,000 utterances. Training on such an extensive dataset requires significant computational requirements, often not feasible in a realistic scenario. In addition to that, their proposed work involves a combination of text-dependent and text-independent speaker verification systems. Our proposed method involves integrating a lightweight embedding with the existing text-independent speaker verification models to reduce linguistic bias in those systems.
Linguistic components in a frame: Speech signals are non-stationary, and hence they are divided into frames. A speech signal is assumed to be stationary within a frame (Malek, 2020). The tem-
'https://github.com/vdivyas/FAtNet.git
poral scope of a frame is usually a few milliseconds. Researchers have investigated the role of temporal scope in their study of phonology in neural models (Chrupa et al., 2020). Studies show that Representational Similarity Analysis (RSA) applied to local representations results in lower correlations between phonemes and neural activation patterns (Chrupa et al., 2020). Thus, considering the low temporal scope of frames, it is intuitive that they carry less linguistic information.
Theoretical Hypothesis: Our proposed work investigates the theoretical hypothesis that utterance-level embedding captures more linguistic information than frame-level embedding. Therefore, a frame-level embedding can be more language-robust than an utterance-level embedding. This abstract knowledge of frame-level features can allow the model to learn relevant discriminatory information from frames and ignore the linguistic information from speech. Researchers have stated that some frames tend to be more critical than others for the final-encoded representation of speech (Havard et al., 2019). Attention mechanisms are popular in state-of-the-art speaker verification models (Zhu et al., 2018; Okabe et al., 2018). We explore the effectiveness of attention for an intelligent selection of features within a frame.
To investigate the theoretical hypothesis, we propose the following two variants of Frame-Attentive Networks: FAtNet-v1 and FAtNet-v2.
As illustrated in Figure 1 and Figure 2, the time-delay neural network (TDNN) paths are similar in both the FAtNet versions and the details are as follows: The model accepts a pair of Mel-frequency cepstral coefficients $^{2}$  (MFCC) for speaker verification (Chen et al., 2020; Zhu et al., 2018; Khoury et al., 2014). MFCCs finetune the features to what human beings hear (Lyons). Let  $d$  be the dimension of input MFCCs and  $l_{1}$  and  $l_{2}$  be the number of frames in the given pair for speaker verification. The values of  $l_{1}$  and  $l_{2}$  may differ due to the duration variability issue. The models were trained on 3-second chunks of speech (Nagrani et al., 2020). Eighty-dimensional MFCCs of shape (94,80) generated using these audio clips for training are input to the model. As shown in Figure 1 and Figure
Figure 1: Architecture diagram for FAtNet-v1.
2, we pass each of the input MFCCs to the AdaptiveAvgPool2d layer to get features of shape (b, 94, 80), where b is the batch-size (Yu et al., 2019). It allows the model to accept variable duration speech recordings during test time without any special augmentation strategy. It facilitates easy integration of FAtNet embedding with other speaker verification models and enhances the usability of the models.
The next step is to compute frame-level features for further analysis. The abstract knowledge of frame-level features can reduce the linguistic information in the final embedding. We use four stacked TDNN $^3$  layers to extract the frame-level features (Vijayaditya Peddinti, 2015). Given two speech recordings as input, the problem is to determine if they belong to the same speaker or not. Thus, we have two such TDNN paths for input audio clips.
FAtNet-v1: We concatenate the frame-level features obtained for both the input speech recordings as shown in Figure 1. We apply batch normalization. We further pass these concatenated features through an eight-head frame-level attention block. In each frame, attention gives more weight to relevant features.
$^3$ Hyper-parameter detail for the stacked TDNN layers is present in the Appendix section.
2We compared the performance of 80-dimensional MFCCs with 300-dimensional spectrogram (Nagrani et al., 2017) as inputs to FAtNet-v1. Details are present in the Appendix section.
Figure 2: Architecture diagram for FAtNet-v2.
FAtNet-v2: We pass the frame-level features obtained for the input speech recordings through separate four-head attention blocks. Thus, we get embeddings 1 and 2 as shown in Figure 2. In each frame, the attention block gives more weight to relevant features. We then concatenate the outputs to get embedding 3.
Attention: Attention mechanisms are popular in state-of-the-art speaker verification models (Zhu et al., 2018; Wu et al., 2020). Our proposed FAtNet attention mechanism is inspired by (Vaswani et al., 2017; Moshnoi). The frame-level features are sent through a multi-head residual self-attention block, as shown in Figure 1 and Figure 2, respectively. The input to the attention block can be a tensor of shape  $(b,l,d)$  where  $b$  is the batch size,  $l$  is the number of frames, and  $d$  is the number of features or dimensions in each frame. Let  $dv$  be the dimension of linear space where the input needs to be projected, and  $nv$  is the number of heads in the multi-head self-attention block. We pass the same tensor to the attention block as the query tensor, key tensor, and, value tensor. The idea is to use the query tensor and key tensor to generate a weight tensor for the value tensor. The first step involves passing each of the three tensors: query, key, and value through separate fully-connected layers consisting of  $dv*nv$  output units and applying the ReLU activation function to get the modified query, key and value tensors, say  $Q$ ,  $K$ , and  $V$ , respectively. After adequate reshaping, the dimensions of  $Q$ ,  $K$ , and  $V$  should be  $(b,l,nv,dv)$ .
For each example  $i$ , the following computation is performed within the attention block using  $Q$ ,
$K$ , and  $V$ :
1.  $K_{\mathrm{i}}^{\mathrm{permute}} \coloneqq K_{\mathrm{i}} \cdot \text{permute}(0,2,1)$
2.  $prob_{i} \coloneqq Q_{i} * K_{i}^{\text{permute}}$
3.  $\text{prob}_{\mathrm{i}}^{\text{scaled}} := \frac{\text{prod}_{\mathrm{i}}}{\sqrt{d v}}$
4. weights $_{\mathrm{i}}$ $\text{attn} := \text{Softmax}(\text{prod}_{\mathrm{i}}^{\text{scaled}}, \dim = -1)$
5.  $rprod_{i} \coloneqq weight_{i}^{\mathrm{attn}*} V_{i}$
We further include a residual connection that acts as a memory to combine the initial set of frame-level features with  $rprod_{i}$ . So, to compute the output of the residual attention block, we add  $rprod_{i}$  to the original query tensor and pass it through a fully-connected layer consisting of  $d_{\mathrm{out}}$  neurons and apply  $ReLU$ .
The remaining layers are similar in both FAtNet-v1 and FAtNet-v2. The details are as follows: The generated attentive-frames are batch normalized. Then these are passed through a fully connected layer for fine-grained analysis. We further apply a leaky-relu activation function with  $L_{2}$ -normalization. We aggregate these frame-level features by computing a mean and then pass them through a fully connected layer consisting of two units for speaker verification.
Training datasets: We trained separate models of FAtNet-v1 on publicly available VoxCeleb-1 $^{4}$  and VoxCeleb-2 dev sets $^{5}$  (Nagrani et al., 2017; Chung et al., 2018). FAtNet-v2 was trained on VoxCeleb-2 dev set (Zhao et al., 2019). VoxCeleb-1 speech corpus contains recordings from 1,251 speakers, out of which 799 and 215 speakers belong to the USA and UK, respectively, where English is a dominant language. It consists of utterances from 1,211 speakers in the dev set and 40 speakers in the test set. The VoxCeleb-2 dataset consists of 5,994 speakers in the dev set. We used the same dev-test split as given in (Nagrani et al., 2017; Chung et al., 2018). The VoxCeleb datasets contain mostly English speech recordings (Chen et al., 2021). Details about the training setup are present in the Appendix section.
Test datasets: Experiments were conducted using trial pairs from the following publicly available
$^{4}$ VoxCeleb-1: https://www.robots.ox.ac.uk/~vgg/data/voxceleb/voxl.html
$5^{\prime}$  VoxCeleb-2: https://www.robots.ox.ac.uk/\~vgg/data/voxceleb/vox2.html
datasets: VoxCeleb-1 $^{6}$  test set (mostly English) (Nagrani et al., 2017), LibriSpeech $^{7}$  test set (English) (Panayotov et al., 2015), Aishell-1 test set $^{8}$  (Non-English) (Bu et al., 2017), and Voxforge $^{9}$  test set (Non-English) (Voxforge.org). Aishell-1 is a Mandarin speech corpus. Voxforge test set contains speech recordings in 10 different languages, namely, Bulgarian, Dutch, French, German, Greek, Italian, Portuguese, Russian, Spanish, and Turkish. We randomly generated the trial pairs for LibriSpeech, Aishell-1, and Voxforge from these publicly available datasets. The VoxCeleb-1 test set, LibriSpeech test set, Aishell-1 test set, and the Voxforge test set contain 37720, 47402, 23800, and 51856 trial pairs, respectively. The majority of the publicly available speech datasets are in the English language. VoxCeleb (used to train the models) datasets contain speech recordings in mostly English. Therefore we primarily investigate the effectiveness of this work on Non-English test sets without domain adaptation.
We performed experiments using two publicly available baselines: RawNet-2 $^{10}$  (Jung et al., 2020) and VGG-M $^{11}$  (Nagrani et al., 2017).
RawNet-2: RawNet-2 is an improved version of RawNet (Jung et al., 2019). It takes raw waveform as input and extracts speaker embedding. The model is pre-trained on VoxCeleb-2 for the speaker identification task to obtain 1024-dimensional embedding (Jung et al., 2020). Speech recordings from the trial pair are fed to the model individually as inputs. Thus, we get two 1024-dimensional embeddings for each input audio in the trial pair. We compute a cosine-similarity score of these two embeddings for the speaker verification task.
VGG-M: The VGG-M model was trained on the entire VoxCeleb-1 dataset for speaker identification (Nagrani et al., 2017). It generates a 4096-dimensional discriminative embedding. We used this pre-trained model to construct a siamese network for speaker verification. We fine
tuned the siamese network on VoxCeleb-1 dev for speaker verification. Speech recordings from a trial pair are inputs to the VGG-M models (frozen weights). We concatenate the generated 4096-dimensional embeddings to get a single 8,192-dimensional embedding. After batch normalization, we pass this embedding through a fully connected layer (consisting of 512-units) and apply the ReLU activation function. Then, after performing  $L_{2}$ -normalization, this 512-dimensional embedding (VGG-embedding) is fed to another fully connected layer consisting of two units for speaker verification.
4.3 Input strategy
For simplicity, we feed the input features to the model without any test time augmentation. The adaptive average pooling layer of the FAtNet model handles the duration variability issue. FAtNet is not a siamese network, and the weights of both the TDNN paths are learned separately. We pass the features from each input audio clip in the trial pair through both the TDNN paths. We further compute a mean as shown below: Assuming  $mfcc_{1}$  and  $mfcc_{2}$  are the MFCC features obtained for clips in the trial pair.
FAtNet-v1:
1.  $\mathrm{prob}_1\coloneqq \mathrm{model}(\mathrm{mfcc}_1,\mathrm{mfcc}_2)$
2.  $\mathrm{prob}_2\coloneqq \mathrm{model}(\mathrm{mfcc}_2,\mathrm{mfcc}_1)$
3.  $\mathrm{prob}_{\mathrm{final}} := \mathrm{mean}(\mathrm{prob}_1, \mathrm{prob}_2)$
FAtNet-v2:
1.  $\mathsf{emb}_{\mathrm{la}},\mathsf{emb}_{\mathrm{2a}}\coloneqq \mathsf{model}(\mathsf{mfcc}_1,\mathsf{mfcc}_2)$
2.  $\mathrm{emb}_{\mathrm{2b}},\mathrm{emb}_{\mathrm{1b}}\coloneqq \mathrm{model}(\mathrm{mfcc}_2,\mathrm{mfcc}_1)$
3.  $\mathrm{emb}_1\coloneqq \mathrm{mean}(\mathrm{emb}_{1\mathrm{a}},\mathrm{emb}_{1\mathrm{b}})$
4.  $\mathrm{emb}_2\coloneqq \mathrm{mean}(\mathrm{emb}_{2\mathrm{a}},\mathrm{emb}_{2\mathrm{b}})$
5.  $\mathrm{prob}_{\mathrm{final}} := \mathrm{CosineSimilarity}(\mathrm{emb}_1, \mathrm{emb}_2)$
4.4 Evaluation Metric
Equal Error Rate (EER) is a standard evaluation metric for biometric systems (Hansen and Hasan, 2015). Therefore, we investigate the effectiveness of this work in terms of EER. A lower EER score indicates a better performance.
5.1 Experimental validation of hypothesis
Our proposed approach aims to reduce the linguistic bias in the existing speaker verification systems by integrating a language-agnostic embedding. To
$^{6}$ VoxCeleb-1: https://wwwrobots.ox.ac.uk/~vgg/data/voxceleb/meta/veri_test.txt  
 $^{7}$ LibriSpeech: https://www.openslr.org/12  
 $^{8}$ Aishell-1: https://www.openslr.org/33/  
 $^{9}$ Voxforge: http://www.voxforge.org/  
 $^{10}$ We used the pre-trained RawNet-2 model available in https://github.com/Jungjee/RawNet  
 $^{11}$ We used the pre-trained VGG-M model available in https://github.com/Derpimort/VGGVox-PyTorch
scientifically validate the theoretical hypothesis, we explore the effect of integrating FAtNet embedding with baselines for out-of-domain test sets.
Consider a trial pair (clip1.wav,clip2.wav) having MFCCs, say,  $(mfcc_{1},mfcc_{2})$  and spectrograms, say,  $(spec_{1},spec_{2})$ .
VGG-M  $\oplus$  FAtNet-v1: In this integration, we directly pass MFCC's for the trial pair, say  $(mfcc_{1},mfcc_{2})$ , through the FAtNet-v1 model to get 1024-dimensional FAtNet embedding. We also pass spectrograms, say (spec1, spec2), through the VGG-M siamese baseline to get the 512-dimensional VGG-embedding. After concatenating these embeddings, we pass it to a fully connected layer of 1024 neurons. Finally, after applying  $ReLU$  and  $L_{2}$ -normalization, we pass it through another fully connected layer consisting of 2 units for speaker verification. We fine-tuned the last two fully connected layers on the VoxCeleb training set for speaker verification.
RawNet-2  $\oplus$  FAtNet-v2: For this integration, we obtained the 512-dimensional FAtNet embeddings for speech recordings in trial pair by feeding  $(mfcc_{1},mfcc_{2})$  through the FAtNet-v2 model. We get these FAtNet embeddings from steps 3 and 4 of the input strategy (described in section 4.3) for FAtNet-v2. We obtained the 1024-dimensional RawNet embeddings for recordings by feeding them to the RawNet-2 baseline. We compute cosine similarity after concatenating audio1's FAtNet-v2 embedding with its RawNet-2 embedding and audio2's FAtNet-v2 embedding with its RawNet-2 embedding.
As illustrated in Table 1, we observe significant improvements in the performance of baselines on out-of-domain test sets after integration with FAtNet embeddings. This observation suggests that with a very little overhead, our proposed FAtNet embeddings may help improve the performance of these baselines on out-of-domain test sets without domain adaptation.
5.2 Language-specific analysis
For an extensive validation of the observations from the previous experiment, we created separate test sets for 11 different languages using the Voxforge dataset. The Bulgarian test set consists of 3,110 trial pairs. The other test sets contain 20,000 trial pairs each. Figure 3 and Figure 4 show that integrating the baselines with our proposed FAtNet embedding consistently reduced the equal error
Figure 3: Figure showing that integrating VGG-M with FAtNet-v1 consistently reduced the EER on test sets generated for speech in different languages.
Figure 4: Figure showing that integrating RawNet-2 with FAtNet-v2 consistently reduced the EER on test sets generated for speech in different languages.