SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianchi Liu $^{1,2}$ , Kong Aik Lee $^{*3,1}$ , Qiongqiong Wang $^{1}$ , Haizhou Li $^{4,2}$
$^{1}$  Institute for Infocomm Research (I²R), Agency for Science, Technology and Research (A*STAR), Singapore
$^{2}$  Dept. of Electrical and Computer Engineering, National University of Singapore, Singapore
$^{3}$  Dept. of Electrical and Electronic Engineering, Hong Kong Polytechnic University, Hong Kong
$^{4}$  School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
{liu_tianchi, wang_qiongqiong}@i2r.a-star.edu.sg, kongaik.lee@ieee.org, haizhouli@cuhk.edu.cn
For speaker recognition, it is difficult to extract an accurate speaker representation from speech because of its mixture of speaker traits and content. This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is realized with the use of three Gaussian inference layers, each consisting of a learnable transition model that extracts distinct speech components. Notably, a strengthened transition model is specifically designed to model complex speech dynamics. We also propose a self-supervision method to dynamically disentangle content without the use of labels other than speaker identities. The efficacy of the proposed framework is validated via experiments conducted on the VoxCeleb and SITW datasets with  $9.56\%$  and  $8.24\%$  average reductions in EER and minDCF, respectively. Since neither additional model training nor data is specifically needed, it is easily applicable in practical use.
Automatic speaker recognition aims to identify a person from his/her voice [2] based on speech recordings [22]. Typically, two fixed-dimensional representations are extracted from the enrollment and test speech utterances, respectively [68]. Then the recognition procedure is done by measuring their similarity [22]. These representations are referred to as the speaker embeddings [67]. The concept of speaker embedding is similar to that of the face embedding [64] in the face recognition task and the token embedding [16] in the language model, while the main difference lies in the carriers of the information source and the nature of downstream tasks. Different from the discrete sequential inputs in Natural Language Processing (NLP) and continuous inputs in Computer Vision (CV), speech signals are continuous-valued variable-length sequences [24]. Speech signals contain both speaker traits and content [32]. For speaker recognition, to form a refined speaker embedding of a speech signal, conventional methods aggregate the frame-level features by pooling across the time-axis to extract speaker characteristics while factoring out content information. Simple temporal aggregation fails to disentangle the content information and therefore affects the quality of the resulting embeddings. The content information is regarded as unwanted variations hindering the accurate representation of voice characteristics.
To reduce the effects of the content variation, prior works model the phonetic content representation and use it as a reference for speaker embedding extraction. In [89], the phonetic bottleneck features from the last hidden layer of a pre-trained automatic speech recognition (ASR) network are combined with raw acoustic features to normalize the phonetic variations. A coupled stem is designed in [88] to jointly learn acoustic features and frame-level ASR bottleneck features. In [42], a phonetic attention mask dynamically generated from a sub-branch is used to benefit speaker recognition. The existing methods can be summarized into two types in terms of content representation modeling: (1) by a
pre-trained ASR model [57, 34, 45, 89, 46], and (2) by a jointly trained multi-task model with extra components for content representations [43, 88, 17, 46, 80, 44, 42]. These methods prove that the utilization of content representation benefits speaker recognition. However, both types lead to an obvious limit in practical applications. The employment of a pre-trained ASR model greatly increases the parameters and computation complexity in the inference, as the ASR models are typically one or two orders of magnitude larger than the speaker recognition model [87, 71]. The joint training of the speaker recognition model and extra components for content representations requires either an extra dataset with text labels in addition to the dataset with speaker identities, or one dataset with speaker identities and text labels simultaneously which is expensive and not easy to come by. For the former type of joint training, additional components are still needed with extra effort and may encounter optimization difficulties.
For the purpose of speaker recognition, one aims to derive an embedding vector representing the vocal characteristics of the speaker. The benefits of leveraging content representation are significant, while the drawbacks of text labels and extra model requirements are obvious. Motivated by this, we seek to design a novel framework to solve these problems.
Speech signals consist of many components, of which the two major parts are speaker traits and speech content [32, 14]. In this paper, we focus on decomposing speech signals into static and dynamic components. The former is static, i.e., fixed, with respect to temporal evolution and dominated by speaker characteristics, while the latter mainly consists of speech content with some other components, such as the prosody. Based on this assumption, we design a framework with three Gaussian inference layers which aims to decompose the static and dynamic components of the speech. The static part is modeled by static Gaussian inference with the criterion of speaker classification loss. During inference, the static latent factor is associated with the vocal characteristics of the speaker, and we refer to it as speaker representation. The remaining dynamic components related to verbal content are modeled by dynamic Gaussian inference. The motivation of the three-layer design is simple and intuitive – the speaker representation from layer 1 is not accurate and may include content information as the content-related reference is absent in training. However, it helps layer 2 extract the content representation which can be used as the reference for layer 3. Thus, a more accurate speaker representation is extracted in layer 3. It is worth noting that the framework is trained without text labels and no extra model or branch is employed thus not much increase in model size. This is achieved with the guide of self-supervision and considered content representations. We named this framework as Recurrent Xi-vector (RecXi). The major contributions are summarized as follows:
- RecXi: A novel disentangling framework with the following features: (1) We enhance the xi-vector [33] with the ability to capture temporal information and name it the recurrent xi-vector layer. (2) A frame-wise content-aware transition model  $\mathbf{G}_t$  is proposed for modeling dynamic components of the speech. (3) A novel design of three layers of Gaussian inference is proposed to disentangle speaker and content representations by static and dynamic modeling with the corresponding counterpart removal, respectively.
- A self-supervision method is proposed to guide content disentanglement by preserving speaker traits. It reduces the impact of the absence of text labels.
Conventional Speaker Recognition Methods. Speaker embeddings are fixed-length continuous-value vectors extracted from variable-length utterances to represent speaker characteristics. They live in a simpler Euclidean space in which distance can be measured for the comparison between speakers [78]. A well-known example of the extractor is the x-vector framework [67], which mainly consisted of the following three components: 1). An encoder is implemented by stacking multiple layers of time-delay neural network (TDNN) and used to extract the frame-level features from utterances [58]. Recent works strengthen the encoder by replacing the simple TDNNs with powerful ECAPA-TDNN [15] and its variants [41, 74, 23, 53, 47], or ResNet series models [90, 36, 84, 83, 51]. 2). A temporal aggregation layer aggregates frame-level features from the encoder into fixed-length utterance-level representations. 3). A decoder classifies the utterance-level representations to speaker classes for supervised learning by a classification loss. The decoder stacks several fully-connected layers including one bottleneck layer used to extract speaker embedding.
Speech Disentanglement. Various informational factors are carried by the speech signal, and the speech disentanglement ultimately depends on which informational factors are desired and how they
will be used [81]. For speaker recognition, in addition to the use of content information we introduced above, some works attempt to disentangle speaker representation with the removal of irrelevant information, like devices, noise, and channel information with corresponding labels [50, 52, 28]. [52] minimizes the mutual information between speaker and device embeddings, with the goal of reducing their interdependence. In [40], nuisance variables like gender and accent are removed from speaker embeddings by learning two separate orthogonal representations. Many other speech-relevant tasks also show great improvements by disentangling these two components properly. For speaker diarization, the ASR model is employed to obtain content representations leveraged by speaker embedding extractions [70, 29]. In voice conversion or personalized voice generation tasks, a popular method is to disentangle the speech into linguistic and speaker representations before performing the generation [26, 86, 92, 11, 18, 91, 38]. In ASR, speaker information is extracted for speaker variants removal for performance improvements [49, 35, 25] or privacy preserving [69]. Similarly, auxiliary network-based speaker adaptation [63] or residual adapter [75] are explored to handle large variations in the speech signals produced by different individuals. These works have led us to the design of RecXi, which is the first speaker and content disentanglement framework for speaker recognition in the absence of extra labels for practical use to the best of our knowledge.
Self-Supervision in Speech. Self-supervised learning has been the dominant approach for utilizing unlabeled data with impressive success [1, 16, 7, 20]. In speaker recognition, contrastive learning [9] is used to force the encoder to produce similar activation between a positive pair. The positive pair can be two disjoint segments from the same utterance [6, 85] or from cross-referencing between speech and face data [73]. The trained speaker encoder also can produce pseudo speaker labels for supervised learning [72, 5]. In target speaker extraction, self-supervision is helpful to find the speech-lip synchronization [56]. In voice conversion, much research focuses on applying variational auto-encoder (VAE) to disentangle the speaker [10, 27, 59]. Similar to reconstruction-based methods, mutual information (MI) and identity change loss are used in [30] for robust speaker disentanglement. ContentVec [60] disentangles and removes speaker variations from the speech representation model HuBERT [24] for various downstream tasks, such as language modeling and zero-shot content probe.
Considering the main objective of this work is to benefit speaker recognition by disentangling the speaker with the speaker classification supervision, we want to clarify that the content disentanglement and corresponding self-supervision serve this final target. Therefore, the self-supervision method is designed to be simple yet effective while avoiding extra signal re-constructors or training efforts.
3.1 Xi-vector
Many works have been proposed to estimate the uncertainty for the speaker embedding [65, 33, 78]. Among them, the xi-vector [33] is proposed to enhance x-vector embedding in handling the uncertainty caused by the random intrinsic and extrinsic variations of human voices. Specifically, an input frame  $\mathbf{x}_t$  is encoded to a point-wise estimate  $\mathbf{z}_t$  in a latent space by an encoder. And an auxiliary network is employed to characterize the frame-level uncertainty with a posterior covariance matrix  $\mathbf{L}_t^{-1}$  associated with  $\mathbf{z}_t$  as shown in the encoder part of Figure 1. These two operations are formulated as
$$
\mathbf {z} _ {t} = f _ {\mathrm {e n c}} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t} ^ {t \pm n}\right), \tag {2}
$$
where the neighbour frames  $\mathbf{x}_t^{t\pm n}$  of time  $t$  are taken into consideration for the frame-wise estimation. The precision matrices  $\mathbf{L}_t$  are assumed to be diagonal and are estimated as log-precision for the convenience of following steps. It is assumed that a linear Gaussian model is responsible to generate the representations  $\mathbf{z}_t$  as follows:
generative model:  $\mathbf{z}_t = \mathbf{h} + \epsilon_t$ , latent variable:  $\mathbf{h} \sim \mathcal{N}(\phi_{\mathrm{p}}, \mathbf{P}_{\mathrm{p}}^{-1})$ , uncertainty:  $\epsilon_t \sim \mathcal{N}(0, \mathbf{L}_t^{-1})$ . (3) Here,  $\mathbf{h}$  is the latent variable for the entire utterance, and  $\epsilon_t$  is a frame-wise random variable for the uncertainty covariance measure,  $\phi_{\mathrm{p}}$  and  $\mathbf{P}_{\mathrm{p}}^{-1}$  are the Gaussian prior mean and covariance matrix of the variable  $\mathbf{h}$ . The posterior mean vector  $\phi_{\mathrm{s}}$  and precision matrix  $\mathbf{P}_{\mathrm{s}}$  for the whole sequence are formulated as
$$
\phi_ {\mathrm {s}} = \mathbf {P} _ {\mathrm {s}} ^ {- 1} \left[ \sum_ {t = 1} ^ {T} \mathbf {L} _ {t} \mathbf {z} _ {t} + \mathbf {P} _ {\mathrm {p}} \phi_ {\mathrm {p}} \right], \tag {5}
$$
where the posterior mean  $\phi_{\mathrm{s}}$  is enriched by frame-wise uncertainty and is further used to be decoded into speaker embedding.
Figure 1: The network architecture of the proposed RecXi system with self-supervision. The structure at the bottom-middle of the figure is a simplified speaker recognition system. The figures in three dotted boxes are the specific explanations of its three parts. At the encoder,  $\{\mathbf{x}_1,\mathbf{x}_2\dots \mathbf{x}_T\}$  is the input sequence of length  $T$ . The three colors of blue, orange, and green indicate three recursive layers. For each layer, the inference flow for frames  $\mathbf{x}_1$ ,  $\mathbf{x}_2$  and  $\mathbf{x}_T$  is drawn, while the frames in between are replaced by '...'. The dashed lines indicate recurrent operation and the solid lines represent the operations within the same frame. The block with  $\mathbf{G}_t$  gen indicates the filter generator. For the decoder,  $\odot$  is a subtraction operation. The dotted line indicates the operation is optional.
3.2 Disentangling Speaker and Content Representations
Basic Recurrent Xi-vector Layer. Xi-vector has the advantage of modeling static components of the speech by the utterance-level speaker representation aggregation with the use of uncertainty estimation, while the drawback of modeling dynamic signals is obvious. For approximating nonlinear functions in high-dimensional feature spaces and restricting the inference through the adjacent Gaussian hidden state efficiently, similar to [4], a learnable linear transition model  $\mathbf{G}$  is applied. We further extend the xi-vector into a recursive form with frame-level estimation. It is worth noting that xi-vector is a special case of the basic recurrent xi-vector when  $\mathbf{G}$  is an identity matrix.
The Gaussian inference in the latent space can be implemented in two stages: predict stage and update stage. In the predict stage, the predictive mean vector  $\rho_t^+$  and precision matrix  $\Phi_t^+$  are formulated as
$$
\rho_ {t} ^ {+} = \mathbf {G} \rho_ {t}, \tag {6}
$$
$$
\boldsymbol {\Phi} _ {t} ^ {+} = \left[ \mathbf {G} \boldsymbol {\Phi} _ {t} ^ {- 1} \mathbf {G} ^ {\mathrm {T}} \right] ^ {- 1}, \tag {7}
$$
where  $\mathbf{G}$  represents a linear transition model, and  $^+$  indicates the results of the given frame  $t$  after the predict stage. T indicates a transpose operation.
For sake of clarity, we assign the time (i.e., frame) index  $t = 0$  to the priors, such that  $\rho_0 = \rho_{\mathrm{p}}$ ,  $\Phi_0 = \Phi_{\mathrm{p}}$ . In the update stage, the posterior mean vector  $\rho_t$  of frame  $t$  is derived by incorporating the previously predicted posterior mean and precision  $\{\rho_{t-1}^+, \Phi_{t-1}^+\}$  of the hidden states of the former frame  $t-1$  with encoded features  $\mathbf{z}_t$  and estimated uncertainty  $\mathbf{L}_t$  of current frame  $t$ . The uncertainty measures  $\mathbf{L}_t$  of frame  $t$  is added into the posterior precision matrix. These two operations are derived as
$$
\rho_ {t} = \boldsymbol {\Phi} _ {t} ^ {- 1} \left[ \mathbf {L} _ {t} \mathbf {z} _ {t} + \boldsymbol {\Phi} _ {t - 1} ^ {+} \rho_ {t - 1} ^ {+} \right], \tag {8}
$$
$$
\boldsymbol {\Phi} _ {t} = \mathbf {L} _ {t} + \boldsymbol {\Phi} _ {t - 1} ^ {+}. \tag {9}
$$
Frame-wise Content-aware Transition Model  $\mathbf{G}_t$ . Speech signals are a complex mixture of dynamic information sources. Even though the transition model  $\mathbf{G}$  is learnable during training, it remains the same across all the hidden states of Gaussian inference for each sample. To enhance the ability to model dynamic speech components for content disentanglement, we propose a transition
model  $\mathbf{G}_t$  which is dynamically adjusted by a filter generator for each frame according to the content during the Gaussian inference and hereby named frame-wise content-aware transition model.
Specifically, a set of  $N$  learnable transition matrices  $\{\mathbf{G}_1',\mathbf{G}_2'\dots \mathbf{G}_N'\}$  is employed to model  $N$  dynamic components in speech signals. For each frame  $t$ , a vector  $\mathbf{w}_t\in \mathbb{R}^N$  is generated representing the importance of different dynamic components to content representation modeling, by observing content information  $\rho_{t}$  from update stage. The content-aware transition model for frame  $t$  is finally obtained as a weighted sum over the  $N$  component-dependent transition models with weight  $w_{t,n}$ .
$$
\mathbf {G} _ {t} = \sum_ {n = 1} ^ {N} w _ {t, n} \mathbf {G} _ {n} ^ {\prime}, \quad (1 0) \quad w _ {t, n} = \left[ \mathbf {w} _ {t} \right] _ {n}, \quad (1 1) \quad \mathbf {w} _ {t} = \sigma \left(f \left(\rho_ {t}\right)\right), \tag {12}
$$
where  $\mathbf{w}_t\in \mathbb{R}^N$ .  $\sigma$  and  $f$  indicate the Softmax function and a non-linear operation, respectively. In this work, the non-linear operation is designed as two fully connected layers with a ReLU [19] activation function in between. It is to be noted that this procedure is frame-wise as the dynamic components vary along the sequence of  $T$ . As an extension to Equations (6) and (7), the predict stage of recurrent xi-vector is reformulated as
$$
\rho_ {t} ^ {+} = \mathbf {G} _ {t} \rho_ {t}, \tag {13} \quad \Phi_ {t} ^ {+} = \left[ \mathbf {G} _ {t} \boldsymbol {\Phi} _ {t} ^ {- 1} \mathbf {G} _ {t} ^ {\mathrm {T}} \right] ^ {- 1}. \tag{14}
$$
Three Layers of Gaussian Inference. We propose a structure based on three layers of Gaussian inference for disentangling speaker and content representations in the absence of text labels. In this work, this structure is utilized in the temporal aggregation layer and named RecXi. The three layers of Gaussian inferences each aim at precursor speaker representation, disentangled content representation, and disentangled speaker representation, as shown in the RecXi part of Figure 1. The last frame's representations from each layer are used for deriving embeddings within the decoder. The details of each layer are discussed as follows.
Layer 1: Precursor Speaker Representation. This is a basic recurrent xi-vector layer that aims to represent the speaker characteristics. Therefore, the transition model of the Gaussian inference is set as an identity matrix to model static components of the speech. The predict stage can be derived from Equations (6) and (7) as
$$
\phi_ {t} ^ {+} = \phi_ {t}, \tag {15} \quad \mathbf {P} _ {t} ^ {+} = \mathbf {P} _ {t}. \tag{16}
$$
The speaker representation from this layer is similar to that in the original xi-vector, where the content information from the high-dimensional frame-level features remains and affects the speaker embedding quality. We refer to the representation of this layer as the precursor speaker representation. Specifically, the frame-wise representations of posterior mean  $\phi_t$  and precision  $\mathbf{P}_t$  of the hidden state  $\mathbf{h}$  of frame  $t$  are estimated. They are derived partly according to frame-level features  $\mathbf{z}_t$  and uncertainty  $\mathbf{L}_t$  extracted from the corresponding frame  $t$ , and partly from previous frames  $\{1,2,\dots t - 1\}$  by recursively passing in the posteriors from the previous frame. The update stage is formulated as
$$
\phi_ {t} = \mathbf {P} _ {t} ^ {- 1} \left[ \mathbf {L} _ {t} \mathbf {z} _ {t} + \mathbf {P} _ {t - 1} ^ {+} \phi_ {t - 1} ^ {+} \right], \tag {18}
$$
Layer 2: Disentangled Content Representation. This layer aims to disentangle content representation from the sequence. To model dynamic subtle content changes, the frame-wise content-aware transition model  $\mathbf{G}_t$  introduced above is applied. As shown in the orange part of Figure 1, the transition model  $\mathbf{G}_t$  is generated by a filter generator according to Equations (10), (11) and (12). Equations (13) and (14) are applied to the predict stage of layer 2.
To disentangle the content representation more effectively, in addition to equipping the layer with dynamic modeling ability, we further attempt to remove speaker information from the frame-level features for each frame during Gaussian inference. This ensures that the remaining information is more likely to be dynamic and associated with the content.
The posterior mean  $\phi^{+}$  and posterior precision  $\mathbf{P}^{+}$  of the hidden state of layer 1 are rich with speaker information and utilized by the update stage of layer 2. Benefiting from the linear operations in this three layers design, the speaker removal operation can simply be done by subtracting the posterior mean  $\phi^{+}$  from features  $\mathbf{z}$  while adding the posterior covariance matrix  $(\mathbf{P}^{+})^{-1}$  into the uncertainty estimation matrix  $\mathbf{L}^{-1}$ . The procedure is dynamically processed for each frame, and the update stage is formulated as
$$
\rho_ {t} = \boldsymbol {\Phi} _ {t} ^ {- 1} \left[ \left(\mathbf {L} _ {t} ^ {- 1} + \left(\mathbf {P} _ {t} ^ {+}\right) ^ {- 1}\right) ^ {- 1} \left(\mathbf {z} _ {t} - \phi_ {t} ^ {+}\right) + \boldsymbol {\Phi} _ {t - 1} ^ {+} \rho_ {t - 1} ^ {+} \right], \tag {19}
$$
$$
\boldsymbol {\Phi} _ {t} = \left(\mathbf {L} _ {t} ^ {- 1} + \left(\mathbf {P} _ {t} ^ {+}\right) ^ {- 1}\right) ^ {- 1} + \boldsymbol {\Phi} _ {t - 1} ^ {+}. \tag {20}
$$
Layer 3: Disentangled Speaker Representation. This layer uses the Gaussian inference with an identity matrix for modeling static components of the speech. Furthermore, the speaker classification loss is applied to the output of this layer and provides supervision restrictions. Different from layer 1, layer 3 is designed to model the desired disentangled speaker representation by removing the content information provided by layer 2 from the frame-level features. The procedure is similar to that in layer 2, and the predict stage is formulated as
$$
\tilde {\phi} _ {t} ^ {+} = \tilde {\phi} _ {t}, \tag {21}
$$
where  $\tilde{\phi}_t^+$  and  $\tilde{\mathbf{P}}_t^+$  are the posterior mean and posterior precision, respectively. The symbol of  $\tilde{}$  is used to differentiate them from the posteriors in layer 1. The update stage is formulated as
$$
\tilde {\phi} _ {t} = \tilde {\mathbf {P}} _ {t} ^ {- 1} \left[ \left(\mathbf {L} _ {t} ^ {- 1} + \left(\boldsymbol {\Phi} _ {t} ^ {+}\right) ^ {- 1}\right) ^ {- 1} \left(\mathbf {z} _ {t} - \rho_ {t} ^ {+}\right) + \tilde {\mathbf {P}} _ {t - 1} ^ {+} \tilde {\phi} _ {t - 1} ^ {+} \right], \tag {23}
$$