SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {\mathbf {P}} _ {t} = \left(\mathbf {L} _ {t} ^ {- 1} + \left(\boldsymbol {\Phi} _ {t} ^ {+}\right) ^ {- 1}\right) ^ {- 1} + \tilde {\mathbf {P}} _ {t - 1} ^ {+}. \tag {24}
$$
It is worth noting that to avoid expensive matrix multiplication operations and numerically problematic matrix inversions, a simplified implementation is adopted (see Appendix A).
3.3 Speech Disentanglement with Self-supervision
A well-trained speaker embedding neural network requires a huge number of speakers and utterances to achieve discriminative ability. For such a large dataset, text labels are very difficult to come by. In the absence of text labels, disentangling content information from the speech is a very difficult task.
In Section 3.2, the layer 3 of RecXi is designed to be the desired speaker representation. This is achieved by static Gaussian inference with the assumption that disentangled content representation provided by layer 2 is reliable. We note that the disentangled speaker representation from layer 3 is directly optimized through the classification criterion, while the optimization for disentangling content information is indirect through content removal operations in Gaussian inference. In order to ameliorate the shortcoming due to the absence of text label and to provide an extra supervisor for layer 2, we propose a self-supervision method to preserve speaker information via knowledge distillation in a similar fashion to those proposed in [62, 76, 8] for the teacher-student pair.
Generally speaking, large models usually outperform small models, while the small model is computationally cheaper [76]. Knowledge distillation aims to benefit a small model with the guidance of a large model. Different from the general knowledge distillation methods, our 'teacher' and 'student' are two different layers within the RecXi. Since this guidance comes from the RecXi itself, and all the training data is considered as unlabelled data for the content disentanglement task, the proposed method is considered a self-supervision method [3].
The output  $\phi$  from layer 1 of RecXi is precursor speaker representation, and the content information remains, while layer 2 is designed to disentangle content representation  $\rho$ . Benefiting from the linear operations used in RecXi, we can remove content information and preserve speaker representation by subtracting the content representation  $\rho$  of layer 2 from the precursor speaker representation  $\phi$  of layer 1. This speaker representation is marked as  $\tilde{\phi}_{\mathrm{lin}}$  and derived as
$$
\tilde {\phi} _ {\mathrm {l i n}} = \phi - \rho \tag {25}
$$
where the 'lin' indicates that it is obtained by a linear operation, differing from  $\tilde{\phi}$  in layer 3 which is disentangled by Gaussian inference. For the same input sequence, the speaker representation  $\tilde{\phi}_{\mathrm{lin}}$  derived by this linear operation should be consistent with  $\tilde{\phi}$  obtained from disentanglement. Therefore, by restricting their difference, the gradient from supervision speaker classification loss will form a guide for layer 2 through the linear operation in Equation (25). This serves as an extra supervisory signal different from the constraints imposed by speaker removal operation during the Gaussian inference. As  $\tilde{\phi}$  is optimized by classification loss directly, it is considered as a 'teacher',
while  $\tilde{\phi}_{\mathrm{lin}}$  is a 'student'. Since  $\tilde{\phi}_{\mathrm{lin}}$  is derived by preserving speaker representation in Equation (25), we name the loss as self-supervision speaker preserving loss  $\mathcal{L}_{\mathrm{ssp}}$ .
The  $\mathcal{L}_{\mathrm{ssp}}$  loss can be generated by different knowledge distillation methods, a simple comparison is available in Appendix D. In this work, the idea of similarity-preserving loss [76] is in line with our layer-wise design and is inherited to guide the student towards the activation correlations induced in the teacher, instead of mimicking the teacher's representation space directly. It is derived as
$$
\mathcal {L} _ {\mathrm {s s p}} = \frac {1}{b ^ {2}} \sum_ {(s, s ^ {\prime}) \in \kappa} \left\| \left(\left\| \tilde {\phi} ^ {(s)} \tilde {\phi} ^ {(s) \mathrm {T}} \right\| _ {2} - \left\| \tilde {\phi} _ {\mathrm {l i n}} ^ {(s ^ {\prime})} \tilde {\phi} _ {\mathrm {l i n}} ^ {(s ^ {\prime}) \mathrm {T}} \right\| _ {2}\right) \right\| _ {F} ^ {2}, \tag {26}
$$
where  $\kappa$  collects all the  $(s, s')$  pairs from the same mini-batch.  $\| \cdot \|_2$  and  $\| \cdot \|_F$  indicate the row-wise L2 normalization and Frobenius norm, respectively.  $T$  indicates a transpose operation and  $b$  indicates the batch size. We define the total loss for training the framework as
$$
\mathcal {L} _ {\text {t o t a l}} = \alpha \mathcal {L} _ {\mathrm {c l s}} + \beta \mathcal {L} _ {\mathrm {s s p}}, \tag {27}
$$
where  $\mathcal{L}_{cls}$  is the speaker classification loss,  $\alpha$  and  $\beta$  are hyperparameters for balancing the total loss.
4 Experiments Setup
4.1 Dataset, Training Strategy, and Evaluation Protocol
The experiments are conducted on VoxCeleb1 [54], VoxCeleb2 [13], and the Speaker in the Wild (SITW) [48] datasets. It is worth noting that the text labels are not available for these datasets. For a fair comparison, the baselines are all re-implemented and trained with the same strategy as RecXi which follows that in ECAPA-TDNN [15]. All the models are evaluated by the performance in terms of equal error rate (EER) and the minimum detection cost function (minDCF). Detailed descriptions of datasets, training strategy, and evaluation protocol are available in Appendix C.
4.2 Systems Description
We evaluate the systems that are combinations of different backbone models in the encoder and different aggregation layers. To verify the compatibility of the proposed RecXi, both 2D convolution (Conv2D)-based and time delay neural network (TDNN)-based backbones are adopted:
- 1) ECAPA-TDNN [15] is a state-of-the-art speaker recognition model based on TDNNs. The layers before the temporal aggregation layer are considered the backbone network.
- 2) ResNet [21] and tResNet represent the models based on 2D convolution. The modified ResNet34 [77] is adopted as a baseline. In order to model more local regions with larger frequency bandwidths at different scales, we further modify the ResNet34 in [77] by simply changing the stride strategy and name it tResNet. The details are provided in Appendix B.
The following are temporal aggregation layers. The first three are considered baselines:
- 1) Temporal Statistics Pooling (term as TSP from here onwards) [66] is a well-known aggregation method. It is the default option for x-vector and adopted in ResNet for speaker recognition [12, 90].
- 2) Channel- and Context-dependent Statistics Pooling (term as Chan.&Con. from here onwards) [15] is the default aggregation layer in ECAPA-TDNN, modified from the attentive statistics pooling [55] by adding channel-dependent frame attention and allowing the self-attention produced across global properties.
- 3) Xi-vector Posterior Inference (term as Xi from here onwards) [33] inserts uncertainty estimation into speaker embeddings as introduced in Section 3.1.
- 4) RecXi  $(\tilde{\phi})$  and RecXi  $(\tilde{\phi}, \tilde{\phi}_{\mathrm{lin}})$  are the proposed methods. The former uses only  $\tilde{\phi}$  as the input to the decoder to derive the speaker embedding, and the latter uses a concatenation of both  $\tilde{\phi}$  and  $\tilde{\phi}_{\mathrm{lin}}$  as the input.
Table 1: Performance in EER(%) and minDCF of the state-of-the-art (SOTA) systems and proposed RecXi on VoxCeleb1 and SITW test sets. Aug. indicates whether the system is trained with data augmentations. Para. is the number of parameters in million. Systems with  $\dagger$  are re-implemented. The systems with  $\ddagger$  following the format of 'aggregation layer + backbone model' are baselines. # is the index number for the implemented systems.
#	System	Aug.	Para.	VoxCeleb1-O	VoxCeleb1-H	VoxCeleb1-E	SITW eval
EER	minDCF	EER	minDCF	EER	minDCF	EER	minDCF
-	ResNet34-SKDFE [39]	X	5.98	1.44	0.168	2.76	0.278	1.59	0.179	-	-
-	H/ASP(AAM-softmax) [31]	X	8.0	1.25	-	2.78	-	1.56	-	-	-
-	H/ASP(AP+softmax) [31]	X	8.0	1.21	-	2.77	-	1.42	-	-	-
-	SI-Net50 [37]	X	-	1.28	-	2.50	-	1.28	-	2.54	-
-	RSKNet-MTSP [82]	X	13.9	1.05	0.159	2.52	0.237	1.30	0.152	2.27	0.228
1	ECAPA-TDNN [15]†	X	6.19	1.377	0.137	2.674	0.235	1.451	0.153	2.542	0.208
2	ResNet34 [77]†	X	6.63	1.489	0.155	2.500	0.224	1.423	0.158	2.378	0.208
3	TSP + proposed tResNet‡	X	6.21	1.396	0.135	2.257	0.204	1.281	0.141	2.250	0.200
4	Xi [33] + proposed tResNet‡	X	6.54	1.295	0.120	2.169	0.202	1.224	0.142	2.150	0.189
5	RecXi(φ, φlin) with Lssp + tResNet	X	7.06	1.196	0.122	2.097	0.196	1.197	0.124	1.832	0.172
-	Res2Net-26w8s [90]	✓	9.3	1.45	0.147	2.72	0.272	1.47	0.169	-	-
-	H/ASP(AAM-softmax) [31]	✓	8.0	1.15	-	2.49	-	1.35	-	-	-
-	H/ASP(AP+softmax) [31]	✓	8.0	0.88	-	2.21	-	1.07	-	-	-
-	ECAPA-TDNN [15]	✓	6.2	1.01	0.127	2.32	0.218	1.24	0.142	-	-
-	MFA-TDNN (standard) [41]	✓	7.32	0.856	0.092	2.049	0.190	1.083	0.118	-	-
-	MFA-TDNN (lite) [41]	✓	5.93	0.968	0.091	2.174	0.199	1.138	0.121
-	ECAPA-TDNN(MBFA-MW) [61]	✓	-	0.87	0.115	2.31	0.222	1.22	0.135	-	-
-	UP-LS UP-PLDA [79]	✓	-	1.01	0.124	2.18	0.224	-	-	1.59	0.167
6	ECAPA-TDNN [15]†	✓	6.19	1.127	0.145	2.249	0.223	1.194	0.133	1.668	0.155
7	ResNet34 [77]†	✓	6.63	1.101	0.128	2.221	0.208	1.252	0.139	1.584	0.161
8	Xi [33] + proposed tResNet‡	✓	6.54	0.936	0.115	1.942	0.186	1.110	0.125	1.394	0.145
9	RecXi(φ, φlin) with Lssp + tResNet	✓	7.06	0.984	0.091	1.857	0.179	1.075	0.114	1.340	0.137
5 Results and Discussions
In Table 1, Table 2 and 3, we report the performance in terms of EER and minDCF of the proposed RecXi, various baselines and SOTA systems. It is worth noting that the VoxCeleb1-O test set is much smaller and easier compared to the other three test sets, which may lead to less convincing results.
Comparison with SOTA methods and baselines. In Table 1, we compare the baseline systems and proposed RecXi systems with SOTA methods. The baseline systems with proposed the tResNet backbone (system #3 and #4) achieve the SOTA performance. Compared to the TSP method (system #3), the use of Xi (system #4) achieves overall improvements with the  $5.02\% / 4.19\%$  average reductions in EER/minDCF. This also renders further performance improvements more difficult. By comparing systems #4 and #5, we observe that the proposed RecXi leads to relative improvements with  $6.96\% / 5.82\%$  average reductions in EER/minDCF compared to the Xi baseline system. The proposed RecXi (system #5) obviously further improves the performance.
Experiments with data augmentation. In the second half of Table 1, we report the results with augmented data (see Appendix C.2). The re-implemented ECAPA-TDNN (system #6) achieves slightly better performance than that reported in [15]. Consistent with the conclusions of training without data augmentation discussed above, the baseline (system #8) with proposed tResNet outperforms all SOTA methods and the proposed RecXi (system #9) shows great superiority over all other systems.
Comparison for ECAPA-TDNN backbone-based systems. The experiments discussed above are for Conv2D-based networks. For speech-related tasks, the TDNN-based networks are widely used as well [74, 15, 41, 53]. In order to verify the superiority of the proposed method under different conditions, the experiment results for the systems based on the popular ECAPA-TDNN [15] backbone
Table 2: Performance in EER(%) and minDCF of the baselines and proposed RecXi with ECAPATDNN backbone [15] on VoxCeleb1 and SITW test sets. # is the index number for the system.
#	Aggregation Layer	Params (Million)	VoxCeleb1-O	VoxCeleb1-H	VoxCeleb1-E	SITW eval
EER	minDCF	EER	minDCF	EER	minDCF	EER	minDCF
1	Chan.&Con.	6.19	1.377	0.137	2.674	0.235	1.451	0.153	2.542	0.208
10	Xi	5.90	1.340	0.140	2.647	0.237	1.429	0.148	2.679	0.204
11	RecXi(φ, φlin) with Lssp	6.43	1.196	0.107	2.467	0.227	1.292	0.141	2.105	0.184
are reported in Table 2. Compared to the original aggregation layer of ECAPA-TDNN (system #1), the use of xi-vector posterior inference (system #10) performs slightly better. The comparison between system #10 and #11 presents that for the ECAPA-TDNN backbone, the proposed RecXi achieves average EER/minDCF reductions of  $12.15\% / 10.66\%$  over the Xi baseline.
Brief summary. Since the Xi baseline performs the best among all baselines and SOTA systems discussed above with different training conditions and backbone networks, we refer to it as a SOTA baseline for the following discussion. We observe that compared to the SOTA baseline, the proposed RecXi consistently improves the performance for both backbone networks on all four test sets with overall average EER/minDCF reductions of  $9.56\% / 8.24\%$  (system #4 vs. #5 and #10 vs. #11). The experiment results remain consistent regardless of the backbone types and whether augmented data is used. This may be due to the fact that the proposed disentanglement framework disentangles the static speaker components effectively and benefits speaker recognition. As the speaker is disentangled under the assistance of disentangled content representation, the significant improvement also proves the quality of the dynamic counterpart modeling. In addition, we found that the performance of tResNet-based systems is generally better than that of the systems with ECAPA-TDNN backbone.
Table 3: Performance in EER(%) and minDCF of various RecXi systems on VoxCeleb1 and SITW test sets for ablation study. # is the index number for the system. BN and Para. indicate the type of backbone network and number of parameters in million, respectively.
#	BZ	Aggregation Layer	Lssp	Para.	VoxCeleb1-H	VoxCeleb1-E	SITW eval	Relative Reduction
EER	minDCF	EER	minDCF	EER	minDCF	EER	minDCF
11	ECAPA	RecXi(φ, φlin)	✓	6.43	2.467	0.227	1.292	0.141	2.105	0.184	-7.58%	-12.18%
12	ECAPA	RecXi(φ)	✓	6.13	2.445	0.233	1.286	0.139	2.160	0.191	-7.30%	-11.04%
13	ECAPA	RecXi(φ, φlin)	X	6.43	2.477	0.228	1.326	0.142	2.351	0.196	-3.41%	-10.40%
14	ECAPA	RecXi(φ)	X	6.13	2.498	0.229	1.325	0.146	2.597	0.273	Benchmark
5	tResNet	RecXi(φ, φlin)	✓	7.06	2.097	0.196	1.197	0.124	1.832	0.172	-5.78%	-5.83%
15	tResNet	RecXi(φ)	✓	6.73	2.117	0.192	1.215	0.128	1.750	0.177	-6.35%	-4.82%
16	tResNet	RecXi(φ, φlin)	X	7.06	2.130	0.198	1.222	0.128	1.886	0.184	-3.71%	-2.38%
17	tResNet	RecXi(φ)	X	6.73	2.185	0.204	1.230	0.128	2.050	0.193	Benchmark
Figure 2: Bar charts for ablation studies. Performance in EER(\%) and minDCF of proposed RecXi under different conditions are drawn. The blue and orange each indicate  $\mathrm{RecXi}(\tilde{\phi})$  and  $\mathrm{RecXi}(\tilde{\phi},\tilde{\phi}_{\mathrm{lin}})$ . Patterned color bars at the left front and solid color bars at the right behind represent the performance with and without  $\mathcal{L}_{\mathrm{ssp}}$  for the same test trial, respectively. For all bars, the shorter the better.
Ablation Study. We perform ablation studies on the proposed RecXi. As the results shown in Table 3 are detailed but not intuitive, we show Figure 2 for a better view. From the charts, the following conclusions are summarized and are consistent for both ECAPA-TDNN and tResNet backbones:
1) As illustrated in Figure 2, by comparing the solid color bars with the patterned color bars while ignoring the difference of the colors, we find that most patterned color bars are shorter. It indicates that the systems with proposed self-supervision  $\mathcal{L}_{\mathrm{ssp}}$  perform better than those without. It also proves the effectiveness of  $\mathcal{L}_{\mathrm{ssp}}$  for RecXi frameworks. The results in Table 3 show that the proposed self-supervision  $\mathcal{L}_{\mathrm{ssp}}$  leads to an overall  $5.07\% / 5.44\%$  average reductions (system #11 vs. #13, #12 vs. #14, #5 vs. #16, #15 vs. #17) in EER/minDCF for all RecXi systems.
2) By comparing  $\mathrm{RecXi}(\tilde{\phi})$  shown as the solid blue color bars and  $\mathrm{RecXi}(\tilde{\phi},\tilde{\phi}_{\mathrm{lin}})$  shown as solid orange color bars, we conclude that when  $\mathcal{L}_{\mathrm{ssp}}$  is not applied, the systems derived only from  $\tilde{\phi}$  perform worse than those using both  $\tilde{\phi}$  and  $\tilde{\phi}_{\mathrm{lin}}$  in most of the trials (system #13 vs. #14 and #16 vs. #17). When we consider the patterned color bars, however, we find that these gaps are overcome, and very similar performance is achieved when  $\mathcal{L}_{\mathrm{ssp}}$  is applied. We believe that the gaps between  $\mathrm{RecXi}(\tilde{\phi})$ -based systems and  $\mathrm{RecXi}(\tilde{\phi},\tilde{\phi}_{\mathrm{lin}})$ -based systems are caused by the constraints on the content disentanglement layer provided by  $\tilde{\phi}_{\mathrm{lin}}$ . This brings  $\mathrm{RecXi}(\tilde{\phi},\tilde{\phi}_{\mathrm{lin}})$  systems an advantage in content disentanglement and further improves the efficiency of disentangling speaker representation. When both kinds of systems are enhanced by the self-supervision loss  $\mathcal{L}_{\mathrm{ssp}}$ , a sufficient guide for disentangling content representations is provided, and it leads to a similar and improved performance. It shows the effectiveness and superiority of the proposed  $\mathcal{L}_{\mathrm{ssp}}$  and also proves that the speaker embedding benefits from the quality of disentangled content.
Additional experiments are provided in the Appendix, covering diverse aspects including visualization, the ablation study on three RecXi layers, comparisons between RecXi and SOTA systems using ASR models or contrastive learning, as well as the evaluation of the effectiveness of  $\mathbf{G}_t$ . For details, please refer to Appendix F, G, H, and I, respectively.
This work has some limitations: 1). The novel  $\mathcal{L}_{\mathrm{ssp}}$  is intuitive and effective but needs further investigation and improvement. 2). As mentioned in Appendix C.2, the number of mini-transition models for deriving  $\mathbf{G}_t$  is set as  $N = 16$ . In Appendix I, we verify the effectiveness and necessity of utilizing the proposed  $\mathbf{G}_t$ . However, a comprehensive investigation of this hyperparameter is not conducted. This hyperparameter can be further exploited as it is related to the acoustic features and dynamic components we wish to disentangle.
The discussion about broader impacts is available in Appendix E.
We propose the RecXi – a Gaussian inference-based disentanglement learning neural network. It models the dynamic and static components in speech signals with the aim to disentangle vocal and verbal information in the absence of text labels and benefit the speaker recognition task. In addition, a novel self-supervised speaker-preserving method is proposed to relieve the effect of text labels absent for fine content representation disentanglement. The experiments conducted on both VoxCeleb and SITW datasets prove the consistent superiority of the proposed RecXi and the effectiveness of the proposed self-supervision method. We expect that the proposed model is applicable to automatic speech recognition (ASR), where speaker-independent representation is desirable. In addition, the disentangled content and speaker embeddings are useful in the voice conversion and speech synthesis tasks. For future work, we plan to reconstruct speech signals and utilize the interaction between these two tasks to benefit each other.
This work is supported by the Agency for Science, Technology and Research (A\*STAR), Singapore, through its Council Research Fund (Project No. CR-2021-005). Haizhou Li is in part supported by the National Natural Science Foundation of China (Grant No. 62271432) and the Agency for Science, Technology and Research (A\*STAR) under its AME Programmatic Funding Scheme (Project No. A18A2b0046). The authors would like to thank the reviewers and the meta-reviewer for their comments, which greatly improved the article.
[82] Yanfeng Wu, Chenkai Guo, Junan Zhao, Xiao Jin, and Jing Xu. RSKNet-MTSP: Effective and portable deep architecture for speaker verification. Neurocomputing, 511:259-272, 2022.
[83] Weidi Xie, Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Utterance-level aggregation for speaker recognition in the wild. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5791-5795, 2019.
[84] Chang Zeng, Xin Wang, Erica Cooper, Xiaoxiao Miao, and Junichi Yamagishi. Attention back-end for automatic speaker verification with multiple enrollment utterances. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6717-6721, 2022.
[85] Haoran Zhang, Yuexian Zou, and Helin Wang. Contrastive self-supervised learning for text-independent speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6713-6717, 2021.
[86] Jing-Xuan Zhang, Zhen-Hua Ling, and Li-Rong Dai. Non-parallel sequence-to-sequence voice conversion with disentangled linguistic and speaker representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:540–552, 2019.
[87] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rnnt loss. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829-7833, 2020.
[88] Siqi Zheng, Yun Lei, and Hongbin Suo. Phonetically-aware coupled network for short duration text-independent speaker verification. In Proc. Interspeech, pages 926-930, 2020.
[89] Tianyan Zhou, Yong Zhao, Jinyu Li, Yifan Gong, and Jian Wu. CNN with phonetic attention for text-independent speaker verification. In Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 718-725, 2019.
[90] Tianyan Zhou, Yong Zhao, and Jian Wu. ResNeXt and Res2Net structures for speaker verification. In IEEE Spoken Language Technology Workshop (SLT), pages 301-307, 2021.
[91] Yi Zhou, Xiaohai Tian, and Haizhou Li. Language agnostic speaker embedding for cross-lingual personalized speech generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3427–3439, 2021.
[92] Yi Zhou, Xiaohai Tian, Haihua Xu, Rohan Kumar Das, and Haizhou Li. Cross-lingual voice conversion with bilingual phonetic posteriogram and average modeling. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6790–6794, 2019.