Disentangling Voice and Content with Self-Supervision for Speaker Recognition
Authors: Tianchi Liu
Kong Aik Lee
Qiongqiong Wang
Haizhou Li
Venue: NeurIPS 2023 poster (2023)
Keywords: speaker recognition
disentanglement learning
self-supervision

Abstract: For speaker recognition, it is difficult to extract an accurate  speaker representation from speech because of its mixture of speaker traits and content. This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is realized with the use of three Gaussian inference layers, each consisting of a learnable transition model that extracts distinct speech components. Notably, a strengthened transition model is specifically designed to model complex speech dynamics. We also propose a self-supervision method to dynamically disentangle content without the use of labels other than speaker identities.  The efficacy of the proposed framework is validated via experiments conducted on the VoxCeleb and SITW datasets with 9.56\% and 8.24\% average reductions in EER and minDCF, respectively. Since neither additional model training nor data is specifically needed, it is easily applicable in practical use.

Disentangling Voice and Content with Self-Supervision for Speaker Recognition
Tianchi Liu 1,, Kong Aik Lee* Qiongqiong Wang', Haizhou Li
Institute for Infocomm Research (I²R), Ageney for Science, Technology and Research (A STAR), Singaporc
2 Dept. of Electrical and Computer Engineering, National University of Singapore, Singapore
Dept. of Electrical and Electronic Engineering, Hong Kong Polytechnic University, Hong Kong
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
(1iu_ tianchi, wang giinggginngnnnneetn. a-star, eg. kongeik, leetieee. org, baizbouli edu.
Abstract
For speaker recognition, it difficult to extract an accurate speaker representation from speech because of its mixture of speaker traits and content. This paper pro- poses a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is realized with the use of three Gaussian inference layers, each consisting of a learnable transition model that extracts distinct speech components. Notably, strengthened transition model is specifically designed to model complex speech dynamics. We also propose a self-supervision method dynamically disentangle content without the of labels other than speaker identities. The efficacy of the proposed framework is validated via experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average re- ductions in EER and minDCF, respectively. Since neither additional model training nor data is specifically needed, is easily applicable practical use.
Introduction
Automatic speaker recognition aims to identify a person from his/her voice 2 based on speech recordings (22). Typically, two fixedddimensional representations are extracted from the enrollment and test speech utterances, respectively [68). Then the recognition procedure is done by measuring their similarity (22). These representations are referred to as the speaker embeddings [67]. The concept of speaker embedding is similar that of the face embedding [64 in the face recognition task and the token embedding [16] in the language model, while the main difference lies in the carriers of the information source and the nature of downstream tasks. Different from the discrete sequential inputs in Natural Language Processing (NLP) and continuous inputs in Computer Vision (CV), speech signals are continuous-values variable-length sequences [24]. Speech signals contain both speaker traits and content 332. For speaker recognition, to form refined speaker embedding of a speech signal, conventional methods aggregate the frame-level features by pooling across the time-axis to extract speaker characteristics while factoring content information. Simple temporal aggregation fails to disentangle the content information and therefore affects the quality of the resulting embeddings. The content information is regarded as unwanted variations hindering the accurate representation of voice characteristics.
To reduce the effects of the content variation, prior works model the phonetic content representation and use it as a reference for speaker embedding extraction. In [89) the phonetic bottleneck features from the last hidden layer of pre-trained automatic speech recognition (ASR) network are combined with raw acoustic features to normalize the phonetic variations. A coupled stem is designed in 88 to jointly learn acoustic features and frame-level ASR bottleneck features. In [42,, a phonetic attention mask dynamically generated from a sub-branch is used to benefit speaker recognition. The existing methods can be summarized into two types in terms of content representation modeling: (1) by
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

89 46 and (2) by jointly trained multi-task model with extra pre-trained ASR model [57 34 45
components for content representations utilization of content representation benefits speaker recognition. However, both types lead to an obvious limit in practical applications. The employment of a pre-trained ASR model greatly increases the parameters and computation complexity in the inference, as the ASR models are typically one or two orders of magnitude larger than the speaker recognition model (1) The joint training of the speaker recognition model and extra components for content representations requires either an extra dataset with text labels in addition to the dataset with speaker identities, or one dataset with speaker identities and text labels simultaneously which is expensive and not easy come by. For the former type of joint training, additional components are still needed with extra effort and may encounter optimization difficulties.
46 80 42]. These methods prove that the
For the purpose of speaker recognition, one aims to derive an embedding vector representing the vocal characteristics of the speaker. The benefits of leveraging content representation are significant, while the drawbacks of text labels and extra model requirements are obvious. Motivated by this, we seek to design novel framework to solve these problems.
Speech signals consist of many components, of which the two major parts are speaker traits and speech content (32 14]. In this paper, we focus on decomposing speech signals into static and dynamic components. The former is static, i.c., fixed, with respect to temporal evolution and dominated by speaker characteristics, while the latter mainly consists of speech content with some other components, such as the prosody. Based on this assumption, we design framework with three Gaussian inference layers which aims to decompose the static and dynamic components of the speech. The static part is modeled by static Gaussian inference with the criterion of speaker elassification loss. During inference, the static latent factor is associated with the vocal characteristics of the speaker, and we refer to it as speaker representation. The remaining dynamic components related to verbal content are modeled by dynamic Gaussian inference. The motivation of the three-layer design is simple and intuitive - the speaker representation from layer / not accurate and may include content information as the content-related reference is absent in training. However, it helps layer 2 extract the content representation which can be used as the reference for layer 3. Thus, a more accurate speaker representation is extracted in layer 3. It is worth noting that the framework is trained without text labels and no extra model or branch is employed thus not much increase in model size. This is achieved with the guide of self-supervision and considered content representations. We named this framework as Recurrent Xi-vector (RecXi). The major contributions are summarized as follows:
RecXi: A novel disentangling framework with the following features: (1) We enhance the xi-vector [33] with the ability to capture temporal information and name it the recurrent xi- vector layer. (2) A frame-wise content-awand transition model G, is proposed for modeling dynamic components of the speech. (3) A novel design of three layers of Gaussian inference is proposed to disentangle speaker and content representations by static and dynamic modeling with the corresponding counterpart removal, respectively.
A self-supervision method is proposed to guide content disentanglement by preserving speaker traits. It reduces the impact of the absence of text labels.
2 Related Work
Conventional Speaker Recognition Methods. Speaker embeddings are fixed-length continuous- value vectors extracted from variable-length utterances to represent speaker characteristics. They live a simpler Euclidean space in which distance can be measured for the comparison between speakers [78] A well-known example of the extractor is the x-vector framework (67], which mainly consisted of the following three components: 1). An encoder is implemented by stacking multiple layers of time-delay neural network (TDNN) and used to extract the frame-level features from utterances [58]. Recent works strengthen the encoder by replacing the simple TDNNs with powerful 84 83] 5T).
ECAPA-TDNN [1]] and its variants 41 74 23 53 47, or ResNet series models [90 36
2). A temporal aggregation layer aggregates frame-level features from the encoder into fixed-length iiterance-levell representations. 3). A decoder classifies the utterance-leve representations to speaker classes for supervised learning by a classification loss. The decoder stacks several fully-connected layers including one bottleneck layer used to extract speaker embedding.
Speech Disentanglement. Various informational factors are carried by the speech signal, and the speech disentanglement ultimately depends on which informational factors are desired and how they

---

will be used [8]] For speaker recognition, in addition to the use of content information we introduced above, some works attempt to disentangle speaker representation with the removal of irrelevant information, like devices, noise, and channel information with corresponding labels (50 52 28
[52] minimizes the mutual information between speaker and device embeddings, with the goal of reducing their interdependence. In [40], nuisance variables like gender and accent are removed from speaker embeddings by learning two separate orthogonal representations. Many other speech-relevan tasks also show great improvements by disentangling these two components properly. For speaker diarization, the ASR model is employed to obtain content representations leveraged by speaker embedding extractions 770, [2] In voice conversion personalized voice generation tasks, popular method is to disentangle the speech into linguistic and speaker representations before performing the 91 38) In ASR, speaker information is extracted for speaker variants
generation 26 86
18
removal for performance improvements [49
35 25 or privacy preserving [69]. Similarly, auxiliary
network-based speaker adaptation [63] residual adapter [75] are explored to handle large variations in the speech signals produced by different individuals. These works have led us to the design of RecXi, which is the first speaker and content disentanglement framework for speaker recognition in the absence of extra labels for practical use to the best of our knowledge.
Self-Supervision in Speech. Self-supervised learning has been the dominant approach for utilizing unlabeled data with impressive success [ 16 7 20). In speaker recognition, contrastive learning [9] is used to force the encoder to produce similar activation between positive pair. The positive pair can be two disjoint segments from the same utterance (6, 85) or from cross- rees-reffrnn between speech and face data [73. The trained speaker encoder also can produce pseudo speaker labels for supervised learning [72 15. In target speaker extraction, self-supervision is helpful to find the
speech-lip synchronization [56]. In voice conversion, much research focuses on applying variational
auto-encoder (VAE) to disentangle the speaker [TO] 27 [59] Similar to reconstruction- baaed methods, mutual information (MI) and identity change loss are used in 30 for robust speaker disentanglement. Content [60] disentangles and removes speaker variations from the speech representation model HuBERT 24 for various downstream tasks, such as language modeling and zero-shot content probe.
Considering the main objective of this work is to benefit speaker recognition by disentangling the speaker with the speaker classification supervision, we want to clarify that the content disentanglement and corresponding self-super serve this final target. Therefore, the self-supervision method is designed to be simple yet effective while avoiding extra signal re-constructors or training efforts.
3 Approach
3.1
Xi-vector
Many works have been proposed to estimate the uncertainty for the speaker embedding 65 Among them, the xi-vector (33 is proposed to enhance x- v-vect embedding in handling the uncertainty caused by the random intrinsic and extrinsic variations of human voices. Specifically, an input frame x, is encoded to a point-wise estimate 2, in a latent space by an encoder. And an auxiliary network is employed to characterize the frame-level uncertainty with posterior covariance matrix L," associated with z, as shown in the encoder part of Figure| These two operations are formulated as
33 78
2t = Senc Xx x
(1)
logL, = Jenc [x, x +*)
(2)
where the neighbour frames x + of time t are taken into consideration for the frame-wiss estimation. The precision matrices L, are assumed to be diagonal and are estimated as log preeriiciooon for the convenience of following steps. is assumed that a linear Gaussian model responsible to generate the representations 2; as follows:
PF'), uncertainty : 6 NN L, (3) generative model: z, h e,, latent variable: h ~ [o (ppp.Ppp)
Here, is the latent variable for the entire utterance, and 6, is a frame-wise random variable for the uncertainty covariance measure, op and P- are the Gaussian prior mean and covariance matrix of the variable h. The posterior mean vector Φ and precision matrix P, for the whole sequence are formulated
T
Φ = P
EL.
L.2, + Pppp
P EL+P
(5)
where the posterior mean &h enriched by frame-wise uncertainty and is further used to decoded into speaker embedding.
3

---

Superised Loss
Fix
Presenvng
Precitt
Prodict
Prodkt
Prodict
Beeaker Classiccaionn
Cuz
Speaker Freserving
Giyy Po
do P.f
92, P:f
Oz, Prf
Filly Corracted
Racki
Update
Update
Enowidig
Layer
pi. +
A
AGz
AG
Decoder
G
G₁
Pr
Gt
gem
Decoder
(Rrr (Te Lay.. Ley)
40
Update
Updatte
Pj-11 9 1
Update
é.p,
Layer
Connocted
dq.
65. P.
oy.
Temporal Aggregation
Futly Connectid
Predict
Predict
Predict
Audillary Network
[Rt, ig.
P.f
Paf
Pr.
[Ly Iey. Loy
Updata
Update
Updata
Pactioons Netwoork
Pi
Encoder
(x.. Xy]
Layer
Encoder
X (X1.
Z. Lec.s
Z, S.
Figure 1: The network architecture the proposed RecXi system with self-supervision. The structure at the bottom-middle of the figure is simplified speaker recognition system. The figures in three dotted boxes are the specific explanations of its three parts. At the encoder, {x1, X2. } the input sequence of length T. The three colors of blue, orange, and green indicate three recursive layers. For each layer, the inference flow for frames X1, X2 and Xy is drawn, while the frames in between are replaced by ...'. The dashed lines indicate recurrent operation and the solid lines represent the operations within the same frame. The block with G, gen indicates the filter generator. For the decoder, O is subtraction operation. The dotted line indicates the operation is optional.
3.2 Disentangling Speaker and Content Representations
Basic Recurrent Xi-vector Layer. Xi-vector has the advantage of modeling static components of the speech by the utterance-level speaker representation aggregation with the use of uncertainty estimation, while the drawback of modeling dynamic signals is obvious. For approximating non- linear functions in nigh-dimensional feature spaces and restricting the inference through the adjacent Gaussian hidden state efficiently, similar to [4) a learnable linear transition model G is applied. We further extend the xi-vector into recursive form with frame-level estimation. It is worth noting that xi-vector is special case
basic recurrent xi-vector when is an identity matrix.
The Gaussian inference in the latent space can be implemented in two stages: predict stage and update stage. In the predict stage, the predictive mean vector p." and precision matrix Φ are formulated as
Gpt-
(6)
+ = -G
(7)
where G represents a linear transition model, and indicates the results of the given frame after the predict stage. T indicates transpose operation.
For sake of clarity, assign the time (i.e., frame) index i 0 to the priors, such that Po Pp- To = =Φp. In the update stage, the posterior mean vector of frame t is derived by incorporating the previously predicted posterior mean and precision {pī-1: ) of the hidden states of the former frame with encoded features z, and estimated uncertainty L, of current frame l. The uncertainty measures L, of frame is added into the posterior precision matrix. These two operations are derived as
Pr 4 L,Z, + 11PPt
(8)
Φ L,
Frame-wise Content-aware Transition Model G,, Speech signals are a complex mixture of dynamic information sources. Even though the transition model G is learnable during training, it remains the same across all the hidden states of Gaussian inference for each sample. To enhance the ability to model dynamic speech components for content disentanglement, propose transition

---

model G2 which is dynamically adjusted by a filter generator for each frame according to the content during the Gaussian inference and hereby named trame-wise content-aware transition model.
Specifically, a set of N learnable transition matrices ,, G.2...G' Ny is employed to model N dynamic components in speech signals. For each frame a vector w, IR is generated representing the importance of different dynamic components to content representation modeling, by observing content information Pt from update stage. The content-aware transition model for frame t is finally obtained as weighted sum over the N component-dependent transition models with weight Wr,n.
G. - Σ Wtnn G'n,
(10)
Wtnn
[welp
(11)
W; = a (J (pt)),
(12)
where W. E R.. and f indicate the Softmar function and non-linear operation, respectively. In this work, the non-linear operation designed as two fully connected layers with a ReLU [19] activation function in between. It is to be noted that this procedure is frame-wise as the dynamic components vary along the sequence of T. As an extension to Equations (6) and (7. the predict stage of recurrent xi-vector is reformulated
pt = Gipt,
(13)
iG G.
(14)
Three Layers of Gaussian Inference. We propose a structure based on three layers of Gaussian inference for disentangling speaker and content representations in the absence of text labels. In this work, this structure is utilized in the temporal aggregation layer and named RecXi. The three layers of Gaussian inferences each aim at precursor speaker representation, disentangled content representation, and disentangled speaker representation, as shown in the RecXi part of Figure [ The last frame's 's representations from each layer are used for deriving embeddings within the decoder. The details of each layer are discussed as follows.
Layer 1: Precursor Speaker Representation. This a basic recurrent xi- xi-tertor layer that aims to represent the speaker characteristics. Therefore, the transition model of the Gaussian inference is set as an identity matrix to model static components of the speech. The predict stage can be derived from Equations (6) and as
= or.
(15)
P+=
(16)
The speaker representation from this layer is similar to that in the original xi-vector, where the content information from the nigh-dimensional frame-level features remains and affects the speaker embedding quality. We refer to the representation of this layer as the precursor speaker representation. Specifically, the frame-wise representations of posterior mean db and precision P, of the hidden state h of frame are estimated. They are derived partly according to frame-level features 2t and uncertainty L, extracted from the corresponding frame and partly from previous frames {1, 21, ...t- - }} by recursively passing in the posteriors from the previous frame. The update stage is formulated as
Φ. P. L, + PT-0t1)
(17)
L
(18)
Layer 2: Disentangled Content Representation. This layer aims to disentangle content representa- tion from the sequence. To model dynamic subtle content changes, the frame-wise content-aware transition model G, introduced above applied. As shown the orange part of Figure the transition model G, is generated by filter generator according to Equations (TO) and (22) Equations
(3 and (14) are applied to the predict stage of layer 2.
To disentangle the content representation more effectively, in addition to equipping the layer with dynamic modeling ability, we further attempt remove speaker information from the frame-level features for each frame during Gaussian inference. This ensures that the remaining information is more likely to be dynamic and associated with the content.
The posterior mean + and posterior precision P+ of the hidden state of layer / are rich with speaker information and utilized by the update stage of layer 2. Benefiting from the linear operations in this three layers design, the speaker removal operation can simply be done by subtracting the posterior mean + from features z while adding the posterior covariance matrix (P+)-1 into the uncertainty estimation matrix L -1 The procedure is dynamically processed for each frame, and the update stage formulated
5

---

Pt = (I, + (--))ー)--(a0 xt) + x- -IPt-il
(19)
2. 2 (L (P))- 27-
(20)
Layer 3: Disentangled Speaker Representation. This layer uses the Gaussian inference with an identity matrix for modeling static components of the speech. Furthermore, the speaker classification loss is applied to the output of this layer and provides supervision restrictions. Different from layer I, layer is designed to model the desired disentangled speaker representation by removing the content information provided by layer 2 from the frame-level features. The procedure is similar to that in layer 2, and the predict stage is formulated as
(21)
P7 = P..
(22)
where क5 and P are the posterior mean and posterior precision, respectively. The symbol of is used differentiate them from the posteriors in layer The update stage formulated as
3. (L, + (+:)) (z+ pt) p و
(23)
(L, + P
(24)
is worth noting that to avoid expensive matrix multiplication operations and numerically problematic matrix inversions, simplified implementation is adopted (see Appendix A).
3.3 Speech Disentanglement with Self-superviiion
A well-trained speaker embedding neural network requires a huge number of speakers and utterances achieve discriminative ability. For such large dataset, text labels are very difficult come by. In the absence of text labels, disentangling content information from the speech is a very difficult task.
In Section 3.2] the layer 3 of RecXi is designed to be the desired speaker representation. This is achieved by static Gaussian inference with the assumption that disentangled content representation provided by layer is reliable. We note that the disentangled speaker representation from layer 3 is directly optimized through the classification criterion, while the optimization for disentangling content information is indirect through content removal operations in Gaussian inference. In order to ameliorate the shortcoming due to the absence of text label and to provide an extra supervisor for layer 2, we propose guff-supevviioon method to preserve speaker information via knowledge distillation in a similar fashion to those proposed in 66 76 g
for the teacher-student pair.
Generally speaking, large models usually outperform small models, while the small model is compu- tationally cheaper (76). Knowledge distillation aims to benefit a small model with the guidance of a large model. Different from the general knowledge distillation methods, our 'teacher' and 'student' are two different layers within the RecXi. Since this guidance comes from the RecXi itself, and all the training data is considered as unlabelled data for the content disentanglement task, the proposed method is considered a self-supervision method
The output & from layer I of RecXi is precursor speaker representation, and the content information remains, while layer 2 is designed to disentangle content representation p. Benefiting from the linear operations used in RecXi, we can remove content information and preserve speaker representation by subtracting the content representation p of layer from the precursor speaker representation & of layer I. This speaker representation marked as dlin and derived as
(25) where the 'lin' indicates that it is obtained by a linear operation, differing from Ō in layer 3 which disentangled by Gaussian inference. For the same input sequence, the speaker representation din derived by this linear operation should be consistent with á obtained from disentanglement. Therefore, by restricting their difference, the gradient from supervision speaker classification loss will form a guide for layer through the linear operation in Equation (25). This serves as an extra supervisory signal different from the constraints imposed by speaker removal operation during the Gaussian inference. As ò is optimized by classification loss directly, it is considered as 'teacher',
Din Φ P

---

while oin is 'student". Since oim is derived by preserving speaker representation in Equation 25 we name the loss self-supervision speaker preserving loss Sepp
The Leap loss can be generated by different knowledge distillation methods, simple comparison is available in Appendix D. In this work, the idea of menilaiity-preseeviin loss [76] is in line with our layer-wise design and is inherited to guide the student towards the activation correlations induced in the teacher, instead of mimicking the teacher's representation space directly, is derived as
Saap
E (B,0")E
(11300200)
(s')T iin
(26)
62
where i collects all the (s, 8') pairs from the same mini-batch, II-ll2 and II-Il indicate the row-wise L2 normalization and Frobenius norm, respectively. T indicates a transpose operation and indicates the batch size. We define the total loss for training the framework
Liotal aCels + BLapps
(27)
where Lets is the speaker classification loss, a and are hyperparameters for balancing the total loss.
Experiments Setup
4.1 Dataset, Training Strategy, and Evaluation Protocol
The experiments are conducted on VoxCeleb1
54 VVxXCleb22 13 and the Speaker in the Wild (SITW) [48] datasets. is worth noting that the text labels are not available for these datasets. For fair comparison, the baselines are all re-implemented and trained with the same strategy as RecXi which follows that in ECAPA-TDNN [5] All the models are evaluated by the performance in terms of equal error rate (EER) and the minimum detection cost function (minDCF), Detailed descriptions of datasets, training strategy, and evaluation protocol are available Appendix
4.2 Systems Description
We evaluate the systems that are combinations of different backbone models in the encoder and different aggregation layers. To verify the compatibility of the proposed RecXi, both 2D convolution (Conv2D)-basee and time delay neural network (TDNN)-based backbones are adopted:
1) ECAPA-TDNN [5 is a sfate-of-the-arr speaker recognition model based on TDNNs. The layers before the temporal aggregation layer are considered the backbone network.
2) ResNet 21 and tResNet represent the models based on 2D convolution. The modified ResNet34 77 is adopted as a baseline. order to model more local regions with larger frequency bandwidths at different scales, we further modify the ResNet34 in 777 by simply changing the stride strategy and name (ResNet. The details are provided in Appendix B.
The following are temporal aggregation layers. The first three are considered baselines:
Temporal Statisties Pooling (term TSP from here onwards) [66] is a well-known aggregation method. It is the default option for x-vector and adopted in ResNet for speaker recognition [12 [0].
2) Channel- and Context-dependert Statistics Pooling (term as Chan.&Con. from here onwards) 15 is the default aggregation layer in ECAPA-TINN, modified from the attentive statistics pooling (55) by adding -hannel-dependent frame attention and allowing the self- attention produced across global properties.
3) Xi-vector Posterior Inference (term as Xi from here onwards) 333 inserts uncertainty estimation into speaker embeddings as introduced in Section 3.1
4) RecXi (6) and RecXi (ò, dun) are the proposed methods. The former uses only ò the input to the decoder to derive the speaker embedding, and the latter uses a concatenation of both and dia as the input.
7

---

Table 1: Performance in (ER(%) and minDCF of the ofate-of-the-art (SOTA) systems and proposed RecXi on VoxCelebl and SITW test sets. Aug. indicates whether the system is trained with data augmentations. Para. is the number of parameters in million. Systems with f are re-implemented. The systems with & following the format of "aggregation layer + backbone model' are baselines. # is the index number for the implemented systems.
Aug.
Para
VoxCeleb1-O
VoxCelebI-HI
VoxCelebI-E
SITW eval minDCH
System
EER minDCF EER
minDCF EER
minDCF EER
ResNec34-SKDDFF,
39
X
5.98
1.44
0.168
2.76
0.278
1.59
0.179
WAASP(AAM-soltmmx
X
8.0
1.25
2.78
1.56
H/AAP(AAPsSofmmax
8.0
1.21
2.77
1.42
ST-NNe50 3
X
1.28
2.50
1.28
2.54
RSKNet-MTSP 82
X 13,9
1.05
0.159
2.52
0.237
1.30
0.152
2.27
0.228
ECAPA-TDNN 15
X
6.19
1.377
0.137
2.674
0.235
1.451
0.153
2.542
0.208
ResNet34 77
X 6.63
1.489
0.155
2.500
0.224
1.423
0.158
2.378
0.208
TSP proposed tResNeti
6.21
1.396
0.135
2.257 0.204
1.281
0.141
2,250
0.200
4
Xi [33
proposed (ResNetE
6.54
1.295
0.120
2.169
0.202
1.224
0.142
2,150 0.189
5 RecXi(e, dlin) with Cop (ResNet
7.06
1.196
0.122
2.097
0.196
1.197
0.124
1,832 0.172
Res2Net-26w8s 90
9.3 1.45
0.147
2.72
0.272
1.47
0.169
H/ASP(AAM-softmax)
8.0 1.15
2.49
1.35
H/ASP(AP+sofImax)
3
8.0 0.88
2.21
1.07
ECAPA-TDNN [5
V
6.2
1.01 0.127
2.32 0.218
1.24 0.142
MFA-TDNN (standard)
1
7.32
0.856
0.092
2.049
0.190
1.083 0.118
MFA-TDNN (lite)
5.93
0.091
2.174 0.199
1.138 0.121
61
V
0.87
0.115
2.31 0.222
1.22
0.135
UP-LS UP-PLDA
791
1.01
0.124
2.18
0.224
1.59 0.167
ECAPA-TDNN 15
v
6.19
1.127
0.145
2.249
0.223
1.194
0.133
1.668 0.155
ResNet34 777t
6.63
1.101
0.128
2.221
0.208
1.252
0.139
1,584 0.161
Xi [33 + proposed tResNets
6.54 0.936
0.115
1,942
0.186
1.110
0.125
1.394 0.145
RecXi(o, (ua) with Cee tResNet
7.06 0.984
0.091
1.857
0.179
1.075
0.114
1.340 0.137
Results and Discussions
In [ Table 2 3 we report the performance in terms of EER and minDCF of the proposed RecXi, various baselines and SOTA systems. It is worth noting that the VoxCeleb1-O0 test set much smaller and easier compared to the other three test sets, which may lead to less convincing results.
Comparison with SOTA methods and baselines. In Table I we compare the baseline systems and proposed RecXi systems with SOTA methods. The baseline systems with proposed the tResNet backbone (system #3 and #4) achieve the SOTA performance. Compared to the TSP method (system #3), the use of Xi (system #4) achieves overall improvements with the 5.02% 19% average reductions in EER/minDCF. This also renders further performance improvements more difficult. By comparing systems #4 and #5, we observe that the proposed RecXi leads to relative improvements with 6.96%/5. 82% average reductions in EER/minDCF compared to the Xi baseline system. The proposed RecXi (system #5) obviously further improves the performance.
Experiments with data augmentation. In the second half of Table I we report the results with aug- mented data (see Appendix C.2). The re-implemented ECAPA -ANNN (system #6) achieves slightly better performance than that reported in [15]. Consistent with the conclusions of training without data augmentation discussed above, the baseline (system #8) with proposed tResNet outperforms all SOTA methods and the proposed RecXi (system #9) shows great superiority over all other systems.
Comparison for ECAPA-TDNN backbone- systems. The experiments discussed above are for Conv2D- based networks. For speech-related tasks, the TDNN- networks are widely used 53 In order to verify the superiority of the proposed method under different
as well 74 15 conditions, the experiment results for the systems based the popular ECAPA-TDNE [15] backbone
8

---

Table 2: Performance in EER(%) and minDCF of the baselines and proposed RecXi with ECAPA-
TDNN backbone [[]] on VoxCelebl and SITW test sets. is the index number for the system.
Aggregation
Params (Million)
VoxCelehI-O
VoxCelebI-H
VoxCelehI-F
SITW eval
Layer
EER
minDCF
EER
minDCF
EER
minDCF
EER
minDCF
Chan
6.19
1.377
0.137
2.674
0.235
1.451
0.153
2.542
0.208
10
Xi
5.90
1.340
0.140
2.647
0,237
1.429
0.148
2.679
0.204
11 RecXilo, (un) with Coop
6.43
1.196
0.107
2.467
0.227
1.292
0.141
2.105
0.184
are reported in Table 2 Compared to the original aggregation layer of ECAPA- - (system #1), the use of xi-vector posterior inference (system #10) performs slightly better. The comparison between system #10 and #11 presents that for the ECAPA-TDNN backbone, the proposed RecXi achieves average EER/minDCF reductions of 12.16//10 over the Xi baseline.
Brief summary. Since the Xi baseline performs the best among all baselines and SOTA systems discussed above with different training conditions and backbone networks, we refer to it as a SOTA baseline for the following discussion. We observe that compared to the SOTA baseline, the proposed RecXi consistently improves the performance for both backbone networks on all four test sets with overall average EER/minDCF reductions of 9.56% 24.%% (system #4 vs. #5 and #10 vs. #11). The experiment results remain consistent regardless of the backbone types and whether augmented data used. This may be due to the fact that the proposed disentanglement framework disentangles the static speaker components effectively and benefits speaker recognition. As the speaker is disentangled under the assistance of disentangled content representation, the significant improvement also proves the quality of the dynamic counterpart modeling. In addition, found that the performance of tResNet- based systems is generally better than that of the systems with ECAPA-TDDE ADNNN backbone.
Table 3: Performance in EER(%) and minDCF of various RecXi systems on VoxCelebl and SITW test sets for ablation study. # the index number for the system. BN and Para. indicate the type of backbone network and number of parameters in million, respectively.
BN
Aggregation Layer
VoxCelebI-II
VoxCelebI-E
SITW eval
Relative Reduction EER
Loep
EER
minDCF
EER minDCF
EER
minDCF
minDCF
ECAPA
RecXito, dua)
6.43
2.467
0.227
1.292
0.141
2.105
0.184
-7.58%
-12.18%
12
ECAPA
RecXi(o)
/
6.13
2
0.233
1.286
0.139
2.160
0.191
-7.30%
-11.04%
13
ECAPA
RecXi(gh, (mn))
x
6.43
2.477
0.228
1.326
0.142
2.351
0.196
-3.11%
-10.40%
ECAPA
ReeXi(())
x
2.498
0.229
1.325
0.146
2.597
0.273
Benchmark
5
(ResNet RecXilo, dti))
/
7.06
2.097
0.196
1.197
0.124
1.832
0.172
-5.78%
-5.83%
15
IResNet RecXi(())
6.73
2.117
0.192
1,215
0.128
1.750
0.177
-6.35%
-4.82%
IResNet
ReeXi(ço vnn)
2.130
0.198
1,222
0.128
1.886
0.184
-3.71%
-2.38%
IResNet
RecXi(())
x
6.73
2.185
0.204
1,230
0.128
2.050
0.193
Benchmark
ECAPA-TDNN
TRESNET BACKBONE
ECAPA-TDNN BACKBONE IN MINDCF
TRESNET BACKBONE IN MINDCF
BACKBONE IN EER
IN EER
2.5
2.5
0.25
0.2
0.2
1.5
1.5
0.1
0.1
Vox-H
Vex-E
Vox-H
Vox-E
SITW
Vox-H
Vox-E
SITW
RecXi( ())
RecXi v il)
IZ RecKi( + Lssp
G RecXi( & *0) Lssp
Figure 2: Bar charts for ablation studies. Performance in EER(%) and minDCF of proposed RecXi under different conditions drawn. The blue and orange each indicate RecXi(O) and RecXi( sin)). Patterned color bars at the left front and solid color bars at the right behind represent the performance with and without Leop for the same test trial, respectively. For all bars, the shorter the better.
Ablation Study. We perform ablation studies on the proposed RecXi. As the results shown in Table 3 are detailed but not intuitive, we show Figure 2 for better view. From the charts, the following conclusions summarized and are consistent for both ECAPA-TDNN and tResNet backbones:

---

1) As illustrated in Figure 2 by comparing the solid color bars with the patterned color bars while ignoring the difference of the colors, we find that most patterned color bars are shorter. It indicates that the systems with proposed self-supervisio Lesp perform better than those without. It also proves the effectiveness of Sep for RecXi frameworks. The results in Table shows that the proposed self-supervisiin Seap leads to an overall 107%/5.44%% average reductions (system #11 1 vs. #13, #12 vs. #14, #5 vs. #16, #15 vs. #17) in EER/minDCF for all RecXi systems.
2) By comparing RecXi(e) shown as the solid blue color bars and RecXi (á, Plin) shown as solid orange color bars, we conclude that when Casp is not applied, the systems derived only from perform worse than those using both & and Φia in most of the trials (system #13 vs. #14 and #16 vs. #17). When we consider the patterned color bars, however, we find that these gaps overcome, and very similar performance is achieved when Leep is applied. We believe that the gaps between RecXi(o)- (cassed systems and RecXi(ěe éline-based systems are caused by the constraints on the content disentanglement layer provided by @lin- This brings RecXi (ò. dum) systems an advantage in content disentanglement and further improves the efficiency of disentangling speaker representation. When both kinds of systems are enhanced by the self-supervvsion loss Lusp, a sufficient guide for disentangling content representations is provided, and it leads to a similar and improved performance. It shows the effectiveness and superiority of the proposed Soop and also proves that the speaker embedding benefits from the quality of disentangled content.
Additional experiments are provided in the Appendix, covering diverse aspects including visualization, the ablation study on three RecXi layers, comparisons between RecXi and SOTA systems using ASR models or contrastive learning. as well as the evaluation of the effectiveness of G, For details, please refer Appendix F, G. H. and respectively.
Limitations
This work has some limitations: 1). The novel Cesp is intuitive and effective but needs further investigation and improvement. 2). As mentioned in Appendix C.2, the number of nini-transition models for deriving G, is set as = 16. Appendix we verify the effectiveness and necessity of utilizing the proposed G, However, comprehensive investigation of this hyperparameter is not conducted. This hyperparameter can be further exploited as it is related to the acoustic features and dynamic components we wish to disentangle.
The discussion about broader impacts is available in Appendix E.
Conclusion and Future Work
We propose the RecXi Gaussian siference-based disentanglement learning neural network. It models the dynamic and static components in speech signals with the aim to disentangle vocal and verbal information in the absence of text labels and benefit the speaker recognition task. In addition, novel self-supervised speaker-preserving method is proposed to relieve the effect of text labels absent for fine content representation disentanglement. The experiments conducted on both VoxCeleb and SITW datasets prove the consistent superiority of the proposed RecXi and the effectiveness of the proposed self-supervision method. We expect that the proposed model is applicable automatic speech recognition (ASR), where speaker-indepenee independent representation is desirable, In addition, the disentangled content and speaker embeddings are useful in the voice conversion and speech synthesis tasks, For future work, we plan to reconstruct speech signals and utilize the interaction between these two tasks benefit each other.
8 Acknowledgements
This work is supported by the Agency for Science, Technology and Research (A (A*ST))), Singapore, through its Council Research Fund (Project No. CR-202] CR-2021-005), Haizhou Li is in part supported by the National Natural Science Foundation of China (Grant No. 62271432) and the Agency for Science, Technology and Research (A STAR) under its AME Programmatic Funding Scheme (Project No. A18226004)). The authors would like to thank the reviewers and the meta-reviewer for their comments, which greatly improved the article.
10

---

References
[1] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0:
A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems (NeurIPS), 15:12499-12460) 2020.
[2] Zhongxin Bai and Xiao-Lei Zhang. Speaker recognition based on deep learning; An overview. Neural Networks, 140:65-99, 2021.
[3] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304. /2210, 2023.
[4] Philipp Becker, Harit Pandya, Gregor Gehhardt, Cheng Zhao, C James Taylor, and Gerhard Neumann. Recurrent kalman networks: Factorized inference in high-dimensional deep feature spaces. In International Conference on Machine Learning (ICML), pages 544-552, 2019.
[5] Andrew Brown, Jaesung Huh, Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxsre 2021: The third voxceleb speaker recognition challenge. arXiv preprint arXiv:2201.0455 2022.
[6] Danwei Cai, Weiqing Wang, and Ming Li. An iterative framework for self-supervised deep speaker representation learning. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6728 07386,7,,225 2021.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 33:9912-9999 2020.
[8] Keshigeyan Chandrasegaran, Ngoc- Tran, Yunqing Zhao, and Ngai-Man Cheung. Re- visiting label smoothing and knowledge distillation compatibility: What missing? In International Conference on Machine Learning (ICML), pages 2890-2916, 2022.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597-1607, 2020.
[10] Ju chich Chou. Cheng chich Ych, Hung yi Lee, and Lin shan Lee. Multi-target voice conversion without parallel data by adversarially learning disentangled audio representations. In Proc. Interspeech, pages 501-505, 2018.
[11] Ju chieh Chou and Hung- Yi Lee. One-shot voice conversion by separating speaker and content representations with instance normalization. In Proc. Interspeech, pages 664-668, 2019.
[12] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee-Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Jin Lee, and lcksang Han. In defence of metric learning for speaker recognition. In Proc. Interspeech, pages 2977-2981, 2020.
[13] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Deep speaker recogmition. In Proc. Interspeech, pages 1086-1090, 2018
[14] Rohan Kumar Das, Sarfaraz Jelil, and S. R. Mahadeva Prasanna. Significance of constraining text in limited data text-independent speaker verification. In International Conference on Signal Processing and Communications (SPCOM), pages 1-5, 2016.
[15] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynek. SCAPA-TDNN- Emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Proc. Interspeech, pages 3830-23 3834, 2020.
[16] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186 2019.
11

---

[17] Subhadeep Dey, Takafumi Koshinaka, Petr Motlicek, and Srikanth Madikeri. DNN based speaker embedding using content information for text-dependent speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5344-5348, 2018.
[18] Xiaoxue Gao, Xiaohai Tian, Yi Zhou, Rohan Kumar Das, and Haizhou Li. Personalized singing voice generation using wavernn. Proc. Odyssey, pages 252-258, 2020.
[19] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks, In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 315-323, 2011.
[20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach self-supervised learning. Advances in Neural 2020.
Information Processing Systems (NeurIPS),
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[22] Georg Heigold, Ignacio Moreno, Samy Bengio, and Noam Shazeer. End-to-end text-dependent speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5115-5119, 2016.
[23] Qian-Bei Hong, Chung-Hsien Wu, and Hsin-Min Wang. Generalization ability improvement of speaker representation and anti-interference for speaker verification. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:486-499, 2023.
[24] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions on Audio, Speech, and Language Processing,
2021.
[25] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and inter- pretable representations from sequential data. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 1878-1889, 2017.
[26] Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu, and James Glass. Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5901-5905, 2019.
[27] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Acvae-ve: Non- parallel voice conversion with auxiliary classifier variational autoencoder. /EEE/ACM Transac- tions on Audio, Speech, and Language Processing, 27(9):1432 1443, 2019.
[28] P. Kenny and P. Dumouchel. Disentangling speaker and channel effects in speaker verification. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 37-40, 2004.
[29] Aparna Khare, Eunjung Han, Yuguang Yang, and Andreas Stolcke. ASR-aware end-to-end neu- ral diarization. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8092-8096, 2022.
[30] Yoohwan Kwon, Soo-Whan Chung, and Hong-Goo Kang. Intra-class variation reduction of speaker representation in disentanglement framework. In Proc. Interspeech, pages 3231-3235 2020.
[31] Yoohwan Kwon, Hee-Soo Heo, Bong-Jin Lee, and Joon Son Chung. The ins and outs of speaker recognition: lessons from voxsre 2020. In IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5809-5813, 2021.
12

---

[32] Anthony Larcher, Kong Aik Lee. Bin Ma, and Haizhou Li. Text-dependent speaker verification: Classifiers, databases and rsr2015. Speech Communication, 60:56-77. 2014.
[33] Kong Aik Lee, Qiongqiong Wang, and Takafumi Koshinaka. Xi-vector embedding for speaker recognition. IEEE Signal Processing Letters, 28: 1385-1389, 2021.
[34] Yun Lei, Nicolas Scheffer, Luciana Ferrer, and Mitchell McLaren. A novel scheme for speaker recognition using a phonetically deep neural network. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1695-1699, 2014.
[35] Haoqi Li, Ming Tu, Jing Huang, Shrikanth Narayanan, and Panayiotis Georgiou. Speaker- invariant affective representation learning via adversarial training. In IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7144-7148, 2020.
[36] Na Li, Deyi Tuo, Dan Su, Zhifeng Li, and Dong Yu. Deep discriminative embeddings for duration robust speaker verification. In Proc. Interspeech, pages 2262-2266, 2018.
[37] Zhuo Li, Ce Fang, Runqiu Xiao, Wenchao Wang, and Yonghong Yan. Si-net: Multi-scale context-aware convolutional block for speaker verification. In Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 220-227, 2021.
[38] Jiachen Lian, Chunlei Zhang, and Dong Yu. Robust disentangled variational speech representa- tion learning for zero-shot voice conversion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6572-6576, 2022.
[39] Bei Liu, Haoyu Wang, Zhengyang Chen, Shuai Wang. and Yanmin Qian. Self-knowledge distillation via feature enhancement for speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7542-7546, 2022.
[40] Lixing Liu, Sayan Ghosh, and Stefan Scherer. Towards learning nuisance-fre representations of speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6817-682 6821, 2018.
[41] Tianchi Liu, Rohan Kumar Das, Kong Aik Lee, and Haizhou Li. MFA: TDNN with multi-scale frequency-channel attention for text-independent speaker verification with short utterances. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7517-7521, 2022.
[42] Tianchi Liu, Rohan Kumar Das, Kong Aik Lee, and Haizhou Li. Neural acoustic-phonetic approach for speaker verification with phonetic attention mask. IEEE Signal Processing Letters, 29:782-786, 2022.
[43] Tianchi Liu, Rohan Kumar Das, Maulik Madhavi, Shengmei Shen, and Haizhou Li. Speaker- utterance dual attention for speaker and utterance verification. In Proc. Interspeech, pages 4293 4293,4427,222 2020.
[44] Tianchi Liu, Maulik Madhavi, Rohan Kumar Das, and Haizhou Li. unified framework for speaker and utterance verification. In Proc. Interspeech 2019, pages 4320-4324, 2019.
[45] Yi Liu, Liang He, Jia Liu, and Michael T. Johnson. Speaker embedding extraction with phonetic information. In Proc. Interspeech, pages 2247-2251, 2018.
[46] Yi Liu, Liang He, Jia Liu, and Michael Johnson. Introducing phonetic information to speaker embedding for speaker verification. EURASIP Journal Audio, Speech, and Music Processing, 19:1-17, 2019.
[47] Yi Ma, Kong Aik Lee, Ville Hautamäki, and Haizhou Li. PL-EESR: Perceptual loss based end- to-end robust speaker representation extraction. In Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 106-113, 2021.
[48] Mitchell McLaren, Luciana Ferrer, Diego Castan, and Aaron Lawson. The speakers in the wild (SITW) speaker recognition database. In Proc. Interspeech, pages 818 822, 2016.
13

---

[49] Zhong Meng, Jinyu Li, Zhuo Chen, Yang Zhao, Vadim Mazalov, Yifan Gong, and Büing Hwang Juang. speaker-invarivan training via adversarial learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5969-5973, 2018.
[50] Zhong Meng, Yong Zhao, Jinyu Li, and Yifan Gong. Adversarial speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6216-6220, 2019.
[51] Xiaoxiao Miao, Ian McLoughlin, Wenchao Wang, and Pengyuan Zhang. D-MONA: A dilated mixed-order non-local attention network for speaker andlanguage recognition. Neural Nenworks, 139:201-211 2211, 2021.
[52] Sung Hwan Mun, Min Hyun Han, Minchan Kim, Dongjune Lee, and Nam Soo Kim. Disen- tangled speaker representation learning via mutual information minimization. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 89-96, 2022.
[53] Sung Hwan Mun, Jee-weon Jung, Min Hyun Han, and Nam Soo Kim. Frequency and multi- scale selective kernel attention for speaker verification. In IEEE Spoken Language Technology Workshop (SLT), pages 554, 2023.
[54] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: A large-scale speaker identification dataset. Proc. Interspeech, pages 2616-2620, 2017.
[55] Koji Okabe, Takafumi Koshinaka, and Koichi Shinoda. Attentive statistics pooling for deep speaker embedding. In Proc. Interspeech, pages 2252-2256, 2018.
[56] Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li. Selective listening by synchronizing speech with lips. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:1650-1664, 2022.
[57] Alex Seungryong Park. ASR dependent techniques for speaker recognition. PhD thesis, Massachusetts Institute of Technology, 2002.
[58] Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. A time delay neural network architecture for efficient modeling of long temporal contexts. In Proc. Interspeech, pages 3214-3218, 2015.
[59] Adam Polyak. Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In Proc. Interspeech, 2021.
[60] Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa- Johnson, and Shiyu Chang. ContentVec: An improved self-supervised speech representation by disentangling speakers. In International Conference Machine Learning (ICML), volume 162. pages 18003-18017, 2022.
[61] Youcai Qin, Qinghua Ren, Qirong Mao, and Jingjing Chen. Multi-branch feature aggregation based on multiple weighting for speaker verification. Computer Speech & Language, 77:101426, 2023.
[62] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412:6:51 2014.
[63] Leda Sari, Mark Hasegawa-Johnsonn and Samuel Thomas. Auxiliary networks for joint speaker adaptation and speaker change detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:324-333, 2021.
[64] Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: unified embedding for face recognition and clustering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[65] Anna Silnova, Niko Brummer, Johan Rohdin, Themos Stafylakis, and Lukas Burget. Probabilis- tic embeddings for speaker diarization. In Proc. Odyssey, pages 24-31, 2020.
14

---

[66] David Snyder. Daniel Romria, Daniel Povey, and Sanjeev Khudanpur. Deep neural network embeddings for rext-independent speaker verification. In Proc. Interspeech, volume 2017, pages 999-1003, 2017.
[67] David Snyder, Daniel Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-vectors: Robust DNN embeddings for speaker recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329-5333, 2018.
[68] David Snyder, Pegah Ghahremani, Daniel Povey, Daniel Sanjeev Khudanpur. Deep neural network- speaker embeddings for end-to-end speaker verification. In IEEE Spoken Lunguage Technology Workshop (SI.7), pages 163-170, 2016.
Yishay Carmiel, and
[69] Brij Mohan Lal Srivastava, Aurélien Bellet, Mare Tommasi, and Emmanuel Vincent. Privacy- preserving adversarial representation learning in ASR: Reality or illusion? In Proc. Interspeech, pages 3700-3700 2019.
[70] Guangzhi Sun, D Liu, Chao Zhang, and Philip C Woodland. Content-aware speaker embeddings for speaker diarisation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
2021.
[71] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end ASR: from supervised to semi- supervieee learning with modern architectures. In ICML 2020 Workshop on Self-xupervision in Audio and Speech, 2020.
[72] Ruijie Tao, Kong Aik Lee, Rohan Kumar Das, Ville Hautamäki, and Haizhou Li. Self-supervised speaker recognition with loss-gated learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6142-6144 2022.
[73] Ruijie Tao, Kong Aik Lee, Rohan Kumar Das, Ville Hautamäki, and Haizhou Li. Self-supervised training of speaker encoder with multi-modal diverse positive pairs. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1706-1719, 2023.
[74] Jenthe Thienpondt, Brecht Desplanques, and Kris Demuynck. Integrating frequency transla- tional invariance in tdnns and frequency positional information in 2d resnets to enhance speaker verification. In Proc. Interspeech, pages 2302 2006, 2021.
[75] Katrin Tomanek, Vicky Zayats, Dirk Padfield, Kara Vaillancourt, and Fadi Biadsy. Residual adapters for parameter-efficient ASR adaptation atypical and accented speech. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6751-6766 2021.
[76] Frederick Tung and Greg Mori. Similarity-preservinn knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1374,, 2019.
[77] Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: A research and production oriented speaker embedding learning toolkit. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5, 2023.
[78] Qionggiong Wang, Kong Aik Lee, and Tianchi Liu. Scoring of large-margin embeddings for speaker verification: Cosine or PLDA? In Proc. Interspeech, pages 600-604, 2022.
[79] Qiongqiong Wang, Kong Aik Lee, and Tianchi Liu. Incorporating uncertainty from speaker embedding estimation to speaker verification. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[80] Shuai Wang, Johan Rohdin, Lukášs Burget, Oldich Pichot, Yanmin Qian, Kai Yu, and Jan Černocky. On the usage of phonetic information for text-independent speaker embedding extraction. In Proc. Interspeech, pages 1148-1152, 2019.
[81] Jennifer A. Williams. Learning disentangled speech representations. PhD thesis, The University of Edinburgh, 2022.
15

---

[82] Yanfeng Wu, Chenkai Guo, Junan Zhao, Xiao Jin, and Jing Xu. RSKNet-MTSP: Effective and portable deep architecture for speaker verification. Neurocompuring, 511:259-2722 2022.
[83] Weidi Xie, Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Utterance-level aggrega- tion for speaker recognition in the wild. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5791-5795, 2019.
[84] Chang Zeng, Xin Wang, Erica Cooper, Xiaoxiao Miao, and Junichi Yamagishi. Attention back-end for automatic speaker verification with multiple enrollment utterances. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6717- 6721, 2022.
[85] Haoran Zhang, Yuexian Zou, and Helin Wang. Contrastive self-supervied learning for text- independent speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6713 111--717,22 2021. 2021.
[86] Jing-Xuan Zhang, Zhen- Ling, and Li-Rong Dai. Non-parallel conversion with disentangled linguistic and speaker representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:540-552, 2019.
voice
[87] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with trans- former encoders and rnn-t loss. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829 -833,, 2020.
[88] Siqi Zheng, Yun Lei, and Hongbin Suo. Phonetically-awarc coupled network for short duration lext-independent speaker verification. In Proc. Interspeech, pages 926-930, 2020.
[89] Tianyan Zhou, Yong Zhao, Jinyu Li, Yifan Gong, and Jian Wu. CNN with phonetic attention for text-independent speaker verification. In Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 718-725, 2019.
[90] Tianyan Zhou, Yong Zhao, and Jian Wu. ResNeXt and Res2Net structures for speaker verifica- tion. In IEEE Spoken Language Technology Workshop (SLT), pages 301-307, 2021.
[91] Yi Zhou, Xiaohai Tian, and Haizhou Li. Language agnostic speaker embedding for cross- cross-lingual personalized speech generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3427-3439, 2021.
[92] Yi Zhou, Xiaohai Tian, Haihua Xu, Rohan Kumar Das, and Haizhou Li. Cross-lingual voice conversion with bilingual phonetic posteriorgram and average modeling. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6790-6794, 2019.
16


Reviews:
[{"review_id": "BPV4oE7JS0", "reviewer": "Reviewer_ffvX", "summary": "The authors propose RecXi, a Gaussian inference based disentanglement method for speech that itself is based on Xi-Vector. Overall, the methods makes sense with sufficient motivation and extensive experiments. The results are promising, and have further mileage in ASR, voice conversion and TTS.\n\nThe paper's main contribution are:\n1. Improve Xi-Vector to capture temporal information occurring naturally in speech.\n2. A frame-wise content aware transition model to model dynamic aspects of spoken signal.\n3. Gaussian inference of 3 layers to disentangle content from speaker.\n4. A self-supervision trick to avoid usage of textual supervision.\n\nThe authors use a number of strong baselines spanning CNN and TDNN architectures. Across 3 datasets, the RecXi shows strong performance, outperforming the baselines on EER and minDCF.\n\nThe authors also do an ablation study to understand the contribution of the proposed methods. While helpful it is not clear what does the system RecXi($\\tilde{\\phi}$, $\\tilde{\\phi}_l$) entail (details below in Questions section).\n\nRebuttal:\nI have read the authors' rebuttal, and they have clarified the questions I had. My question was mainly due to the presentation of the equation, and the authors have taken the feedback and agreed to update the notation they have used in equations.", "questions": "The concept of $\\tilde{\\phi}\\_l$ is introduced in section 3.3; it is used for computing $l\\_{ssl}$. So, what exactly does RecXi($\\tilde{\\phi}$, $\\tilde{\\phi}\\_l$) entail when $l\\_{ssl}$ is not used?", "limitations": "As noted by the authors, the self-supervised loss makes intuitive sense but needs more investigation in terms of the loss function itself as well as some probing about why it works. Further, the number of transition models in layer 2 of Gaussian inference is a hyper-parameter whose interpretation is not clear is heavily dependent on experimentation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Well-motivated methods to the central research question of disentanglement of content and speaker.\n2. Sound modeling approach with extensive experiments.\n3. Good ablation study of the various components of the modeling technique.\n4. Has more mileage for work in ASR, TTS and Voice conversion.", "weaknesses": "1. The authors claim that disentangling speaker and content information is sufficient for getting accurate speaker representation. However, the effect of prosody is not clearly examined in this work.\n\n2. Reproducibility since it's not clear if the code/model are shared or will be shared in future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for reviewing our work and sharing the valuable suggestions and generous comments! Following is our rebuttal to address the concerns.\n\n**Weakness 1**: Thanks for pointing this out. It is true that we do not consider explicitly prosody features  in our approach. As observed in most SOTA speaker embedding representations, such as the x-vector and xi-vector, the inclusion of prosody remains ambiguous. The empirical experimental results in this paper demonstrate that focusing on the static component of speech, which represents the speaker's characteristics determined by the vocal tract's shape, proves adequate for obtaining accurate speaker embeddings. Nonetheless, an important avenue for future investigation lies in the decomposing prosody from the dynamic components. We express our gratitude for this valuable suggestion.\n\n**Weakness 2**: Thank you for pointing this out. We fully acknowledge the significance of sharing code and pre-trained models to benefit the community and attract more attention. Due to the internal approval procedure in our organizations, it may take a longer time for the code to be available. Alternatively, while we work towards the official release, we will offer help and guidance for any third-party interested in reproducing, particularly for academic purposes.\n\n\n**Question**: We appreciate your question. In our work, $\\ella$ denotes the loss function, which  includes components such as $\\ella_{cls}$ and $\\ella_{ssp}$. The $\\mathcal{l}$ in $\\tilde{\\phi_{l}}$ indicates that it is obtained by a linear operation, differing from $\\tilde{\\phi}$ in $layer$ 3 which is disentangled by Gaussian inference, as mentioned in Page 6, Section 3.3, line 245. Based on your insightful question, we observe that $\\ella$ and $l$ are similar. To avoid any potential misunderstandings, we will replace all the $\\tilde{\\phi_{l}}$ with $\\tilde{\\phi_{lin}}$, while referring to the loss function as $\\mathcal{L}$. \n\nSincerely thanks for your valuable suggestions which inspire us and help us to revise this work. We hope our rebuttal adequately addresses your concerns. If anything is still unclear, please feel free to inquire during the discussion phase. We eagerly anticipate further communication."}, {"review_id": "PCb9CkNqWC", "reviewer": "Reviewer_6npJ", "summary": "This paper proposes an improvement over Xi-vector speaker recognition algorithm. They propose an iterative 3 layer Gaussian inference mechanism to disentangle the content information from speaker information. The first layer gets precursor speaker representation similar to Xi-vector framework. The second layer uses the encoder embeddings and embeddings of first layer to disentangle the content information. The third layer finally gets the disentangled speaker representation by subtracting the second layer embeddings from encoder embeddings.\nThey also take inspiration from knowledge distillation framework and use a similarity preserving loss that preserves the similarity between third layer embeddings and the difference between first and second layer embedding. This seems to improve performance in their ablation studies. They compare their model with various baselines that do not disentangle non-speaker info from speaker and that do not use the similarity preserving loss show an improvement in performance.", "questions": "1. Authors do one round of content disentanglement in 2nd layer before getting the final speaker representation in 3rd layer. Theoretically, they can keep disentangling content and speaker info for more than 1 iteration. Did authors do any experiment if further rounds of content/speaker disentanglement leads to any performance improvement?\n2. How much is the increase in training time when RecXi is used compared with Xi due to its iterative nature?\n3. Authors try using just the \"phi\" and \"phi\" concatenated with \"phi_l\" embedding for speaker recognition in their ablation studies Table 3. Did they try using just the \"phi_l\" vector defined in eqn 24 for speaker prediction to quantify what information it carries?\n4. What about adding an extra loss term that uses the precursor embedding to predict speaker info similar to Xi-vector algorithm. One can also concatenate it with the third layer's embedding to get final prediction. Did the authors try any such experiments to see if it can provide additional benefit?\n5. Computation of w_t,N in equation 11 is not clear to me. Did the author convert rho_t to a N dimensional vector using the non-linear operation f and take softmax along the N dimensions of the resulting vector to get the vector w_t?", "limitations": "Authors discuss limitation and broader impacts of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "A key contribution of the paper is improving Xi-vector framework to disentangle content info from speaker info. The iterative mechanism to achieve that is well motivated. The proposed similarity preserving loss designed further leads to performance improvement. Another minor contribution of the paper is changing the ResNet34 backbone to a tResNet34 backbone by changing the stride strategies based on the improvement seen in preliminary experiments. That could be a reason why the results reported here with Xi-vector with tResNet34 backbone is better than what is reported in original Xi-vector paper for SITW-eval set leading to a stronger baseline.\n", "weaknesses": "1. The paper did not compare their disentanglement method with any other disentanglement method pre-exisitng in literature. As mentioned in their Related Work section most other disentanglement methods use ASR labels and they might perform better than this method which does not use ASR labels. But since disentanglement is a major contribution of the paper it would have been interesting to compare them in terms of accuracy, training time and model parameters used. Also an approach based on [1, 2] could also have been explored to learn nuisance free disentangled representations for speaker recognition which does not use ASR labels.\n2. Another major contribution is the self-supervised similarity preserving loss. However, authors do not compare with other methods pre-existing in literature that use contrastive learning as mentioned in their related work section.\n3. Utility of module \"G\" is not shown in the paper (as defined in \"Frame-wise Content Aware Transition Model\"). No ablation studies are provided as to if it is required at all. No experiments were done to show the benefit of using multiple G matrices over just one matrix. Without this module the Rec-Xi layer is same as a Xi layer and hence it is important to show its utility and what improvement we get by modifying the baseline Xi layer. \n\n[1] Liu et al. \"Towards Learning Nuisance-Free Representations of Speech\", ICASSP 2018\n[2] Mun et al. \"Disentangled Speaker Representation Learning via Mutual Information Minimization\"\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer: \n\n\nThanks for reviewing our work and providing valuable and insightful suggestions. **More experiments are reported in the PDF file in the ‘Global’ rebuttal, as per NeurIPS guideline.** Please let us know if you have any difficulty accessing the file. **Due to the word limit of the rebuttal (6000 characters), we have simplified some of the content below.** We eagerly look forward to further discussions with the reviewer. And here is our rebuttal to address your concerns.\n\n**Weakness 1 & 2**: \n\n**1). Pretrained ASR models**: Thanks for pointing this out. As mentioned in Section I, line 40, \"ASR models are typically one or two orders of magnitude larger than the speaker recognition model\".  This issue has led to limited adoption of pre-trained ASR models in SOTA methods. A related work [1] published in 2019 utilizes this method. A comparison is presented in Table 1 in the PDF. Notably, our proposed method (#9) outperforms the approach (#New1) from [1].\n\nIn addition, we find a recent  work (which became available after our submission)  that utilizes a pre-trained ASR model to initialize the speaker verification model [2]. Due to word limitations, a detailed analysis is presented in the global rebuttal. In summary, our approach achieves comparable performance to the model in [2] while also being **55.5% smaller than its smallest model.**\n\n**2). Text labels**. As mentioned in our paper, acquiring text labels for large-scale speaker verification datasets, such as the VoxCeleb dataset with over 1 million samples, is very expensive. Therefore, to the best of our knowledge, there is no suitable system to compare with.\n\n**3). Contrastive learning.** We sincerely appreciate this valuable comment provided. \nAs presented in Table 1, we compare the proposed RecXi and the recently reported SOTA contrastive learning system [3], both evaluated on the same dataset. Notably, even though the models in [3] incorporate extra visual information beyond speech, our proposed RecXi consistently demonstrates substantial superiority over the system detailed in [3] across all three test sets.\n\n**4). Nuisance free disentangled representations**: We appreciate this good point. The reconstruction is required in [4], while as we mentioned in Section II, line 117, the self-supervision is designed to be simple yet effective and avoid extra signal re-contractors. In [5], device labels are used to train the device classifier, whereas in our work, text labels are not available for training. We briefly discuss and cite [5] on Page 2, Section II, line 93, ‘removal of irrelevant information, like **devices**... with corresponding labels [**54**]’.\nThese two papers are insightful and serve as valuable references for our related work discussion. The following sentences will be added in Section II, line 93: \"[5] utilizes MI estimators to minimize the mutual information between speaker and device embeddings and reduce the interdependence between embeddings and labels. In [4], nuisance variables, such as gender and accent, are removed from speaker embeddings through the learning of two separate orthogonal representations.\"\n\n[1] Zhou et al. \"CNN with Phonetic Attention for Text-Independent Speaker Verification\", IEEE ASRU 2019.\n\n[2] Cai et al. \"Pretraining Conformer with ASR for Speaker Verification\", IEEE ICASSP 2023.\n\n[3] Tao et al. \"Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs\", IEEE TASLP 2023.\n\n[4] Liu et al. \"Towards Learning Nuisance-Free Representations of Speech\", IEEE ICASSP 2018.\n\n[5] Mun et al. \"Disentangled Speaker Representation Learning via Mutual Information Minimization\", IEEE APSIPA ASC 2022.\n\n**Weakness 3**:  Thanks for this great suggestion. We add additional experiments by replacing the $\\mathbf{G_t}$ with single learnable matrix or identity matrix following that in xi-vector, while maintaining the three-layer design with the self-supervision loss. The results are presented in Table 3 in the PDF. A detailed analysis is presented in the global rebuttal. In summary, the results validate the effectiveness of $\\mathbf{G_t}$.\n\n**Question 1**: Yes. We agree with the reviewer. While the idea of stacking layer 2 and layer 3 seemed promising, we believe that the benefits of additional layer stacking diminish rapidly. The current three-layer configuration has yielded concrete results, effectively supporting our assumptions and claims.\nWe reserve this further investigation as a future work.\n\n**Question 2**: Thanks for raising this question. The training process is 25% slower because of the recursive. However, the increase in Floating Point Operations (FLOPs) is only 13%. It is worth noting that further optimization, similar to the LSTM, could be implemented through coding improvements. Additionally, for larger models, the encoder component consumes a significant portion of the computation time, and the impact of RecXi's computational overhead is expected to diminish in such cases. \n\n**Question 3**: Thanks for this suggestion. We add the ablation study in Table 2 in the PDF. The results clearly show that $\\tilde{\\phi}_l$ carries speaker information and demonstrates strong discriminative capabilities. A detailed analysis is presented in the global rebuttal. \n\n**Question 4**: Our preliminary results indicate that there is no obvious improvement obtained by adding the loss function for the precursor embedding. Our explanation for this outcome is that the additional loss could complicate gradient propagation and lead to confusion for the model's learning objectives in layer 1.\n\n**Question 5**: Yes, the reviewer’s understanding is correct. We will add Equation 1 in PDF to make it clear to readers.\n\nWe express our sincere gratitude to the reviewer for providing numerous valuable suggestions. Inspired by these suggestions, we conducted additional experiments, which yielded more evident results that strongly support our claims and demonstrate the success of disentanglement. \n"}, {"review_id": "gCJdQicGIb", "reviewer": "Reviewer_48tf", "summary": "In this paper, authors propose a disentanglement framework for speaker recognition that extract speaker representations from the mixture of speaker traits and content. The proposed framework utilizes three Gaussian inference layers to decompose the static and dynamic speech components. Additionally, a self-supervision method is also proposed to help disentangle content without relying on text labels. Based on experimental results, the authors claim that the proposed method achieves remarkable performance on the VoxCeleb and SITW datasets, which demonstrates the effectiveness of the framework.", "questions": "I was wondering if authors evaluate the proposed method on some more challenging datasets, such as VOiCES?", "limitations": "The authors analyze the limitations in Section 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors conduct solid experiments and the proposed method shows good performance on VoxCeleb and SITW. \n\n2. This paper shows detailed experimental results and a comprehensive comparison with several solid baselines. \n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The proposed RecXi model with self-supervision speaker preserving loss generative very limited improvement compared to the Xi baseline system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our work and providing us with your valuable suggestions and insightful comments. We have carefully considered your feedback, and we would like to address your concerns as follows:\n\n**Weakness:** Thanks for pointing this out. We use the best baseline that outperforms those reported in the literature [38, 45, 36, 43, 87, 16, 82, 96, 46, 63, 70, 84] on the same dataset, and we even further enhanced the baseline with the proposed tResNet encoder. The comparison of the baselines and SOTA systems is reported on Page 7, Table 1. In addition, the xi-vector baseline we compared to is also much stronger than that in the original paper [38], on the same dataset (SITW dataset). Since the EER is already very small (below 2%), making further improvements of more than 10% is quite challenging.  As shown in Table 2 and analysis in line 315 in Section 5, by comparing system #10 and #11, \"the proposed RecXi achieves average EER/minDCF reductions of 12.15%/10.66% over the Xi baseline.\" \n\nWe apologize if this aspect was not clearly emphasized in the paper. We will add the following sentences to line 303 for clarification: \"We observed that the baseline system, comprising a tResNet and Xi combination, is powerful and achieves state-of-the-art (SOTA) performance. This also renders further performance improvements more difficult. However, the proposed disentangling method, RecXi, exhibits even more outstanding performance.\"\n\n\n**Question:** Thanks for this good suggestion. The reason that we didn’t use VOiCES dataset is that this dataset was specially designed for single channel far-field speech under noisy conditions, which is not the focus of this paper.\n\nAnd since we already include the easy set (Vox1-O), the hard cases (Vox1-H), the large set (Vox1-E) and a cross domain test set (SITW), the experimental results would be sufficient to support our claims.  But your suggestion is very valuable, we will take note of this and add VOiCES and CNCeleb datasets in future work.\n\nWe sincerely appreciate the reviewer for offering a multitude of valuable suggestions. We eagerly anticipate engaging in further discussions during the next phase.\n"}, {"review_id": "NpzhKAHOoO", "reviewer": "Reviewer_krxU", "summary": "This paper describes a novel speaker representation based on a three layer Gaussian inference network for speaker recognition.  The model approach uses a static representation for layers 1 and 3, but a time-variant representation for layer 2.  The authors position this network structure as disentangling speaker and content information, as speaker information is constant across a speech signal, while content information is time-varying.", "questions": "Would a 5 layer network provide improved quality by including a second layer of time-varying inference as layer 4 and a third static layer as layer 5 to further refine the static/time variant representations? (if so, what about 7? etc.\n\nIs T, the maximum length fixed a priori for all samples? If so, are RecXi representations robust to different utterance lengths with varying T?\n\nHow is N tuned in the Frame-wise Content-aware Transition model G_t?  \n\nPage 5 Section 3.2 re Layer 3: how expensive are the matrix multiplications such that simplification is necessary here? How much improvement to computation (FLOPS? latency?) are these optimizations providing?\n\nTable 1: Are 4 significant digits necessary to understand the results? many include only 2.  It might be clearer to be consistent across all results.\n\nAny concern about other channel characteristics, noise, or reverberant environment characteristics being captured by the static representation?\n", "limitations": "Limitations are included in the last Appendix in the supplemental material.   Per https://neurips.cc/public/guides/PaperChecklist \"You are encouraged to create a separate \"Limitations\" section in your paper.\"  I'd encourage the authors to include this section in the body of the paper, but the limitations are raised nonetheless", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation is well justified.  The description of the approach is well written and relatively easy to follow.\n\nGood details are included in Table 1 about which models are reimplemented and which are reported from published results.\n\nThe paper includes a nice ablation to understand the contribution of l_ssp and the availability of both \\phi and \\phi_t\n", "weaknesses": "The best performing model includes \\phi and \\phi_t both the static and dynamic components for speaker representation.  This has two limitations – first, it makes for a much larger representation vector, and the length of the vector is dependent on T (the max length) being fixed across samples, second, it somewhat undermines the central claim of the proposed technique that the static \\phi should contain speaker information while the dynamic \\phi_t should isolate content contributions.\n\nThe introduction claims that using pretrained ASR to isolate content is suboptimal because it requires too many parameters.  Therefore the aim of this work is to isolate content from the representation with fewer parameters than a full ASR model and without text labels available during training.  This is a fair set of constraints, but it would be useful to have a comparison to a model that uses an explicit content-aware decomposition.  That is, how well does a speaker recognition model like those enumerated in the Introduction that uses an ASR model, even at the cost of additional computation, perform at this task?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\nWe sincerely appreciate your effort in reviewing our work and providing valuable and insightful suggestions. In response to your feedback, **we have conducted additional experiments, and the results are included in the PDF file attached to the 'Global' rebuttal, following this year's NeurIPS guideline.** If you encounter any difficulty accessing the PDF file or have any questions during the discussion, please do not hesitate to inform us. We have carefully considered your concerns, and the following is our rebuttal to address them.\n\n**Weakness 1**:\nThe motivation of using $\\tilde{\\phi_{l}}$ is to provide supervision when the text label is absent. According to the ablation study presented in Table 3 of the paper, we observe that similar results can be achieved even without $\\tilde{\\phi_{l}}$ when $l_{ssp}$ is applied, as evident from the comparison between #11 and #12, and #5 and #15.\nFurthermore, we would like to clarify that we use only the last frame of the recursive structure. We will emphasize and clarify this point further in the main body of the paper. Thank you for bringing this to our attention, and we apologize for any misunderstanding. \n\n**Weakness 2**:  Thank you for understanding that we didn't compare the ASR model-based systems because of their significantly larger model size compared to the speaker recognition model. Meanwhile, we also agree with the reviewer that including a comparison with the system using the ASR model is more convincing. Therefore, we add the comparison in Table 1 of the uploaded PDF in the 'Global' rebuttal.\n\nThis issue of large model size has led to limited adoption of pre-trained ASR models in SOTA methods. We find a work published in 2019 [1] that utilizes pre-trained ASR models, and a comparison is presented in Table 1. Notably, our proposed method (#9) outperforms the approach (#New1) from [1].\n\nIn addition, we find recent work (which became available after our submission)  that utilizes a pre-trained ASR model to initialize the speaker verification model [2]. In Table 1, we observe that RecXi outperforms the model without pre-training in [2] (system #9 vs #new2). Additionally, the proposed method achieves similar performance with the model utilizing ASR pre-training in [2] (system #9 vs #new3). Our proposed method offers a significant advantage: it achieves competitive results without requiring the pre-training of an ASR model. Additionally, our method is approximately **55.5% smaller** than the **smallest** model in [2], highlighting its efficiency and effectiveness.\n\n**Question 1**: Thanks for this great comment. We also came up with the same idea when we were designing the RecXi. While the idea of stacking layer 2 and layer 3 seemed promising, we believe that the benefits of additional layer stacking diminish rapidly. The current three-layer configuration has yielded concrete results, effectively supporting our assumptions and claims. We reserve this further investigation as a future work.\n\n\n**Question 2**: The training samples have a fixed length of 3 seconds, as explained in Appendix C.2 Training Strategy (line 709). During testing, the whole sample length is utilized, resulting in varying input lengths (T). RecXi demonstrates robustness to these varying input lengths, as shown in the test results. This training and testing configuration adheres to the standard practice in the majority of ASV work, ensuring a fair comparison.\n\n**Question 3**: Thank you for bringing up this point. In Table 3 of the PDF and in the analysis provided in the global rebuttal, additional investigations related to N have been presented. However, it is important to underscore that **due to the constraints of the rebuttal time, these new experiments were conducted to verify the effectiveness of $\\mathbf{G_t}$ rather than fully fine-tuning the hyperparameter N**. As mentioned in Section 6 Conclusion (line 362), N is a hyperparameter that can be further explored as it is  related to the acoustic features and dynamic components we wish to disentangle. We will continue our investigation to find a better way of selecting this hyperparameter based on the textual representation information. However, its value does not impact our core concept, but only affects performance. Further research on this aspect may reveal more insights into dynamic components, making it a valuable subject for future investigation.\n\n**Question 4**: Thank you for your question. Same as the xi-vector, we assume the matrix is diagonal to simplify the computation from a time complexity of O(n^3) to O(n). Meanwhile, the log domain is applied to provide a better numerical stability. The diagonal assumption and log domain utilization are mentioned in Section 3.1.\n\n**Question 5**: Thank you for providing this great suggestion. We will reduce the results in all tables (including the tables in the PDF file) to 3 significant digits for improved clarity and consistency. \n\n**Question 6**: Thanks a lot for this detailed and insightful question.\nThe static representation is specifically designed to capture static signal components such as speaker characteristics, channels, and background noise.  However, channels and noise can be effectively handled through data augmentation by the encoder, making the static representation effective for its intended purpose.\n\n**Limitations**: Thank you for bringing this to our attention. In the final draft, we will include a separate section to address the limitations in more detail if the page limit allows. \n\nWe extend our heartfelt appreciation for your valuable suggestions, which have been a great source of inspiration and have played a pivotal role in the revision of this work.\n"}, {"review_id": "cdE1Bx4Tbq", "reviewer": "Reviewer_P6Zh", "summary": "In this paper, the authors proposed RecXi, a framework that disentangles the voice and content for speaker recognition. Also, the authors used self-supervised learning methods to improve performance. The authors focused on the fact that speaker information is static and the content information is dynamic, so the main idea is the method to decompose static information and dynamic information. To achieve their goal, they designed a three-layer Gaussian inference framework. They argued that more accurate speaker representation is decomposed from layer 1 to layer 3 because they extract the content representation in layer 2.\n\nIn the experiment section, the authors compared the proposed model with previous works and show the proposed model outperforms several different datasets.", "questions": "- It is a novel method but it is quite complex to understand intuitionally. Could you provide more visualized results or example audio for the supplementary material?", "limitations": "- In this architecture, it is very difficult to interpret how the model decomposes different features.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Disentangling different features clearly with self-supervised learning is a very challenging task in many domains. The authors sapientially solve this problem by aiming at the temporal characteristics of the speaker and content.\n\n- The proposed method outperforms the existing works.", "weaknesses": "- The authors argued that more accurate speaker representation is decomposed from layer 1 to layer 3. However, it is not verified with the ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\nThank you for taking the time to review our work and providing us with your valuable suggestions and generous comments! **More experiments are presented in the PDF file in the ‘Global’ rebuttal following this year’s NeurIPS guideline.** Please kindly let us know during discussion if you can’t find or access that PDF file. And following is our rebuttal to address your concerns.\n\n\n**Weaknesses**: Thanks for this insightful comment. We have included the ablation study in Table 2 in the PDF. The results clearly show that the speaker representation decomposed in layer 3 provides better discriminative ability than that in layer 1.\n\nAdditionally, inspired by this, we further tested the content representation from layer 2. The results further support our claims, confirming its role in representing content while effectively removing speaker-related information.  The EER being close to the maximum value, 50%, indicates that layer 2 does not contain any speaker-related information and does not exhibit any speaker discriminative ability. \n\nWe express our deep gratitude to the reviewer for inspiring us to conduct this ablation study, which unmistakably demonstrates the success of our disentanglement approach.\n\n\n**Questions & Limitations**:  Thanks for this great comment. This work focuses on disentangling speaker voice and content information from speech, which proves beneficial for the speaker verification task. As mentioned in the Conclusion and Broader Impact, the disentangled speaker and content representations hold potential for reconstructing the speech signal and enabling voice conversion. And we intend to explore these possibilities in future research.\n\nFollowing your kind suggestion, we provide the visualization using t-SNE in Figure 1 in the PDF. The speakers are taken from the test set of VoxCeleb. The visualization depicts speakers from the test set of VoxCeleb, where the first 200 speakers (indexed from id10001 to 10200) are included, each with 20 utterances. From the figure, it is evident that in layer 3, the disentangled speaker representations ($\\tilde{\\phi}$) are highly discriminative for speakers. Moreover, benefiting from the self-supervision loss, the speaker representation $\\tilde{\\phi_{l}}$ acquired through the linear operation (Eq. 24) also exhibits notable speaker discriminative ability, comparable to that derived from layer 3 ($\\tilde{\\phi}$). However, for layer 2 ($\\rho$), as its main objective is to disentangle content information, it lacks the discriminative ability observed in layer 3 ($\\tilde{\\phi}$). These observations align with those in Table 2 of the attached PDF and strongly support our claims. We appreciate this great suggestion.\n\nWe sincerely express our gratitude for your valuable suggestions, which have been a great source of inspiration and have significantly contributed to the revision of this work.\n"}]

Author rebuttal:
Dear all reviewers,

We greatly appreciate the insightful suggestions and valuable comments from each reviewer. These have been immensely helpful and enlightening for refining this paper.

We have included a figure and several experimental results to aid our rebuttal. **Following the NeurIPS guidelines for this year, we have placed these tables and figures in the attached PDF file of this global rebuttal.** Please let us know if you are unable to access this PDF. The following is the analysis of the content in the PDF.

**Figure 1: t-SNE visualization of speaker discriminative ability for ablation study.** (related to reviewer P6Zh, Question 1) 

**Analysis**: The speakers used in Figure 1 are taken from the test set of VoxCeleb.The visualization depicts speakers from the test set of VoxCeleb, where the first 200 speakers (indexed from id10001 to 10200) are included, each with 20 utterances. From the figure, it is evident that in layer 3, the disentangled speaker representations ($\tilde{\phi}$) are highly discriminative for speakers. Moreover, benefiting from the self-supervision loss, the speaker representation $\tilde{\phi_{l}}$ acquired through the linear operation (Eq. 24) also exhibits notable speaker discriminative ability, comparable to that derived from layer 3 ($\tilde{\phi}$). However, for layer 2 ($\rho$), as its main objective is to disentangle content information, it lacks the discriminative ability observed in layer 3 ($\tilde{\phi}$). These observations align with those in Table 2 of the attached PDF and strongly support our claims.


**Table 1: A comparison between proposed RecXi with SOTA methods using pre-trained ASR models or contrastive learning.**  (related to reviewer 6npJ, Weakness 1 & 2; reviewer krxU, Weakness 2) 

**Analysis**: Thank reviewers for kind understanding that we didn't compare the ASR model-based systems because of their significantly larger model size compared to the speaker recognition model. 
This issue has led to limited adoption of pre-trained ASR models in SOTA methods. A related work [1] published in 2019 utilizes this method. A comparison is presented in Table 1. Notably, our proposed method (#9) outperforms the approach (#New1) from [1].

In addition, we find a recent work (which became available after our submission)  that utilizes a pre-trained ASR model to initialize the speaker verification model [2]. In Table 1, we observe that RecXi outperforms the model without pre-training in [2] (system #9 vs #New2). Additionally, the proposed method achieves similar performance with the model utilizing ASR pre-training in [2] (system #9 vs #New3). Our proposed method offers a significant advantage: it achieves competitive performance without requiring pre-training of an ASR model. Additionally, our method is approximately 55.5% smaller than the smallest model in [2], highlighting the efficiency and effectiveness of the proposed RecXi.

We also compare the proposed RecXi and the recently reported SOTA contrastive learning system, both evaluated on the same dataset. Notably, even though the models in [3] incorporate extra visual information beyond speech, our proposed RecXi consistently demonstrates substantial superiority over the system detailed in [3] across all three test sets.

[1] Zhou et al. "CNN with Phonetic Attention for Text-Independent Speaker Verification", IEEE ASRU 2019.

[2] Cai et al. "Pretraining Conformer with ASR for Speaker Verification", IEEE ICASSP 2023.

[3] Tao et al. "Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs", IEEE TASLP 2023.

**Table 2: Ablation study using different posteriors from the three layers of RecXi.**  (related to reviewer 6npJ, Question 3; reviewer P6Zh, Weakness 1) 

**Analysis**: The results clearly show that both $\tilde{\phi}$ and $\tilde{\phi_{l}}$ carries speaker information and demonstrates strong discriminative capabilities. Additionally, reporting on layer 2 further supports our claims, confirming its role in representing content while effectively removing speaker-related information. The EER being close to the maximum value, 50%, indicates that layer 2 does not contain any speaker-related information and does not exhibit any speaker discriminative ability. The precursor speaker representation ${\phi}$ also exhibits fine speaker discriminative ability, but it is slightly inferior to $\tilde{\phi}$. We express our deep gratitude to the reviewers for inspiring us to conduct this ablation study, which unmistakably demonstrates the success of our disentanglement approach.

**Table 3: Verification for the effectiveness of $\mathbf{G}_t$** (related to reviewer 6npJ, Question 5) 

**Analysis**: To verify the effectiveness of $\mathbf{G}_t$, we conduct experiments by replacing the $\mathbf{G_t}$ with single learnable matrix (#New10) or the identity matrix (#New11) following that used in the xi-vector, while maintaining the three-layer design with the self-supervision loss. Based on the results, it is evident that using an identity matrix or a single learnable matrix leads to poor performance. This clearly demonstrates that if the second layer lacks the capacity to model dynamic patterns, the model becomes confused. This, in turn, has a significant impact on layer 3's ability to obtain accurate speaker embeddings, as it relies on removing the content information provided by layer 2. Above observations further validate the effectiveness of the proposed $\mathbf{G_t}$. Furthermore, when N is set to 6 (#New9), the model achieves performance that is better than xi-vector baseline (#4 on Page 7) and is close to N=16 (#5) but slightly worse. This demonstrates the necessity of our proposed $\mathbf{G_t}$ in facilitating the capability of layer 2 to model dynamic components.

**Equation 1**  (related to reviewer 6npJ, Weakness 3) 

This equation is utilized to provide a clearer demonstration of the derivation of $\mathbf{G}_t$, and it will be included in the paper.

Meta-review: The authors introduce an innovative three-layer Gaussian inference mechanism to disentangle content information from speaker characteristics. The first layer creates a precursor speaker representation akin to the Xi-vector framework. The second layer employs the encoder embeddings along with the first layer's embeddings to disentangle content information. Lastly, the third layer yields a fully disentangled speaker representation by subtracting the second layer's embeddings from the encoder embeddings.

A primary critique raised by reviewers was the paper's need for comparing with more existing disentanglement methodologies. Specifically, there was a noticeable absence of comparisons to methods employing ASR and contrastive learning. In their rebuttal, the authors address this concern by including comparisons with ASR-based methods published in 2019 and a recent work that became available post-submission. They also add more comparisons using contrastive learning methods. Impressively, the proposed approach demonstrates competitive performance without necessitating the pre-training of an ASR model. I believe this effectively addresses the concerns initially raised by the reviewers.
Decision: Accept (poster)