SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we regularize deep-learning models with the ICC to improve the embeddings' repeatability in three different speech tasks: (1) speaker embeddings for text-independent speaker verification, where the ICC regularizer ensures the embeddings are repeatable for the same person and the contrastive loss aims to separate embeddings of different speakers; (2) speaker embeddings for zero-shot voice style conversion, where using embeddings with better quality results in higher-quality conversion; (3) voice feature embeddings for dysphonic voice detection, where the ICC regularizer ensures the embeddings do not change from day to day for the healthy group and a contrastive loss forces maximum separability between dysphonic and healthy speech.
4.1 Task 1: text-independent speaker verification
Text-independent speaker verification (TI-SV) systems verify the speaker's identification using a speech signal input without any constraints on the speech content. Many previous methods use speaker embeddings for the TI-SV task [54, 24, 48, 26]. For TI-SV, the speaker embeddings should only capture the difference in the speaker identities and be invariant to all other confounding factors, including utterance content and length. Herein we include our proposed ICC regularizer when learning speaker embeddings to improve their repeatability.
Experiment setup: We select three well-known contrastive losses as baselines: (1) GE2E [54], (2) Angular Prototypical (AngleProto) [12], and (3) SupCon [25]. Then we use these contrastive losses with and without the ICC regularizer to train two encoders, VGG-M-40 and FastResNet-34 which is described in Chung et al. paper [12], and compare the performance and quality of these speaker embeddings. We use the VoxCeleb 1 & 2 development dataset for training, and VoxCeleb 1 testing dataset for TI-SV performance evaluation [37, 11]. For training VGG-M-40, each batch contains  $N = 8$  speakers and  $M = 30$  utterances per speaker, and the loss formula  $L(\mathbf{e}) = 1.0 \times L_{\text{contr}}(\mathbf{e}) + 0.06 \times R_{\text{ICC}}(\mathbf{e})$ , where the e is the embeddings of speakers' in one batch. For training FastResNet-34, each batch contains  $N = 100$  speakers and  $M = 2$  utterances per speaker, and the loss formula  $L(\mathbf{e}) = 1.0 \times L_{\text{contr}}(\mathbf{e}) + 0.25 \times R_{\text{ICC}}(\mathbf{e})$ . During training, we use the Adam optimizer, maintaining a static learning rate of 0.001 without implementing any learning rate schedule. The dropout rate is set to 0.2 for all dropout layers. As for data augmentation: (1) we use variation in input audio length by randomly fixing the audio duration within a range of 1.5 to 3.0 seconds, and (2) we add Gaussian noise with a SNR randomly selected between 15 to 60 dB. No other augmentation methods are used. The hyper-parameter is tuned on the development dataset. The EER for subjects in the development dataset are used to determine the optimized hyperparameter. We use the EER and minDCF to evaluate the performance of TI-SV, ICC (Equation 5) to evaluate the repeatability of embeddings.
Table 1: TI-SV task EER and ICC results for contrastive losses with and without ICC regularizer.
VGG-M-40	FastResNet-34
EER	minDCF	ICC	EER	minDCF	ICC
GE2E [54]	4.39%	0.2925	0.4494	2.49%	0.2133	0.7215
GE2E + ICC	3.96%	0.2778	0.5487	2.39%	0.2012	0.7366
AngleProto [12]	4.36%	0.2809	0.4399	2.28%	0.1960	0.7501
AngleProto + ICC	4.02%	0.2790	0.5455	2.17%	0.1871	0.7627
SupCon [25]	3.91%	0.2791	0.5693	2.30%	0.1956	0.7500
SupCon + ICC	3.78%	0.2597	0.6661	2.16%	0.1867	0.7615
Results: Table 1 presents the results of the TI-SV evaluation for contrastive losses, both with and without the ICC regularizer. When using VGG-M-40 and FastResNet-34 models, the GE2E achieves EER of  $4.39\%$  and  $2.49\%$  on the VoxCeleb 1 test set, respectively. Incorporating the
ICC regularizer improves the GE2E's performance, reducing EER to  $3.96\%$  for VGG-M-40 and  $2.39\%$  for FastResNet-34. This represents a approximately  $10\%$  enhancement in performance for the VGG-M-40 model. The repeatability of speaker embeddings also improves with the ICC regularizer as the ICC score is increased from 0.4494 to 0.5487 and from 0.7215 to 0.7366 for VGG-M-40 and FastResNet-34, respectively. In addition, the benefits of the ICC regularizer are also observed with two other contrastive loss methods. For the AngleProto method, introducing the ICC regularization achieves a  $7.8\%$  and  $4.8\%$  improvement over baseline models without ICC regularizer. Meanwhile, the SupCon method, records a  $3.3\%$  and  $6.08\%$  improvement over SupCon without ICC regularizer, also resulting in more repeatable embeddings.
The experimental results of the TI-SV task demonstrate that the proposed ICC regularizer can improve the repeatability of embeddings learned to be sensitive to speaker identities. The speaker embeddings with higher repeatability achieve improved performance on the TI-SV task.
4.2 Task 2: zero-shot voice style conversion
One important application of speaker embeddings is voice style conversion, i.e., modifying the speech of a source speaker to sound like that produced by another target speaker without changing the linguistic information [40, 21, 58]. In the previous section, we used the ICC regularizer together with a contrastive loss during training to obtain speaker embeddings with higher repeatability. The embeddings with high repeatability should benefit downstream applications. While in the previous section we demonstrated that they benefit the TI-SV task for several different contrastive losses, in this section, we use a zero-shot voice conversion model, AutoVC [40], and evaluate it with two different speaker embeddings generated from the previous section: (1) GE2E loss trained speaker embeddings which are less repeatable, and (2)  $\mathrm{GE2E + ICC}$  regularizer trained speaker embeddings which are more repeatable. We choose the most challenging voice style conversion task, the zero-shot voice style conversion (unseen speaker to unseen speaker), to compare the downstream performance of these two embeddings with different repeatability.
Experiment setup: We use exactly same training procedure described in Qian et al. [40] to train the two models. For evaluating the conversion quality perspectively, we conduct AB preference test to compare the generated samples: we randomly select 10 unseen source and 10 unseen target speakers from the VCTK corpus [52], to generate a total of 100 source-target speaker pairs<sup>6</sup>. For each listening test, 15 pairs are randomly selected from the 100 pairs. The order of presentation is randomized. A total of 18 listeners participated in this AB preference test. They were instructed to select the sample that better matched the target speaker without knowing what method was used to generate the samples. We also evaluate two methods objectively by using objective scores based on the word error rate (WER) and character error rate (CER). We use an opensource speaker encoder<sup>7</sup> to calculate the speaker similarity score between target speaker's audio and transformed output. And we then use Wav2Vec<sup>28</sup> to do ASR on the transformed output, and calculate the WER and CER by using jiwer module<sup>9</sup>.
Table 2: AB preference result for zero-shot voice style conversion.
GE2E Loss	GE2E Loss + ICC Regularizer
Selection Ratio	38.10%	61.90%
Table 3: Objective evaluation result for zero-shot voice style conversion.
Speaker Similarity Score	WER	CER
GE2E Loss	0.2231	0.5810	0.3817
GE2E Loss + ICC	0.2309	0.5109	0.3324
Results: The AB preference result is shown in Table 2. On average,  $61.90\%$  of samples generated by the model trained with the more repeatable speaker embeddings were preferred over those trained
<sup>6</sup>We provide several examples on our project page: https://vigor-jzhang.github.io/icc-reg-project-page/.
$^{7}$ huggingface.co/speechbrain/spkrec-ecapa-voxceleb
$^{8}$ huggingface.co/docs/transformers/model_doc/wav2vec2
9 pypi.org/project/jiwer/2.5.1/
using the original GE2E loss speaker embeddings. The objective evaluation results are shown in Table 3. The objective evaluation metrics demonstrate that speaker embeddings with higher repeatability also result in better performance across all these metrics for voice style conversion. All these results demonstrate that speaker embeddings with higher repeatability also result in better performance for voice style conversion task.
4.3 Task 3: assessment of vocal quality for dysphonic voice detection
Dysphonia is a term that refers to difficulty producing clear voicing during speech production. Automatic dysphonic voice detection by deep learning has attracted academic and clinical interest. To develop reliable clinical models, it is important to use highly repeatable voice features and embeddings [45]. Zhang et al. proposed voice feature embeddings sensitive to vocal quality and robust across different corpora [62]. However, in their work there is no constraint on the voice feature embeddings' repeatability.
We rebuild the Zhang et al. network and implement the ICC regularizer to enhance the repeatability of embeddings. We follow the procedure described in [62] for training the voice feature embeddings. To enforce repeatability using the ICC regularizer, we use the recordings of healthy subjects from the mPower corpus [5] to ensure the embeddings do not change daily for the healthy person.
Model structure: We use the same encoder and MLP classifier from the Zhang et al. paper. A two-branch structure is used for optimizing the dysphonic sensitivity and repeatability of voice feature embeddings simultaneously. All three embeddings have a dimension of 256. For more information about our model structures, we refer the reader to the Appendix C.
Training loss: We use the following formula for training:  $L = 0.5 \times R_{ICC} + 1.0 \times L_{contr} + 1.0 \times L_{class}$ , where the  $R_{ICC}$  is the ICC regularizer on repeat-constrained embeddings,  $L_{contr}$  is the contrastive loss on the voice feature embeddings, and  $L_{class}$  is the classification loss.
Training and evaluation datasets: We use the Saarbruecken Voice Database (SVD) [57] as the training and in-corpus validation dataset for dysphonic voice detection, and the mPower corpus [5] is used only for improving the repeatability of voice feature embeddings. The Massachusetts Eye and Ear Infirmary (MEEI) database and Hospital Principe de Asturias (HUPA) [1] dataset are used for cross-corpus testing datasets for dysphonic voice detection task, and the ALS [44] dataset is used for repeatability evaluation. The MEEI, HUPA, and ALS are unseen during training for all methods. A summary of these datasets are provided in Appendix D.
Training details: We perform cross-validation six times to characterize the variability in performance. We fixed the random seed to 233. The SGD optimizer is used with a learning rate of 0.001 and other default settings. We use one NVIDIA Titan Xp graphic card to train our models. We train the model for 20k steps, which takes approximately 16 hours under our configurations.
**Baseline methods:** We compare against four baselines: (1) Zhang et al. [62]; (2) P. Harar et al. [18], which is based on a recurrent convolutional neural network model; (3) L. Verde et al. [53], which used a conventional features set with different classical machine learning classifiers; (4) M. Huckvale et al. [22], which uses the ComPare feature set from the OpenSMILE toolkit with the SVM and neural networks methods. All baseline methods are rebuilt and trained on our data using the same procedures as they outlined in the original papers.
Evaluation metrics: We use balanced accuracy to evaluate the dysphonic voice classification accuracy. We use the ICC (Equation 5) to evaluate the repeatability of our trained voice feature embeddings and other baseline methods' features. For a fair comparison, we evaluate the repeatability by using ALS dataset [44], which is unseen to all methods during training.
Table 4: Dysphonic voice detection accuracy and features' repeatability of our and baseline methods.
Dysphonic Voice Classification / [mean accuracy] (95% CI)	Repeatability / ICC
SVD Train	SVD Validation	MEEI Testing	HUPA Testing	ALS
Proposed Method	0.7353 (0.004)	0.7289 (0.009)	0.8214 (0.004)	0.6894 (0.011)	0.5708
Zhang et al. (2022)	0.8003 (0.018)	0.7077 (0.011)	0.8209 (0.014)	0.6651 (0.008)	0.4368
Harar et al. (2017)	0.7742 (0.017)	0.6914 (0.009)	0.6614 (0.024)	0.4918 (0.008)	0.4743
Verde et al. (2018)	0.8910 (0.006)	0.6274 (0.009)	0.7042 (0.018)	0.5976 (0.015)	0.0182
Huckvale et al. (2021)	0.7290 (0.019)	0.6255 (0.012)	0.6978 (0.044)	0.5487 (0.014)	0.2914
Results: Implementing the ICC regularizer significantly improves the repeatability of the voice feature embeddings as shown in Table 4. Our proposed method achieves the highest ICC score of 0.5708, while the Zhang et al. model only achieves an ICC score of 0.4368, an improvement of  $30.68\%$ . Similarly, the repeatability of embeddings generated with the ICC regularizer regularizer significantly exceeds that of all baseline methods.
Our experimental results demonstrate that the voice feature embeddings with higher repeatability also achieve better classification accuracy and generalizability. Our proposed method achieves good classification accuracy on SVD in-corpus validation 0.7289 (±0.009), MEEI cross-corpus testing 0.8214 (±0.004), and HUPA cross-corpus testing 0.6894 (±0.011). For a comparison, the original HUPA publication achieved accuracy of 0.6962 (±0.047) with MFCC when trained and tested in-corpus [1]. Our proposed method's classification accuracy across the three corpora is quite good, considering the differences between the corpora.
The experimental results of voice feature embeddings for dysphonic voice detection task demonstrate that the ICC regularizer can improve the repeatability of embeddings and embeddings with higher repeatability exhibit better accuracy and generalizability in dysphonic voice detection.
This paper portes the concept of repeatability from measurement theory to representation learning. We propose to use the ICC as an evaluation metric in representation learning and use the ICC regularizer as a complementary component for contrastive loss to regularize deep-learned embeddings to be more repeatable. We use an example and intuition to explain why the ICC regularizer has better performance on minimizing intra-class variance than contrastive loss. We evaluate the ICC regularizer on three speech tasks that use learned embeddings: speaker embeddings for TI-SV and zero-shot voice style conversion, and voice feature embeddings for a clinical application. The experimental results demonstrate that the ICC regularizer can improve the repeatability of learned embeddings, and embeddings with higher repeatability exhibit better performance in downstream tasks. There several directions for future works: (1) applying the ICC regularizer to other domains, including computer vision and natural language processing; (2) extension to self-supervised methods that use contrastive-style training; (3) a more thorough theoretical analysis of the ICC and its properties.
Potential Negative Societal Impact: Methods for learning new feature representations that focus on separability between classes can amplify biases that exist in the data. This is a well-known problem and it can occur when the data used to train the representation model is biased. This is especially problematic in high-stakes applications like healthcare, where biased predictions or decisions can lead to unequal treatment or access. Safe deployment of models based on the feature representations proposed herein will require thorough validation to detect potential biases and mitigation strategies for dealing with them.
This work was funded in part by Office of Naval Research grants N00014-21-1-2615 and N00014-23-1-2406. The authors acknowledge Research Computing at Arizona State University for providing GPU resources that have contributed to the research results reported within this paper.
[1] Julián David Arias-Londoño, Juan I Godino-Llorente, Maria Markaki, and Yannis Stylianou. On combining information from modulation spectra and mel-frequency cepstral coefficients for automatic detection of pathological voices. Logopedics Phoniatrics Vocology, 36(2):60-69, 2011.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
[3] Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In 2016 The British Machine Vision Conference (BMVC), volume 1 (2), page 3, 2016.
[57] Bogdan Woldert-Jokisz. Saarbruecken voice database, 2007. URL http://stimdb.coli.uni-saarland.de/.
[58] Ruitong Xiao, Haitong Zhang, and Yue Lin. Dgc-vector: A new speaker embedding for zero-shot voice conversion. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6547-6551. IEEE, 2022.
[59] Hao-Chun Yang and Chi-Chun Lee. An attribute-invariant variational learning for emotion recognition using physiology. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1184-1188. IEEE, 2019.
[60] Mang Ye, Jianbing Shen, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Augmentation invariant and instance spreading feature for softmax embedding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[61] Zhengxu Yu, Yilun Zhao, Bin Hong, Zhongming Jin, Jianqiang Huang, Deng Cai, and Xian-Sheng Hua. Apparel-invariant feature learning for person re-identification. IEEE Transactions on Multimedia, 24:4482–4492, 2021.
[62] Jianwei Zhang, Julie Liss, Suren Jayasuriya, and Visar Berisha. Robust vocal quality feature embeddings for dysphonic voice detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1348-1359, 2023.
[63] Quanhua Zhao, Shuhan Jia, and Yu Li. Hyperspectral remote sensing image classification based on tighter random projection with minimal intra-class variance algorithm. Pattern Recognition, 111:107635, 2021.
[64] Liang Zheng, Yujia Huang, Huchuan Lu, and Yi Yang. Pose-invariant embedding for deep person re-identification. IEEE Transactions on Image Processing, 28(9):4500–4509, 2019.
A ICC Formulation for Imbalanced Classes
The proposed ICC and ICC regularizer in the main paper require equal class size, i.e.,  $M$  is the same for all classes. However, this is not typically the case in actual usage. In this section, we propose an extended version of the ICC and ICC regularizer, which are capable of handling datasets with unbalanced class sizes.
Assume there are  $N$  classes in the dataset or batch, and  $k_{j}$  is the number of samples for  $j$ -th class, the  $\mathbf{x}_{ji}$  represents the  $i$ -th samples of the  $j$ -th class,  $\mathbf{e}_{ji}$  represents the corresponding embedding vector, and  $e_{ji}^{l}$  represents the  $l$ -th embedding dimension of the  $\mathbf{e}_{ji}$ . The between-class variance can be written as:
$$
M S _ {B} \left(e ^ {l}\right) = \frac {\sum_ {j = 1} ^ {N} k _ {j} \cdot \left(\overline {{e _ {j} ^ {l}}} - \overline {{e ^ {l}}}\right)}{N - 1}, \tag {9}
$$
where  $\overline{e_j^l}$  represents the mean of  $l$ -th embedding dimension for the  $j$ -th class:
$$
\overline {{e _ {j} ^ {l}}} = \frac {\sum_ {i = 1} ^ {k _ {j}} e _ {j i} ^ {l}}{k _ {j}}, \tag {10}
$$
and  $\overline{e^l}$  represents the overall mean of  $l$ -th embedding dimension,
$$
\bar {e ^ {l}} = \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\sum_ {i} ^ {k _ {j}} e _ {j i} ^ {l}}{k _ {j}}. \tag {11}
$$
Then  $ICC(e^{l})$ , the ICC score for  $l$ -th embedding dimension, can be calculated as follows:
$$
I C C \left(e ^ {l}\right) = \frac {M S _ {B} \left(e ^ {l}\right) - \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\sum_ {i = 1} ^ {k _ {j}} \sigma_ {j i , l} ^ {2}}{k _ {j} - 1}}{M S _ {B} \left(e ^ {l}\right) + \frac {1}{N} \sum_ {j = 1} ^ {N} \sum_ {i = 1} ^ {k _ {j}} \sigma_ {j i , l} ^ {2}}, \tag {12}
$$
where  $\sigma_{ji,l}^2 = (e_{ji}^l -\overline{e_j^l})^2$  represents the within-class variance of  $l$ -th embedding dimension for  $i$ -th sample of the  $j$ -th class. The Equation 12 computes the intra-class variance for each class individually, taking into account the number of samples present in each class, denoted as  $k_{j}$ . Thus, Equation 12 effectively addresses issues related to class size imbalance.
Then the ICC score and ICC regularizer still can be calculated by using
$$
I C C (\mathbf {e}) = \frac {\sum_ {l} ^ {L} I C C (e ^ {l})}{L}, \tag {13}
$$
$$
R _ {I C C} = 1 - I C C (\mathbf {e}), \tag {14}
$$
respectively, for the datasets with unbalanced class sizes.
B Hyperparameter Ablation for GE2E Loss with ICC Regularization
As described in the Section 3.2 of the main paper, the ICC regularizer cannot be used alone and requires hyperparameter fine-tuning when combined with the contrastive loss. Therefore, based on the simulation data in Section 3.2 in the paper, the simulation contour figures of the  $(1 - \lambda)L_{GE2E} + \lambda R_{ICC}$  function value of intra- and inter-class variance for different values hyperparameter  $\lambda$  are provided in Figure B.1.
Figure B.1: The simulation contour figures of the  $(1 - \lambda)L_{GE2E} + \lambda R_{ICC}$  function value of intra- and inter-class variance, where  $\lambda = \{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\}$ .
Referring to Figure B.1, it's observed that the impact of the hyperparameter  $\lambda$  on the shape of the  $(1 - \lambda)L_{GE2E} + \lambda R_{ICC}$  function's contour lines is linear under the given simulation conditions. As the value of this hyperparameter  $\lambda$  increases, the contour of  $(1 - \lambda)L_{GE2E} + \lambda R_{ICC}$  leans more towards the ICC regularizer. This means it increasingly emphasizes on reducing the intra-class variance.
C Model and Training Details for Task 3: Dysphonic Voice Detection
We rebuild the Zhang et al. network [62] and implement the ICC regularizer to enhance the repeatability of voice feature embeddings. We follow the procedure described in their work [62] for training the voice feature embeddings. To enforce repeatability using the ICC regularizer, we use the recordings of healthy subjects from the mPower corpus [5] to ensure the embeddings do not change daily for the healthy person.
Model Structure: We use the same encoder and MLP classifier from the Zhang et al. paper [62]. A two-branch structure is used for optimizing the dysphonic sensitivity and repeatability of voice feature embeddings simultaneously as shown in Figure C.2, where the MLP networks is composed by three linear layers and two Leaky-ReLU activation layers (negative slope is 0.4). All three embeddings have a dimension of 256.
Training Loss: We use the following loss:
Figure C.2: The diagram of training repeatability enhanced voice feature embeddings for dysphonic voice detection.
$$
L = 0. 5 \times R _ {I C C} + 1. 0 \times L _ {\text {c o n t r}} + 1. 0 \times L _ {\text {c l a s s}}, \tag {15}
$$
where the  $R_{ICC}$  is the ICC regularizer on repeat-constrained embeddings,  $L_{contr}$  is the contrastive loss on the voice feature embeddings, and  $L_{class}$  is the classification loss.
Training and Evaluation Datasets: We use the Saarbruecken Voice Database (SVD) [57] as the training and in-corpus validation dataset for dysphonic voice detection task, and the mPower corpus [5] is used only in training for improving the repeatability of voice feature embeddings. The Massachusetts Eye and Ear Infirmary (MEEI) database and Hospital Principe de Asturias (HUPA) [1] dataset are used for cross-corpus testing datasets for dysphonic voice detection task, and the ALS [44] dataset is used for repeatability evaluation. The MEEI, HUPA, and ALS are unseen during training for all methods.