SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianwei Zhang
School of Electrical, Computer and Energy Engineering
Tempe, AZ 85281
jianwei.zhang@asu.edu
Suren Jayasuriya
School of Arts, Media and Engineering
sjayasar@asu.edu
Visar Berisha
College of Health Solutions
visar@asu.edu
A good supervised embedding for a specific machine learning task is only sensitive to changes in the label of interest and is invariant to other confounding factors. We leverage the concept of repeatability from measurement theory to describe this property and propose to use the intra-class correlation coefficient (ICC) to evaluate the repeatability of embeddings. We then propose a novel regularizer, the ICC regularizer, as a complementary component for contrastive losses to guide deep neural networks to produce embeddings with higher repeatability. We use simulated data to explain why the ICC regularizer works better on minimizing the intra-class variance than the contrastive loss alone. We implement the ICC regularizer and apply it to three speech tasks: speaker verification, voice style conversion, and a clinical application for detecting dysphonic voice. The experimental results demonstrate that adding an ICC regularizer can improve the repeatability of learned embeddings compared to only using the contrastive loss; further, these embeddings lead to improved performance in these downstream tasks.
Embeddings, which are relatively low-dimensional latent representations of high-dimensional inputs, are widely used in deep learning applications and often trained through supervised learning techniques. In such cases, effective embeddings should be sensitive to changes in the target class (e.g., speaker identity, clinical class) while remaining invariant to unrelated factors (e.g., noise, natural variations of the data). Embeddings that satisfy this desired property have high repeatability, a term borrowed from measurement theory where it characterizes the consistency between the outcomes of consecutive
measurements of the same target when the underlying conditions remain unaltered [23, 45]. For instance, in a text-independent speaker verification (TI-SV) task, the speaker embedding extracted from recordings with varying content from the same speaker should remain consistent. Similarly, in a dysphonic voice detection task, voice feature embeddings derived from the vowel phonation of a healthy person over consecutive days should remain consistent. By achieving high repeatability, embeddings effectively capture essential features while disregarding irrelevant factors.
A key to improving repeatability is reducing the intra-class variance, which is often increased by various confounding factors. Several studies have proposed different approaches to handling intra-class variation including increasing the variability of training data [60, 47, 7, 13, 61] and novel learning algorithms [2, 4, 42]. However, repeatability is rarely explicitly considered during training or evaluation of embeddings. In most cases, only the downstream applications' performance is used to indirectly evaluate the embeddings' effectiveness. We posit that directly assessing repeatability, regardless of downstream application, can help improve the quality of learned latent representations.
In this paper, we propose to use the intra-class correlation coefficient (ICC) to evaluate embeddings' repeatability. The ICC was designed to assess the consistency between two or more quantitative measurements [36], and often used to evaluate the repeatability of metrics across different fields [28, 33, 49]. We further propose a novel regularizer based on the ICC as a complementary component to traditional contrastive losses to enforce deep architectures to learn repeatable embeddings. Some contrastive losses, such as GE2E [54], push embeddings towards the centroid of the true class to reduce their intra-class variance. Via analysis and intuition, we explain why the ICC regularizer better focuses on minimizing the intra-class variance than contrastive loss alone and provide a new perspective for latent representation learning.
Repeatability is an especially difficult property to enforce in complex, high-dimensional signals like speech. Speech characteristics depend on the speaker's neurological and physiological state, the degrees of freedom in the speaking task, the recording setup, the environment, etc. [45]. These sources of variation challenge the development of embeddings for a particular task (e.g., learning speaker embeddings). We use the ICC regularizer to improve the embeddings' repeatability in three speech tasks: speaker embeddings for TI-SV, zero-shot voice style conversion, and voice feature embeddings for a clinical application of dysphonic voice detection. Our experimental results demonstrate that the proposed ICC regularizer can significantly improve the repeatability of learned embeddings, and embeddings with higher repeatability perform better in the downstream tasks. In the TI-SV task, the speaker embeddings' repeatability is significantly enhanced, and the EER decreases by  $\sim 10\%$  compared to the methods without ICC regularizer. AB preference test results for zero-shot voice style conversion show that embeddings with higher repeatability are preferred  $62\%$  to  $38\%$  over those with lower repeatability. Objective evaluation metrics further confirm that more repeatable embeddings lead to improved performance in the voice style conversion task. In the clinical application, we demonstrate that highly repeatable voice feature embeddings further improve the in-corpus classification accuracy by  $\sim 3\%$  and model generalizability across different corpora.
- We connect the concept of repeatability from measurement theory to deep-learned embeddings and suggest the ICC for evaluating the quality of embeddings.
- We propose a novel regularizer, the ICC regularizer, as a complementary component for contrastive loss to regularize the deep-learned embeddings such that they are repeatable.
- We illustrate the reason why the ICC regularizer better minimizes the intra-class variance than the contrastive loss and provide a new perspective for latent representation learning.
- Our experimental results demonstrate that ICC regularizer can improve the repeatability of learned embeddings, and embeddings with higher repeatability exhibit better performance in downstream tasks.
For reproducibility of our work, the code for the ICC regularizer and experiments is available open-source in our GitHub repository<sup>1</sup>.
$^{1}$ https://github.com/vigor-jzhang/icc-regularizer/
Learning invariant embeddings: Previous literature has proposed to improve the robustness of learned latent representations by learning invariant embeddings. Most of these studies made the embeddings invariant to one or two types of variation, e.g. pose-invariance for re-identification [64, 35], noise-invariance for speaker recognition [7, 38], personality-invariance for emotion recognition [59]. There are two main approaches to promoting invariance to confounding factors: (1) increase the variability of training data, e.g., combining datasets from different domains [13], data warping [60, 47], data augmentation by GANs [61]; and (2) novel learning paradigms, such as variance-invariance-covariance regularization [4], invariant risk minimization [2], and simultaneously enforcing equivariance and invariance [42]. In this work, we follow the second approach by introducing a new regularizer for training embeddings.
Intra-class and inter-class variance: The variation between multiple observations of a class (intra-class variance) and the variation between classes (inter-class variance) define the performance of many machine learning criteria. Minimizing intra-class variance and maximizing inter-class variance are keys to many deep learning tasks, including classification [39, 63], representation learning [17, 29, 41], and few-shot learning [10, 8, 46]. However, these works typically rely on visualization (e.g., t-SNE, UMAP) to show how well their methods minimize intra-class variance and maximize inter-class variance [29, 34]. The proposed metric for repeatability, the ICC, can directly evaluate the performance of a given method using an already-established and well-understood metric from measurement theory. Furthermore, we can construct a regularizer centered around this metric.
Contrastive loss: Researchers have proposed finding task-relevant embedding features by contrastive representation learning and leveraging labeled data [50]. There are several popular contrastive losses in deep learning, such as the triplet loss [3, 31], tuple-based end-to-end (TE2E) loss [20], generalized end-to-end (GE2E) loss [54], momentum contrast (MoCo) [19], and SimCLR [9]. A contrastive loss encourages inputs of the same label class to have more similar latent representations compared to inputs from different classes. We select GE2E loss as a representative contrastive loss for comparison as it is widely adopted and used for supervised learning. The similarities and differences between the contrastive loss and the ICC regularizer are discussed in Section 3.2.
Intra-class correlation coefficient: Repeatability is most frequently measured via an intra-class correlation coefficient (ICC) [56]. Shrout and Fleiss elaborated several cases and corresponding formulas for ICC [43]. The ICC is used widely in different fields to estimate the reliability and repeatability, including clinical applications [45, 14], psychology and behavioral science [33, 15], and medical imaging [6, 49]. To the best of our knowledge, the ICC has not been applied for embeddings training and evaluation.
In this section, we present the intra-class correlation coefficient (ICC) for evaluating the repeatability of deep-learned embeddings, and we present a novel ICC regularizer for enforcing repeatability in learned embeddings during training. Then, we illustrate the similarities and differences between contrastive loss and ICC regularizer by analyzing the intra-class and inter-class variance in relation to these two elements. Finally, we highlight why the ICC regularizer cannot function independently and its requirement for hyperparameter fine-tuning.
3.1 Intra-class correlation regularizer
Shrout and Fleiss elaborated several cases and corresponding formulas for the ICC [43], and we select the ICC 1-1 formulation, which is a measure of absolute agreement (i.e., the model generates the same embeddings to different samples from the same target) [16, 27], for assessing the repeatability of deep-learned embeddings.
In our problem formulation, we assume there are a set of high-dimensional data  $\mathbf{x} \in \mathbb{R}^D$ , each with label  $\mathbf{y} \in [1,\dots,N]$ . The goal is to learn lower-dimensional latent representations  $\mathbf{e} \in \mathbb{R}^L$  that exhibit maximum separability: embeddings belonging to the same class should be closely clustered together, while embeddings from different classes should be distinctly separated from each other [32]. An encoder with weights  $\mathbf{w}$  and characterized by  $f(\mathbf{x};\mathbf{w})$ , takes  $\mathbf{x}$  as input and outputs
lower-dimensional representation  $\mathbf{e}$ . During training, the weights of the encoder are modified by optimizing a loss function that depends on  $\mathbf{x}$  and labels  $\mathbf{y}$ . Considering a dataset with  $\mathbf{y} \in [1, \dots, N]$  and  $M$  input samples per class, the embedding vector  $\mathbf{e}_{ji}$  is defined as the  $\ell_2$  normalization of the encoder output  $f(\mathbf{x}_{ji}; \mathbf{w})$  ( $1 \leq j \leq N$ ,  $1 \leq i \leq M$ ,  $1 \leq l \leq L$ ):
$$
\mathbf {e} _ {j i} = \left[ e _ {j i} ^ {1}, e _ {j i} ^ {2}, \dots , e _ {j i} ^ {l}, \dots \right] = \frac {f (\mathbf {x} _ {j i} ; \mathbf {w})}{\| f (\mathbf {x} _ {j i} ; \mathbf {w}) \| _ {2}}, \tag {1}
$$
where the  $\mathbf{x}_{ji}$  represents the  $i$ -th sample of  $j$ -th class and the  $\mathbf{e}_{ji}$  represents the corresponding embedding vector (the notation is similar to other contrastive losses), and  $e_{ji}^{l}$  represents the  $l$ -th embedding dimension of  $\mathbf{e}_{ji}$ .
Then  $ICC(e^{l})$ , the ICC score for  $l$ -th embedding dimension, can be calculated as follows [16]:
$$
I C C \left(e ^ {l}\right) = \frac {M S _ {B} \left(e ^ {l}\right) - M S _ {W} \left(e ^ {l}\right)}{M S _ {B} \left(e ^ {l}\right) + (M - 1) M S _ {W} \left(e ^ {l}\right)}. \tag {2}
$$
Here the  $MS_B(e^l)$  represents inter-class (between-class) variance $^2$  and  $MS_W(e^l)$  represents the intra-class (within-class) variance for  $l$ -th embedding dimension.  $MS_B(e^l)$  is calculated as
$$
M S _ {B} \left(e ^ {l}\right) = \frac {M \cdot \sum_ {j = 1} ^ {N} \left(\overline {{e _ {j} ^ {l}}} - \overline {{e ^ {l}}}\right) ^ {2}}{N - 1}, \tag {3}
$$
where  $\overline{e_j^l} = \sum_i^M e_{ji}^l /M$  represents the mean of  $l$ -th embedding dimension for the  $j$ -th class and  $\overline{e^l} = \sum_j^N\sum_i^M e_{ji}^l /(N\times M)$  represents the overall mean of  $l$ -th embedding dimension. Intuitively,  $MS_B(e^l)$  measures the inter-class variance of  $l$ -th embedding dimension.
$MS_W(e^l)$  is calculated as
$$
M S _ {W} \left(e ^ {l}\right) = \frac {\sum_ {j} ^ {N} M \sigma_ {j , l} ^ {2}}{N (M - 1)}, \tag {4}
$$
where  $\sigma_{j,l}^2 = \sum_i^M (e_{ji}^l -\overline{e_j^l})^2 /M$  represents the variance of  $l$ -th embedding dimension for  $j$ -th class. Intuitively,  $MS_W(e^l)$  measures the overall intra-class variance of  $l$ -th embedding dimension.
After obtaining the ICC score for each embedding point, we use the mean value of all embedding dimensions' ICC scores as the repeatability metric:
$$
I C C (\mathbf {e}) = \frac {\sum_ {l} ^ {L} I C C (e ^ {l})}{L}. \tag {5}
$$
ICC score interpretation: When the learned embeddings exhibit perfect repeatability (characterized by intra-class variance  $MS_W = 0$  and inter-class variance  $MS_B > 0$ ), the ICC score is equal to 1. A decrease in the ICC score signifies reduced repeatability, which implies a relative increase in intra-class variance  $MS_W$  compared to inter-class variance  $MS_B$ . If the intra-class variance  $MS_W$  exceeds the inter-class variance  $MS_B$ , the ICC score may become negative<sup>3</sup>.
We propose a novel regularizer, the ICC regularizer, for regularizing learned representations to enforce high repeatability. The ICC regularizer operates on each batch: we assume a single batch contains samples from  $N$  classes, and  $M$  input samples from each class, and the dimension of the embedding is  $L$ . The ICC loss firstly uses Equation 5 to calculate the mean ICC score  $ICC(\mathbf{e})$  for the embeddings  $\mathbf{e}$  of current batch, and then the ICC regularizer can be written as
${}^{2}$  Often noted as  $M{S}_{R}$  in ICC related literature [43, 16].
For instance, consider a simple numerical example where there is  $N = 2$  classes and  $M = 6$  samples per class. Class 1 has mean  $\mu_{1} = 0$  and variance  $\sigma_1^2 = 100$ , class 2 has mean  $\mu_{2} = 0.1$  and variance  $\sigma_2^2 = 100$ . In this situation,  $MS_{B} = 0.3$ ,  $MS_{W} = 120$ , then  $ICC = -0.199 < 0$ .  $M$  has big effect in this situation: when  $M$  is large, the ICC score is still negative but very close to 0.
$$
R _ {I C C} = 1 - I C C (\mathbf {e}). \tag {6}
$$
In this paper, the ICC and ICC regularizer require equal class size, i.e.,  $M$  is the same for all classes. However, to expand the ICC regularizer usage for scenarios where the number of samples per class may not be equal, we provide an extended version of the ICC formulation and code for the imbalanced classes in the Appendix A and in our GitHub repository<sup>4</sup>.
3.2 ICC regularizer vs. contrastive loss
Figure 1: The contour figures for (a) GE2E loss and (b) ICC regularizer value of intra- and inter-class variance. To explore the gradient trend of GE2E loss and ICC regularizer, some starting points (red dots) are selected, then the maximum gradient descent path (red dashed lines) of value is calculated and plotted. (c) The contour figure for SVM classification error rate on simulation data per intra- and inter-class variance.
The ICC regularizer and contrastive loss have similarities in their optimization criteria: both aim to minimize the intra-class variance and maximize the inter-class variance. However, they exhibit different tradeoffs between the two variances. We use a Monte Carlo simulation to study the similarities and differences of the ICC regularizer and contrastive loss on the intra-class and inter-class variance. We use the GE2E loss [54] as a representative contrastive loss in simulation<sup>5</sup>.
Simulation setup: In simulation, we vary the intra- and inter-class variances to generate samples as follows: (1) the intra-class variance, i.e., the variance of simulated embeddings within one class, varies from 0.02 to 2.0 with a step size of 0.02; (2) the inter-class variance, i.e., the variance of class centroids, varies from 0.01 to 0.60 with a step size of 0.01. For a pair of configurations (e.g. inter-class, intra-class pair), we draw 400 samples from an 8-dimensional, 4-class Gaussian mixture such that the class-conditional mean and variance yield the desired inter-class, intra-class variance pair. We calculate the ICC regularizer value and GE2E loss value for these random samples. We repeat the Monte Carlo simulation 100 times and plot the loss values in Figure 1 for the ICC regularizer and GE2E loss as a function of the inter-class and intra-class variance. To explore their landscapes, we select several starting points, then trace the path of maximum gradient descent.
Discussion: The ICC regularizer and GE2E loss have very different contours. They both have a low value when the inter-class variance is high and the intra-class variance is low. However, when embeddings are normalized (see Equation 1), the normalization places an upper limit on the total variance of embeddings and limits how large the inter-class variance can be (the inter-class variance cannot go to infinity). So, a regularizer that places a greater emphasis on minimizing the intra-class variance for a bounded inter-class variance will naturally lead to embeddings with higher repeatability compared to a loss that focuses on simultaneously maximizing both variances.
The GE2E loss tries to optimize both intra-class and inter-class variances simultaneously: the maximum gradient descent direction of the GE2E loss overall is from the lower-right corner to the upper-left corner. This loss continues to decrease as the inter-class variance increases, even after the intra-class variance is small enough (the descent path of point 1 in Figure 1(a)). We note that the
4https://github.com/vigor-jzhang/icc-regularizer/
We implement the GE2E loss with 'softmax' type, i.e., using Equation 6 in their paper. The GE2E loss use two learnable parameters  $w, b$  for their similarity metric calculation  $S = w \cdot \cos(\cdot) + b$ , but we fixed  $w = 10, b = -5$  as initial values suggested in their paper for simulation.
embeddings of different classes are already clustered well under this scenario. Therefore, further increasing the inter-class variance does not improve the separability between the embeddings as they are already separated. It is difficult for the GE2E loss to further minimize the intra-class variance without increasing the inter-class variance based on the contours of the loss. Our simulation explains findings from several studies showing that contrastive losses, including the GE2E, do not perform well in reducing intra-class variance [30, 55].
In contrast, the ICC regularizer focuses on decreasing the intra-class variance and places less emphasis on the inter-class variance, thereby enhancing repeatability of the learned embeddings. The minimum for the ICC regularizer occurs when the embeddings' intra-class variance is approximately equal to 0, or the inter-class variance reaches a relatively large value compared to the intra-class variance. The ICC regularizer pushes the embeddings towards lower intra-class variance and not towards larger inter-class variance once the inter-class variance exceeds the intra-class variance (ensuring the embeddings are clustered well) as shown in Figure 1 (b). This naturally leads to embeddings with better repeatability compared to the GE2E loss.
A simple analysis of gradients explains this observation. The gradient of the ICC regularizer with respect to the two variance terms is
$$
\frac {\partial R _ {I C C}}{\partial M S _ {B}} = - \frac {M \times M S _ {W}}{\left(M S _ {B} + (M - 1) \times M S _ {W}\right) ^ {2}}, \tag {7}
$$
$$
\frac {\partial R _ {I C C}}{\partial M S _ {W}} = \frac {M \times M S _ {B}}{\left(M S _ {B} + (M - 1) \times M S _ {W}\right) ^ {2}}. \tag {8}
$$
When the intra-class variance  $(MS_W)$  is already small, the derivative of the ICC regularizer with respect to the inter-class variance  $(MS_B)$  is small in absolute value; therefore, the gradient descent step along the inter-class variance dimension is small. When the inter-class variance is relatively large, the derivative of the ICC regularizer with respect to the intra-class variance is larger, which means the gradient descent step along the intra-class variance dimension is relatively large. This is clear from Figure 1 (b) where the points that begin with a small intra-class variance (e.g. point 2 in Figure 1 (b)) do not focus on further increasing the inter-class variance to improve the ICC as it is unnecessary. In contrast, the trajectory of point 1 in 1 (a) goes towards increasing the inter-class variance, even when the intra-class variance is small and the clusters are already well separated.
The ICC regularizer is also better aligned with classification error rate. We use the same simulation data and train a SVM for each intra- and inter-class variance pair to generate the contour figure for SVM classification error rate. As we show in Figure 1 (c), the classification error rate has a similar contour when compared with the ICC regularizer values: the lower ICC regularizer (higher the repeatability), the lower the classification error rate. This result supports our hypothesis that embeddings with improved repeatability will benefit downstream applications.
In summary, given the constraints on the total variance of the embeddings, it is beneficial to focus on the intra-class variance for increasing representation repeatability. Although the optimization objectives of contrastive loss and ICC regularizer are similar, the contrastive loss focuses on optimizing intra-class and inter-class variance simultaneously whereas the ICC regularizer places greater emphasis on minimizing intra-class variance.
The ICC is not a replacement for contrastive loss: The ICC is computed for each dimension of the embedding independently, while contrastive loss is calculated for the entire embedding vector. It's important to note that high separability in each embedding dimension doesn't necessarily translate into good separability in the overall embedding space and can lead to learned embedding dimensions that are highly correlated; this can have a negative impact on downstream model performance [51]. Relatedly, the cosine similarity score is commonly used to evaluate the fidelity of learned embeddings. A high ICC value per dimension does not necessarily imply a high cosine similarity. For these reasons, we propose to use the ICC as a regularizer rather than a stand-alone loss.
The ICC regularizer requires hyperparameter fine-tuning: As with other regularizers, a linear combination of contrastive loss and the ICC regularizer, represented by the equation  $L_{\text{contr}} + \lambda R_{\text{ICC}}$ , requires selection of the hyperparameter  $\lambda$ . In our experiments, we used a grid search to determine the optimal  $\lambda$ . For visualizations of the loss contour (similar to those in Figure 1) of the combined
contrastive+ICC loss, we refer the reader to the Appendix B where we have included the combined loss contour with varying  $\lambda$  values.