SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weihang Dai $^{1}$ , Yao Du $^{1}$ , Hanru Bai $^{3}$ , Kwang-Ting Cheng $^{1}$ , Xiaomeng Li $^{1,2*}$
<sup>1</sup>The Hong Kong University of Science and Technology
$^{2}$ HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Futian, Shenzhen
$^{3}$ Fudan University
eepromli@ust.hk
Contrastive learning methods can be applied to deep regression by enforcing label distance relationships in feature space. However, these methods are limited to labeled data only unlike for classification, where unlabeled data can be used for contrastive pretraining. In this work, we extend contrastive regression methods to allow unlabeled data to be used in a semi-supervised setting, thereby reducing the reliance on manual annotations. We observe that the feature similarity matrix between unlabeled samples still reflect inter-sample relationships, and that an accurate ordinal relationship can be recovered through spectral seriation algorithms if the level of error is within certain bounds. By using the recovered ordinal relationship for contrastive learning on unlabeled samples, we can allow more data to be used for feature representation learning, thereby achieve more robust results. The ordinal rankings can also be used to supervise predictions on unlabeled samples, which can serve as an additional training signal. We provide theoretical guarantees and empirical support through experiments on different datasets, demonstrating that our method can surpass existing state-of-the-art semi-supervised deep regression methods. To the best of our knowledge, this work is the first to explore using unlabeled data to perform contrastive learning for regression. Code is available at https://github.com/xmed-lab/CLSS.
Contrastive learning is an effective technique for improving feature representations in deep neural networks (DNNs) [3, 4, 19, 33, 10]. These methods involve identifying positive and negative sample pairs based on feature similarity, where positive pairs are usually defined as augmented samples from the same input or samples from the same class. Early works such as SimCLR [3], MoCo [4], and SupCon [19], showed that pretraining DNNs through contrastive learning can lead to state-of-the-art classification results due to improved feature representations. These techniques have since been extended to segmentation [33, 43], video recognition [10], multi-modality learning [28], as well as deep regression [8, 42, 37] to great effect.
It is possible to perform contrastive learning for classification in an unsupervised manner, which allows large amounts of unlabeled data to be used [3, 4, 28]. Unsupervised contrastive learning is not possible for deep regression tasks however, because in order for feature representations to be effective for regression, they must reflect label distance relationships in feature space [8, 42, 37]. Contrastive regression loss functions ensure that sample pairs with smaller label distances have features that are
Figure 1: Obtaining rankings from feature similarity. (a) Given labeled samples with well trained features, the ranking of an unlabeled sample can be easily inferred to be between the two with the highest similarity values. (b) The correct ranking can still be inferred with noisy features, as long as the noise levels are within certain thresholds. (c) Given a feature similarity matrix of unlabeled samples, the correct ranking can still be obtained if noise levels are within certain bounds.
more similar, compared to those with greater label distance. Because of this, existing methods can only be used on fully labeled datasets, which can be a limiting factor when annotations are costly to obtain. This is especially true for medical imaging analysis, where deep regression is important for estimating real-number medical indicators but often require expert knowledge for manual annotation [8, 20, 40].
In this work, we extend contrastive learning for deep regression such that unlabeled data can also be used. We consider a semi-supervised setting where only a small portion of the training dataset is labeled, and the majority of samples are unlabeled. We observe that by enforcing supervised contrastive learning on labeled samples, the feature similarity matrix of unlabeled samples will also learn to reflect label distance [8]. Although the unlabeled feature similarity matrix will be inaccurate and noise corrupted, it is still possible to infer the relative ordering of samples if errors are within certain bounds. To give an intuitive example, it is easy to infer the relative ranking of an unlabeled sample based on its pair-wise feature similarity against labeled samples, assuming that features are well-trained (see Fig. 1a). Even in the presence of noisy features, it is still possible to infer the correct ranking if most of the similarity values are reliable (Fig. 1b). Similarly, given a feature similarity matrix for a batch of unlabeled samples, it is still possible to infer their relative ordering even with some degree of noise present (Fig. 1c).
To this end, we propose a novel semi-supervised contrastive regression method by making use of ordinal rankings recovered from the unlabeled feature similarity matrix. We make use of the spectral seriation algorithm proposed by Atkins et al. [2] for ranking recovery, which can then be used to construct a distance matrix for supervising contrastive learning on the unlabeled samples. The spectral seriation algorithm is based on an error-minimization approach and allows for some degree of error correction when extracting rankings from the feature similarity matrix. Thus, the ordinal rankings can also be used to supervise predictions on unlabeled samples, which we demonstrate leads to further improvements for semi-supervised regression. We term our method Contrastive Learning with Spectral Seriation (CLSS). Fig. 2 illustrates our overall framework.
We provide theoretical proofs and empirically show that our method can achieve state-of-the-art performance on multiple datasets. Our method is highly beneficial for critical applications such as medical imaging analysis, where annotations can be expensive to obtain. To the best of our knowledge, CLSS is the first to extend contrastive learning for deep regression to unlabeled data. To summarize, our main contributions are: (1) we propose CLSS, a novel semi-supervised contrastive learning method that is the first to use unlabeled data for contrastive regression; (2) we demonstrate that spectral seriation can be used to extract robust rankings from the feature similarity matrix of
Figure 2: Framework for Contrastive Learning with Spectral Seritation (CLSS). Unlike existing works for contrastive learning that are only able to use labeled data, we make use of spectral seriation to obtain ordinal rankings for unlabeled samples. This can then be used for constrative learning and ranking supervision for unlabeled samples.
unlabeled samples for supervision; (3) we demonstrate that our method can outperform existing state-of-the-art alternatives on different deep regression tasks using multiple datasets.
Contrastive Learning
Contrastive learning for classification involves identifying positive and negative sample pairs based on feature similarity, which allows similar samples to be grouped closer together in feature space [3, 19, 15]. These methods can be performed supervised or unsupervised depending on how positive pairs are defined. SimCLR [3] and MoCo [15] defines positive pairs as augmented samples from the same input data, which means labels are not required. Supervised methods such as SupCon [19] define positive pairs as samples belonging to the same class.
Contrastive learning for deep regression requires features to reflect label distance relationships in feature space. Dai et al. propose AdaCon, which uses adaptive margins in the SupCon loss function to encourage similarity values to reflect pair-wise label distance [8]. Xue et al. [37] find that determining positive and negative pairs based on distance thresholds can be effective for regression tasks. Zhang et al. introduce ordinal entropy, which also encourages features to be spread out depending on their pair-wise label distance [42]. Results from these methods show that improving feature learning is a highly effective way to improve deep regression model performance. Because label information is required for these existing contrastive regression loss functions, they cannot be used with unlabeled data however. Our method, CLSS, is the first to extend contrastive learning for regression to allow unlabeled data to also be used for training.
Semi-Supervised Regression
Semi-supervised learning allows unlabeled data to be used with labeled data for model training. It is an effective way of reducing reliance on manual annotations and is particularly valuable for applications where labeled data is costly to obtain [9, 22, 23, 39, 21]. Semi-supervised classification methods are well studied [29, 41], but semi-supervised deep regression problems receive significantly less attention [7]. Early works such as COREG [44] propose using co-trained KNN neural networks to enforce consistency between two models. Consistency based methods have also been explored in [36, 34]. Iterative approaches for generating pseudo-labels [12], deep kernel learning methods [25, 35, 18], and graph based methods [31] have been proposed for semi-supervised learning in more recent works. A major limitation of these methods is that they are primarily designed for structured tabular data however, and cannot be trained end-to-end with a deep feature extractor for unstructured inputs.
Dai et al. proposed UCVME [7], which enforces consistent uncertainty predictions between constrained models, and can be used for unstructured inputs such as images and videos. Although this
method achieves good performance, it ignores potential improvements that can be made on the feature level, which is critical to the performance of deep regression models [8, 42]. In this work, we are the first to address the challenge of performing contrastive learning for deep regression by making use of unlabeled data.
Our novel method allows contrastive learning to be performed with unlabeled data under a semi-supervised framework. We denote  $\mathcal{D} \coloneqq \{(x_i, y_i)\}_{i=1}^N$  as the labeled dataset consisting of  $N$  samples, where  $x_i$  is the input data and  $y_i$  is its corresponding label in  $\mathbb{R}$ . We denote  $\mathcal{D}' \coloneqq \{x_{i'}'\}_{i'=1}^{N'}$  as the unlabeled dataset consisting of input data only. We denote the feature extractor as  $f(\cdot)$  and the feature vector as  $z$  for labeled data, such that  $z_i = f(x_i)$ . Similarly, we denote features for unlabeled data as  $z_i'$ . Contrastive learning is performed on the L2 normalized feature vector, with some methods also performing feature projection onto a lower dimension [8, 42, 3]. We denote the transformed, normalized feature vector for labeled data as  $\tilde{z}$  such that  $\tilde{z}_i = \frac{z_i}{||z_i||_2}$ . We use  $\tilde{z}'$  for unlabeled data. The feature similarity matrix for labeled and unlabeled samples are denoted as  $\mathcal{S}$  and  $\mathcal{S}'$  respectively.
We denote the regression head as  $g(\cdot)$ . The overall deep regression model can be expressed as:
$$
Y = g (f (X)) + \varepsilon ; \varepsilon \sim N (0, \sigma^ {2}). \tag {1}
$$
The model can be trained through supervised regression by minimizing mean squared error (MSE) loss, which we denote as  $\mathcal{L}^{SR}$ :
$$
\mathcal {L} ^ {S R} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(y _ {i} - \hat {y} _ {i}\right) ^ {2}. \tag {2}
$$
Supervised contrastive learning can be performed on the normalized feature vectors for labeled samples  $\tilde{z}$  using different contrastive regression losses, which we denote as  $\mathcal{L}^{SC}$ . Spectral seriation can then be used to extract the ordinal rankings  $R^{\prime}$  from  $S^{\prime}$  for supervising contrastive learning and predictions on labeled samples. We describe our method in detail below.
3.2 Using ordinal ranking from spectral seriation for supervision
3.2.1 Spectral seriation for retrieving ordinal ranking from similarity matrix
Contrastive learning for deep regression aims to ensure that feature representations reflect label distance relationships in feature space [8, 42, 37]. If  $y_{i}$  and  $y_{j}$  are closer together than  $y_{i}$  and  $y_{k}$ , then the distance between features  $z_{i}$  and  $z_{j}$  should be closer together than  $z_{i}$  and  $z_{k}$ . Thus:
$$
\left| \left| \tilde {z} _ {i} - \tilde {z} _ {j} \right| \right| _ {2} <   \left| \left| \tilde {z} _ {i} - \tilde {z} _ {k} \right| \right| _ {2} \text {f o r} \left| y _ {i} - y _ {j} \right| <   \left| y _ {i} - y _ {k} \right|. \tag {3}
$$
Alternatively, because we use L2 normalized feature vectors,
$$
\gamma \left(\tilde {z} _ {i}, \tilde {z} _ {j}\right) > \gamma \left(\tilde {z} _ {i}, \tilde {z} _ {k}\right) \text {f o r} | y _ {i} - y _ {j} | <   | y _ {i} - y _ {k} |, \tag {4}
$$
where  $\gamma$  is the cosine similarity function, and  $\gamma (\tilde{z}_i,\tilde{z}_j)$  and  $\gamma (\tilde{z}_i,\tilde{z}_k)$  correspond to entries in similarity matrix  $S$ . By performing supervised contrastive learning on labeled samples, we can also expect entries in  $S^{\prime}$  to also approximate Eq. 4 with noise, as they have been trained to reflect distance relationships in feature space.
To retrieve the ordinal ranking  $R'$  of samples in an unlabeled batch, we can make use of the spectral seriation algorithm originally proposed by Atkins et al. [2]. The algorithm is designed to recover an ordinal ranking given a correlation matrix, where higher correlation values indicate closer proximity in ranking. The algorithm is especially useful for when it is easy to determine the similarity between sample pairs but difficult to obtain a direct ordering. Cosine similarity is equivalent to correlation after L2 normalization. Therefore spectral seriation can also be used to recover  $R'$  from  $S'$ .
The seriation problem can be formulated as a loss minimizing function:
$$
\underset {R ^ {\prime}} {\arg \min } \sum_ {i, j} \mathcal {S} _ {i, j} ^ {\prime} \left(R _ {i} ^ {\prime} - R _ {j} ^ {\prime}\right) ^ {2}, \tag {5}
$$
where  $S_{i,j}^{\prime} = \gamma (\tilde{z}_i,\tilde{z}_j)$ . Since sample pairs closer together in ranking have higher correlation values, minimizing this loss encourages these samples to have  $R_{i}$  and  $R_{j}$  that are closer together. As per spectral seriation, the solution to  $R^{\prime}$  can be derived from the Fiedler vector, as stated in Theorem 1.
Theorem 1 Given similarity matrix  $S'$  such that  $S_{i,j}' > S_{i,k}'$  for  $|y_i - y_j| < |y_i - y_k|$ , the ordinal ranking that best satisfies observed  $S'$  is the ranking of the values in the Fiedler vector of  $\mathbf{L}$ , where  $\mathbf{L}$  is the Laplacian of  $S_{i,j}'$ .
Intuitively, the proof of this theorem can be obtained by approximating discrete rankings  $R'$  with real-number values  $r'$  and expressing Eq. 5 in the form of:
$$
\min  _ {r ^ {\prime T} e = 0, r ^ {\prime T} r ^ {\prime} = 1} r ^ {\prime T} \mathbf {L} r ^ {\prime}. \tag {6}
$$
$r'$  can be solved by computing the Fiedler vector, which is the eigenvector corresponding with the smallest non-zero eigenvalue. The relative rankings of  $r'$  then give us  $R'$ . We refer interested readers to [2] for detailed derivations.
3.2.2 Unlabeled contrastive learning with seriation rankings
Spectral seriation gives us rank  $R'$  of unlabeled samples within a batch. From this, we can obtain rankings on the distances between sample pairs for some anchor sample  $i$ . The distance ranking can be used to supervise contrastive learning on unlabeled samples by ensuring that the feature similarity values are consistent with the rankings. We define the unlabeled contrastive learning loss with respect to some subset  $\mathcal{B}$  as:
$$
\mathcal {L} ^ {U C} = \sum_ {i = 1} ^ {| \mathcal {B} |} \ell \left(\mathbf {r k} \left(\mathcal {S} _ {[ i,: ]} ^ {\prime}\right), \mathbf {r k} \left(- \left| R ^ {\prime} - R _ {[ i ]} ^ {\prime} \right|\right); \lambda\right), \tag {7}
$$
where  $[i, :]$  denotes the  $i$ th row in the matrix,  $[i]$  denotes the  $i$ th value of a vector, rk denotes the ranking operator, and  $\ell$  is the ranking similarity function. This loss function ensures that the ranking of the feature similarity values with anchor sample  $i$  are consistent with that of the distances between derived rankings obtained from seriation. For the differentiable ranking similarity loss function  $\ell$ , we directly use the differential combinatorial solver proposed in [27], which takes an additional parameter  $\lambda$ . On a high level, the function makes use of interpolation methods to allow combinatorial inputs to be differentiable, although detailed explanations can be found in the original work.
3.2.3 Ranking supervision of unlabeled predictions with seriation rankings
Given  $R'$  for unlabeled samples, we can also perform supervision on the prediction output  $\hat{y}$ . The spectral seriation algorithm is inherently robust to error, which we show in section 3.3, which means pair-wise distance rankings from seriation are likely to be more accurate than the predicted output. We define the unlabeled prediction ranking loss with respect to some subset  $\mathcal{B}$  as:
$$
\mathcal {L} ^ {U R} = \sum_ {i = 1} ^ {| \mathcal {B} |} \ell \left(\mathbf {r} \mathbf {k} \left(- \left| \hat {y} ^ {\prime} - \hat {y} _ {[ i ]} ^ {\prime} \right|\right), \mathbf {r} \mathbf {k} \left(- \left| R ^ {\prime} - R _ {[ i ]} ^ {\prime} \right|\right); \lambda\right), \tag {8}
$$
where  $[i]$  denotes the  $i$ th value of a vector. This serves as an additional supervision for unlabeled data.
3.2.4 Overall framework
The total loss function  $\mathcal{L}$  of our method is:
$$
\mathcal {L} = \mathcal {L} ^ {S R} + w _ {S C} \mathcal {L} ^ {S C} + w _ {U C} \mathcal {L} ^ {U C} + w _ {U R} \mathcal {L} ^ {U R}, \tag {9}
$$
where  $\mathcal{L}^{SR}$ ,  $\mathcal{L}^{SC}$ ,  $\mathcal{L}^{UC}$ , and  $\mathcal{L}^{UR}$  represent the loss values of supervised regression, supervised contrastive loss, unsupervised contrastive loss, and unsupervised ranking loss.  $w_{SC}$ ,  $w_{UC}$ , and  $w_{UR}$  are the corresponding loss weights.  $\mathcal{L}^{UC}$  and  $\mathcal{L}^{UR}$  are calculated using unlabeled data with the ordinal rankings recovered through spectral seriation. Fig. 2 shows the overall framework.
3.3 Robustness analysis of spectral seriation
Because spectral seriation is based on a loss minimization approach, the algorithm itself is robust to noisy inputs. In this section, we provide theoretical proofs to formally show that spectral seriation is robust to two different types of noise: noisy similarity matrices, and noisy feature representations. We derive approximate bounds to analytically demonstrate that spectral seriation can be a reliable method for recovering ordinal rankings from noisy similarity values of unlabeled samples. Additional derivations are also included in the supplementary materials for more technical readers.
3.3.1 Robustness to noise in the similarity matrix
We make use of matrix perturbation theory to show that spectral seriation ranking is robust to noisy values in similarity matrix  $S'$ . We give an upper bound for the noisy entries of the similarity matrix, and show that when the errors of the similarity matrix are bounded, our spectral ranking algorithm recovers the true ranking, thereby demonstrating the error correcting nature of spectral seriation.
The main result is presented in Theorem 2. We first present two related lemmas to assist with the proof, where Lemma 1 provides the perturbation bounds for eigenvalues of symmetric matrices, and Lemma 2 provides the upper bound of the Fiedler value.
Lemma 1 Let  $\mathbf{A},\mathbf{B}\in \mathbb{C}^{n\times n}$  be Hermitian matrices,  $\lambda (\mathbf{A}) = \{\lambda_i\}$ ,  $\lambda (\mathbf{B}) = \{\mu_i\}$ ,  $\lambda_1\leq \lambda_2\leq \dots \leq \lambda_n$ ,  $\mu_{1}\leq \mu_{2}\leq \dots \leq \mu_{n}$ , then:
$$
| \mu_ {i} - \lambda_ {i} | \leq \| \mathbf {B} - \mathbf {A} \| _ {2}.
$$
Lemma 2  $\lambda$  is the Fiedler value of the Laplacian matrix  $\mathbf{L}$  of the similarity matrix  $S^{\prime}$ , then:
$$
\lambda \leq \frac {n}{n - 1} \min _ {1 \leq i \leq n} \{\mathbf {L} _ {i i} \}.
$$
Theorem 2 For a similarity matrix  $S'$ , suppose the error matrix of it is  $\Delta S'$ . When  $2||\Delta S'|_{F} \leq 1 - \frac{\min_{1 \leq i \leq n} \left\{ \sum_{t \neq i} |S_{it}'| \right\}}{n-1}$ , the seriation obtained by the spectral ranking algorithm using  $S'$  is the same as that obtained by the spectral ranking algorithm using  $S' + \Delta S'$ .
The proof of Theorem 2 relies mainly on the definition of the Fielder vector, Lemma 1 and Lemma 2. First order approximation is used to approximate changes in the Fielder vector after adding noise to the similarity matrix, thus allowing us to obtain upper bounds for the noise level. Detailed derivations are included in the supplementary materials.
3.3.2 Robustness to noise in feature representations
We next consider the case where the feature representation for some given sample  $\tilde{z}_i^\prime$  is noisy. The direct consequence of this is that all entries in rows  $i$  and column  $j$  of  $S^{\prime}$  will be noisy. We can also use a similar approach to show that the spectral seriation algorithm can recover the correct ranking as long as the error values are within an upper bound.
Theorem 3 For a similarity matrix  $S'$ , suppose rows  $i$  and column  $i$  are corrupted due to inadequate feature representations being learnt for sample  $i$ . When  $||\Delta S_{[i, :]}'|_2 - ||\Delta S_{[i, :]}'|_1 + \max_{1 \leq j \leq n} |\Delta S_{ij}'| \leq 1 - \frac{\min_{1 \leq i \leq n} \left\{ \sum_{t \neq i} |S_{it}'|\right\}}{n-1}$ , the seriation obtained by the spectral ranking algorithm using  $S'$  is the same as that obtained by the spectral ranking algorithm using  $S' + \Delta S'$ .
The proof for Theorem 3 can be obtained following a similar approach to Theorem 2. We include detailed derivations in the supplementary materials.
We evaluate our proposed method using three different types of datasets to demonstrate its effectiveness as a general approach for semi-supervised deep regression. We use a synthetic non-linear dataset for operator learning, a medical imaging dataset for brain age estimation from MRI scans, and a natural image dataset for age estimation from photographs.
4.1 Synthetic dataset for non-linear operator learning
We use the non-linear synthetic dataset generated by Lu et al. in problem 6 of [24] and train a neural network to estimate the operator function. The target is the stochastic partial differential equation:
$$
- \operatorname {d i v} \left(e ^ {b (x; w)} \nabla u (x; w)\right) = f (x), \tag {10}
$$
where  $w$  is stochastic,  $x \in (0,1)$ , and  $e^{b(x;w)}$  is a diffusion coefficient where  $b(x;w)$  follows a random Gaussian process. The Dirichlet boundary conditions are  $u(0) = u(1) = 0$ , and  $f(x) = 10$ . The input data  $\{x\}_{i=1}^{N}$  are outputs generated by  $(x;w)$  and the target label  $\{y\}_{i=1}^{N}$  is the solution of  $u(x;w)$ . More details of the data generation process can be found in problem 6 of [24].