Title: Semi-Supervised Contrastive Learning for Deep Regression with Ordinal Rankings from Spectral Seriation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Semi-supervised learning

deep regression

contrastive learning
Abstract: Contrastive learning methods can be applied to deep regression by enforcing label distance relationships in feature space. However, these methods are limited to labeled data only unlike for classification, where unlabeled data can be used for contrastive pretraining. In this work, we extend contrastive regression methods to allow unlabeled data to be used in a semi-supervised setting, thereby reducing the reliance on manual annotations. We observe that the feature similarity matrix between unlabeled samples still reflect inter-sample relationships, and that an accurate ordinal relationship can be recovered through spectral seriation algorithms if the level of error is within certain bounds. By using the recovered ordinal relationship for contrastive learning on unlabeled samples, we can allow more data to be used for feature representation learning, thereby achieve more robust results. The ordinal rankings can also be used to supervise predictions on unlabeled samples, which can serve as an additional training signal. We provide theoretical guarantees and empirical support through experiments on different datasets, demonstrating that our method can surpass existing state-of-the-art semi-supervised deep regression methods. To the best of our knowledge, this work is the first to explore using unlabeled data to perform contrastive learning for regression.

Semi-Supervised Contrastive Learning for Deep Regression with Ordinal Rankings from Spectral Seriation

Abstract

Contrastive learning methods can be applied to deep regression by enforcing label distance relationships in feature space. However, these methods are limited to labeled data only unlike for classification, where unlabeled data can be used for contrastive pretraining. In this work, we extend contrastive regression methods to allow unlabeled data to be used in a semi-supervised setting, thereby reducing the reliance on manual annotations. We observe that the feature similarity matrix between unlabeled samples still reflect inter-sample relationships, and that an accurate ordinal relationship can be recovered through spectral seriation algorithms if the level of error is within certain bounds. By using the recovered ordinal relationship for contrastive learning on unlabeled samples, we can allow more data to be used for feature representation learning, thereby achieve more robust results. The ordinal rankings can also be used to supervise predictions on unlabeled samples, which can serve as an additional training signal. We provide theoretical guarantees and empirical support through experiments on different datasets, demonstrating that our method can surpass existing state-of-the-art semi-supervised deep regression methods. To the best of our knowledge, this work is the first to explore using unlabeled data to perform contrastive learning for regression. Code is available at https://github.com/xmed-lab/CLSS .

1 Introduction

Contrastive learning is an effective technique for improving feature representations in deep neural networks (DNNs) [3, 4, 19, 33, 10]. These methods involve identifying positive and negative sample pairs based on feature similarity, where positive pairs are usually defined as augmented samples from the same input or samples from the same class. Early works such as SimCLR [3], MoCo [4], and SupCon [19], showed that pretraining DNNs through contrastive learning can lead to state-of-the-art classification results due to improved feature representations. These techniques have since been extended to segmentation [33, 43], video recognition [10], multi-modality learning [28], as well as deep regression [8, 42, 37] to great effect.

It is possible to perform contrastive learning for classification in an unsupservised manner, which allows large amounts of unlabeled data to be used [3, 4, 28]. Unsupervised contrastive learning is not possible for deep regression tasks however, because in order for feature representations to be effective for regression, they must reflect label distance relationships in feature space [8, 42, 37]. Contrastive regression loss functions ensure that sample pairs with smaller label distances have features that are

(a) Inferring rank fromfeature similarity

(b)Inferringrankfromfeaturesimilaritywithnoise

口

Unlabeled features

Labeled features

y

Unlabeled features

Labeledfeatures

y

Feature

Similarity

←Likely position of

1

口

-0.3

Noisy labeled features

1

口

0.4

unlabeled sample

2

口

-0.0

Likely position of

unlabeled sample

2

口

0.3

Feature

3

口

0.5

Feature

3

口

0.5

？

Cosine

4

口

0.8

？

Cosine

4

口

0.8

Similarity

5

口

0.7

Similarity

5

口

0.7

6

口

0.5

6

口

0.5

(c)Recovering ordinalrankfromfeature similarity of unlabeledsamples

C

Feature

Feature

Similarity

Ordinal Ranking

?

？

MatrixSU

Spectral

口

3

?

Seriation

口

5

？

口

2

Cosine

？

min

Si,;(R; -R;)²

口

6

Similarity

？

R

(ij)

口

4

？

R=Ranking of samples

口

1

more similar, compared to those with greater label distance. Because of this, existing methods can only be used on fully labeled datasets , which can be a limiting factor when annotations are costly to obtain. This is especially true for medical imaging analysis, where deep regression is important for estimating real-number medical indicators but often require expert knowledge for manual annotation [8, 20, 40].

In this work, we extend contrastive learning for deep regression such that unlabeled data can also be used . We consider a semi-supervised setting where only a small portion of the training dataset is labeled, and the majority of samples are unlabeled. We observe that by enforcing supervised contrastive learning on labeled samples, the feature similarity matrix of unlabeled samples will also learn to reflect label distance [8]. Although the unlabeled feature similarity matrix will be inaccurate and noise corrupted, it is still possible to infer the relative ordering of samples if errors are within certain bounds . To give an intuitive example, it is easy to infer the relative ranking of an unlabeled sample based on its pair-wise feature similarity against labeled samples, assuming that features are well-trained (see Fig. 1a). Even in the presence of noisy features, it is still possible to infer the correct ranking if most of the similarity values are reliable (Fig. 1b). Similarly, given a feature similarity matrix for a batch of unlabeled samples, it is still possible to infer their relative ordering even with some degree of noise present (Fig. 1c).

To this end, we propose a novel semi-supervised contrastive regression method by making use of ordinal rankings recovered from the unlabeled feature similarity matrix. We make use of the spectral seriation algorithm proposed by Atkins et al . [2] for ranking recovery, which can then be used to construct a distance matrix for supervising contrastive learning on the unlabeled samples. The spectral seriation algorithm is based on an error-minimization approach and allows for some degree of error correction when extracting rankings from the feature similarity matrix. Thus, the ordinal rankings can also be used to supervise predictions on unlabeled samples, which we demonstrate leads to further improvements for semi-supervised regression. We term our method Contrastive Learning with Spectral Seriation (CLSS). Fig. 2 illustrates our overall framework.

We provide theoretical proofs and empirically show that our method can achieve state-of-the-art performance on multiple datasets. Our method is highly beneficial for critical applications such as medical imaging analysis, where annotations can be expensive to obtain. To the best of our knowledge, CLSS is the first to extend contrastive learning for deep regression to unlabeled data . To summarize, our main contributions are: (1) we propose CLSS, a novel semi-supervised contrastive learning method that is the first to use unlabeled data for contrastive regression; (2) we demonstrate that spectral seriation can be used to extract robust rankings from the feature similarity matrix of

Contrastivelearningfordeepregression

Existingworks

Labeledfeature

Label distance

matrix (yT -y)

similaritymatrix

LSC(

Supervised

Deepencoder

ContrastiveLoss

LabeledData

Labeled

feature

FCLayer

LSR (y,y)

Supervised

LabeledData

Regression Loss

UnlabeledData

Unlabeled

FC Layer

y'

CUR(', R')

Unsupervised

Ranking Loss

UnlabeledData

feature

Spectral Seriation

min

Si,j(Ri-R;)

R'→cUc(

Unsupervised

R

(i))

ContrastiveLoss

Unlabeledfeature

R=Rankingofsamples

Distanceranking

1

similaritymatrix

matrix (R'T - R’)

Ourmethod,CLSS

unlabeled samples for supervision; (3) we demonstrate that our method can outperform existing state-of-the-art alternatives on different deep regression tasks using multiple datasets.

2 Related Work

Contrastive Learning

Contrastive learning for classification involves identifying positive and negative sample pairs based on feature similarity, which allows similar samples to be grouped closer together in feature space [3, 19, 15]. These methods can be performed supervised or unsupervised depending on how positive pairs are defined. SimCLR [3] and MoCo [15] defines positive pairs as augmented samples from the same input data, which means labels are not required. Supervised methods such as SupCon [19] define positive pairs as samples belonging to the same class.

Contrastive learning for deep regression requires features to reflect label distance relationships in feature space. Dai et al . propose AdaCon, which uses adaptive margins in the SupCon loss function to encourage similarity values to reflect pair-wise label distance [8]. Xue et al . [37] find that determining positive and negative pairs based on distance thresholds can be effective for regression tasks. Zhang et al . introduce ordinal entropy, which also encourages features to be spread out depending on their pair-wise label distance [42]. Results from these methods show that improving feature learning is a highly effective way to improve deep regression model performance. Because label information is required for these existing contrastive regression loss functions, they cannot be used with unlabeled data however. Our method, CLSS, is the first to extend contrastive learning for regression to allow unlabeled data to also be used for training.

Semi-Supervised Regression

Semi-supervised learning allows unlabeled data to be used with labeled data for model training. It is an effective way of reducing reliance on manual annotations and is particularly valuable for applications where labeled data is costly to obtain [9, 22, 23, 39, 21]. Semi-supervised classification methods are well studied [29, 41], but semi-supervised deep regression problems receive significantly less attention [7]. Early works such as COREG [44] propose using co-trained KNN neural networks to enforce consistency between two models. Consistency based methods have also been explored in [36, 34]. Iterative approaches for generating pseudo-labels [12], deep kernel learning methods [25, 35, 18], and graph based methods [31] have been proposed for semi-supervised learning in more recent works. A major limitation of these methods is that they are primarily designed for structured tabular data however, and cannot be trained end-to-end with a deep feature extractor for unstructured inputs.

Dai et al . proposed UCVME [7], which enforces consistent uncertainty predictions between cotrained models, and can be used for unstructured inputs such as images and videos. Although this

method achieves good performance, it ignores potential improvements that can be made on the feature level, which is critical to the performance of deep regression models [8, 42]. In this work, we are the first to address the challenge of performing contrastive learning for deep regression by making use of unlabeled data.

3 Method

3.1 Overview

Our novel method allows contrastive learning to be performed with unlabeled data under a semisupervised framework. We denote D := { ( x i , y i ) } N i =1 as the labeled dataset consisting of N samples, where x i is the input data and y i is its corresponding label in I R . We denote D ′ := { x ′ i ′ } N ′ i ′ =1 as the unlabeled dataset consisting of input data only. We denote the feature extractor as f ( · ) and the feature vector as z for labeled data, such that z i = f ( x i ) . Similarly, we denote features for unlabeled data as z ′ i . Contrastive learning is performed on the L2 normalized feature vector, with some methods also performing feature projection onto a lower dimension [8, 42, 3]. We denote the transformed, normalized feature vector for labeled data as ˜ z such that ˜ z i = z i || z i || 2 . We use ˜ z ′ for unlabeled data. The feature similarity matrix for labeled and unlabeled samples are denoted as S and S ′ respectively.

We denote the regression head as g ( · ) . The overall deep regression model can be expressed as:

The model can be trained through supervised regression by minimizing mean squared error (MSE) loss, which we denote as L SR :

Supervised contrastive learning can be performed on the normalized feature vectors for labeled samples ˜ z using different contrastive regression losses, which we denote as L SC . Spectral seriation can then be used to extract the ordinal rankings R ′ from S ′ for supervising contrastive learning and predictions on labeled samples. We describe our method in detail below.

3.2 Using ordinal ranking from spectral seriation for supervision

3.2.1 Spectral seriation for retrieving ordinal ranking from similarity matrix

Contrastive learning for deep regression aims to ensure that feature representations reflect label distance relationships in feature space [8, 42, 37]. If y i and y j are closer together than y i and y k , than the distance between features z i and z j should be closer together than z i and z k . Thus:

Alternatively, because we use L2 normalized feature vectors,

where γ is the cosine similarity function, and γ (˜ z i , ˜ z j ) and γ (˜ z i , ˜ z k ) correspond to entries in similarity matrix S . By performing supervised contrastive learning on labeled samples, we can also expect entries in S ′ to also approximate Eq. 4 with noise, as they have been trained to reflect distance relationships in feature space.

To retrieve the ordinal ranking R ′ of samples in an unlabeled batch, we can make use of the spectral seriation algorithm originally proposed by Atkins et al . [2]. The algorithm is designed to recover an ordinal ranking given a correlation matrix, where higher correlation values indicate closer proximity in ranking. The algorithm is especially useful for when it is easy to determine the similarity between sample pairs but difficult to obtain a direct ordering. Cosine similarity is equivalent to correlation after L2 normalization. Therefore spectral seriation can also be used to recover R ′ from S ′ .

The seriation problem can be formulated as a loss minimizing function:

where S ′ i,j = γ (˜ z i , ˜ z j ) . Since sample pairs closer together in ranking have higher correlation values, minimizing this loss encourages these samples to have R i and R j that are closer together. As per spectral seriation, the solution to R ′ can be derived from the Fiedler vector, as stated in Theorem 1.

Theorem 1 Given similarity matrix S ′ such that S ′ i,j > S ′ i,k for | y i -y j | < | y i -y k | , the ordinal ranking that best satisfies observed S ′ is the ranking of the values in the Fiedler vector of L , where L is the Laplacian of S ′ i,j .

Intuitively, the proof of this theorem can be obtained by approximating discrete rankings R ′ with real-number values r ′ and expressing Eq. 5 in the form of:

r ′ can be solved by computing the Fiedler vector, which is the eigenvector corresponding with the smallest non-zero eigenvalue. The relative rankings of r ′ then give us R ′ . We refer interested readers to [2] for detailed derivations.

3.2.2 Unlabeled contrastive learning with seriation rankings

Spectral seriation gives us rank R ′ of unlabeled samples within a batch. From this, we can obtain rankings on the distances between sample pairs for some anchor sample i . The distance ranking can be used to supervise contrastive learning on unlabeled samples by ensuring that the feature similarity values are consistent with the rankings. We define the unlabeled contrastive learning loss with respect to some subset B as:

where [ i, :] denotes the i th row in the matrix, [ i ] denotes the i th value of a vector, rk denotes the ranking operator, and ℓ is the ranking similarity function. This loss function ensures that the ranking of the feature similarity values with anchor sample i are consistent with that of the distances between derived rankings obtained from seriation. For the differentiable ranking similarity loss function ℓ , we directly use the differential combinatorial solver proposed in [27], which takes an additional parameter λ . On a high level, the function makes use of interpolation methods to allow combinatorial inputs to be differentiable, although detailed explanations can be found in the original work.

3.2.3 Ranking supervision of unlabeled predictions with seriation rankings

Given R ′ for unlabeled samples, we can also perform supervision on the prediction output ˆ y . The spectral seriation algorithm is inherently robust to error, which we show in section 3.3, which means pair-wise distance rankings from seriation are likely to be more accurate than the predicted output. We define the unlabeled prediction ranking loss with respect to some subset B as:

where [ i ] denotes the i th value of a vector. This serves as an additional supervision for unlabeled data.

3.2.4 Overall framework

The total loss function L of our method is:

where L SR , L SC , L UC , and L UR represent the loss values of supervised regression, supervised contrastive loss, unsupervised contrastive loss, and unsupervised ranking loss. w SC , w UC , and w UR are the corresponding loss weights. L UC and L UR are calculated using unlabeled data with the ordinal rankings recovered through spectral seriation. Fig. 2 shows the overall framework.

3.3 Robustness analysis of spectral seriation

Because spectral seriation is based on a loss minimization approach, the algorithm itself is robust to noisy inputs. In this section, we provide theoretical proofs to formally show that spectral seriation is robust to two different types of noise: noisy similarity matrices, and noisy feature representations. We derive approximate bounds to analytically demonstrate that spectral seriation can be a reliable method for recovering ordinal rankings from noisy similarity values of unlabeled samples. Additional derivations are also included in the supplementary materials for more technical readers.

3.3.1 Robustness to noise in the similarity matrix

We make use of matrix perturbation theory to show that spectral seriation ranking is robust to noisy values in similarity matrix S ′ . We give an upper bound for the noisy entries of the similarity matrix, and show that when the errors of the similarity matrix are bounded, our spectral ranking algorithm recovers the true ranking, thereby demonstrating the error correcting nature of spectral seriation.

The main result is presented in Theorem 2. We first present two related lemmas to assist with the proof, where Lemma 1 provides the perturbation bounds for eigenvalues of symmetric matrices, and Lemma 2 provides the upper bound of the Fiedler value.

Lemma 1 Let A , B ∈ C n × n be Hermitian matrices, λ ( A ) = { λ i } , λ ( B ) = { µ i } , λ 1 ≤ λ 2 ≤ · · · ≤ λ n , µ 1 ≤ µ 2 ≤ · · · ≤ µ n , then:

Lemma 2 λ is the Fiedler value of the Laplacian matrix L of the similarity matrix S ′ , then:

Theorem 2 For a similarity matrix S ′ , suppose the error matrix of it is ∆ S ′ . When 2 || ∆ S ′ || F ≤ 1 -min 1 ≤ i ≤ n { ∑ t = i | S ′ it | } n -1 , the seriation obtained by the spectral ranking algorithm using S ′ is the same as that obtained by the spectral ranking algorithm using S ′ +∆ S ′ .

̸

The proof of Theorem 2 relies mainly on the definition of the Fielder vector, Lemma 1 and Lemma 2. First order approximation is used to approximate changes in the Fielder vector after adding noise to the similarity matrix, thus allowing us to obtain upper bounds for the noise level. Detailed derivations are included in the supplementary materials.

3.3.2 Robustness to noise in feature representations

We next consider the case where the feature representation for some given sample ˜ z ′ i is noisy. The direct consequence of this is that all entries in rows i and column j of S ′ will be noisy. We can also use a similar approach to show that the spectral seriation algorithm can recover the correct ranking as long as the error values are within an upper bound.

Theorem 3 For a similarity matrix S ′ , suppose rows i and column i are corrupted due to inadequate feature representations being learnt for sample i . When || ∆ S ′ [ i, :] || 2 - || ∆ S ′ [ i, :] || 1 +

̸

max 1 ≤ j ≤ n ∣ ∣ ∆ S ′ ij ∣ ∣ ≤ 1 -min 1 ≤ i ≤ n { ∑ t = i | S ′ it | } n -1 , the seriation obtained by the spectral ranking algorithm using S ′ is the same as that obtained by the spectral ranking algorithm using S ′ +∆ S ′ .

The proof for Theorem 3 can be obtained following a similar approach to Theorem 2. We include detailed derivations in the supplementary materials.

4 Experiments

We evaluate our proposed method using three different types of datasets to demonstrate its effectiveness as a general approach for semi-supervised deep regression. We use a synthetic non-linear dataset for operator learning, a medical imaging dataset for brain age estimation from MRI scans, and a natural image dataset for age estimation from photographs.

4.1 Synthetic dataset for non-linear operator learning

We use the non-linear synthetic dataset generated by Lu et al . in problem 6 of [24] and train a neural network to estimate the operator function. The target is the stochastic partial differential equation:

where w is stochastic, x ∈ (0 , 1) , and e b ( x ; w ) is a diffusion coefficient where b ( x ; w ) follows a random Gaussian process. The Dirichlet boundary conditions are u (0) = u (1) = 0 , and f ( x ) = 10 . The input data { x } N i =1 are outputs generated by ( x ; w ) and the target label { y } N i =1 is the solution of u ( x ; w ) . More details of the data generation process can be found in problem 6 of [24].

We use the same architecture and training scheme following [42], which consists of a two-layer fully connected neural network with 100 hidden units. We perform 10 separate training runs on 1,000 samples each and test our model on a set of 100,000 samples. Mean and standard deviation of the 10 runs are reported. We use ordinal entropy [42] as the contrastive loss function L SC , which is imposed on the feature layer after L2 normalization. Training is performed with a learning rate of 1 × 10 -3 . We use the entire dataset as an input batch and train for 100,000 epochs. Smaller batches of 10 samples are used to calculate L SC , L UC , and L UR to reduce computation time due to the quadratic scaling of the feature similarity matrix. We set w SC , w UC , and w UR to 1 × 10 -3 and λ to 2. Implementation is done in PyTorch and training is performed on a single V100 Nvidia GPU.

4.1.1 Comparison with state-of-the-art alternatives

We compare with state-of-the-art semi-supervised deep regression methods to demonstrate the effectiveness of our method, CLSS. We adapt conventional mean-teacher [30] (Mean-teacher) and cross psuedo-label [5] supervision ( CPS ) semi-supervised learning methods for deep regression using a single output value as the regression prediction. We also compare with the state-of-the-art method, UCVME, proposed in [7]. For reference, we show results using a supervised naïve regression method using only labeled data. We show in Table 1 results for different settings, where 1/5, 1/4, 1/3, and 1/2 of available labels are used, and the remaining samples are treated as unlabeled data. Additional implementation details are provided in the supplementary materials section.

R

2

↑

We can see that semi-supervised methods generally outperform naïve supervised regression. Our proposed method, CLSS, convincingly outperforms alternative methods however and consistently improves R 2 by 3-4% over the next best alternative across all settings. Computational and memory costs for each method are also provided in Section S3.1.5 of the supplementary materials for reference.

4.1.2 Ablation Study

To analyse the effect of different components in our methodology, namely the use of L SC , L UC , and L UR , we perform training with the loss functions added separately to study their impact. We plot the results in Fig. 3 for easier visualization. We can see that each individual component leads to

significant contributions in improved performance across all settings. This provides further empirical support of the effectiveness of CLSS.

(a) MAE evaluation

0.12

Regression

0.10

Regression+LSC+CUC

0.08

CLSS (Ours)

MAE

0.06

0.04

0.02

0.00

10%

20%

30%

40%

50%

60%

Proportionoflabeleddata

1.00

0.95

0.90

0.85

R

0.80

0.75

Regression

0.70

Regression+LSc

0.65

Regression+CsC+cuc

CLSS(Ours)

0.60

10%

20%

30%

40%

50%

60%

Proportionoflabeleddata

(b)

R

2

evaluation

4.1.3 Quality of ordinal rankings from spectral seriation

CLSS uses the ordinal rankings recovered from S ′ to supervise predictions on unlabeled samples. This is based on the observation that the spectral seriation algorithm is robust to errors and more likely to provide accurate supervision. To validate this, we compare with results using rankings derived from predictions ˆ y instead of from spectral seriation. We formulate a new loss L UCP :

and train the model using loss function L = L SR + w SC L SC + w UCP L UCP . We do not include L UR since predictions ˆ y are already used to derive the ranking. Results are shown in Table 2.

We can see that CLSS consistently performs better than using rankings derived from predictions ˆ y ′ . This validates the observation that rankings obtained through spectral seriation are more robust, leading to better supervision on unlabeled samples.

4.2 Validation on Brain Age estimation from MRI Scans

Semi-supervised deep regression problems are particularly valuable for medical applications, as real-number medical indicators are common for disease tracking [8, 26, 17], and labeled data are usually costly to obtain [20, 21, 9]. We validate CLSS on the IXI brain MRI dataset for brain age estimation [13]. Brain age estimation involves training a model to learn relevant phenotypes from MRI scans associated with brain health and has important applications in detection of diseases such as Alzeimer's [6, 14]. Typically, a model is trained using brain MRIs from healthy patients to predict chronological age, and then used to make predictions for unhealthy patients to estimate their biological brain age for disease screening [1]. This process assumes that patients used for training have similar chronological and biological ages however, which requires data from healthy individuals to be used. By making use of unlabeled data, we can make training more robust since chronological age is not strictly enforced as the ground truth for these samples. This also reduces the reliance on healthy patients for training.

The IXI dataset consists of 588 MRI brain scans with corresponding ages between 20 and 86 [13]. Out of these, we use 88 samples as the test set and 80 samples as the validation set. A 3D ResNet-18 [32] is used as our model. We use a learning rate of 1 × 10 -3 with 0.1 decay every 10 epochs and

train for a total of 30 epochs. We use a batch size of 16 for labeled samples and a batch size of 8 for unlabeled samples. We set w SC = 1 , w UC = 0 . 05 , w UR = 0 . 01 , and λ = 2 , which were chosen based on the validation set. To account for unstable training, all experiments were run separately 10 times using different random seeds. Mean and standard deviation of the 10 runs are reported. We use PyTorch for implementation and train on a single V100 Nvidia GPU.

4.2.1 Comparison with state-of-the-art alternatives and ablation studies

We compare results from using CLSS with alternative state-of-the-art semi-supervised methods and show results in Table 3. We use settings where 1/5, 1/4, 1/3, and 1/2 of the dataset is treated as labeled data and remaining samples are treated as unlabeled data.

MAE

↓

We can see from the results that our method, CLSS, performs the best under all semi-supervised settings except 1/3 labels. CLSS therefore can be used as an effective way of reducing reliance on labeled data for medical imaging analysis applications. Additional ablation experiments are included in the supplementary materials and demonstrate the importance of each component in CLSS.

4.3 Validation on Age-Estimation from photographs

We also validate our method on a natural image dataset to provide further empirical support. We use the AgeDB-DIR dataset [38] for performing age-estimation from photographs, a common benchmark task for deep regression. Although photographs of people can easily be obtained online, accurate age labels are not always available due to privacy issues. This challenge can be addressed through semi-supervised deep regression methods to reduce reliance on labeled data. AgeDB-DIR consists of 16,488 images of people with ages ranging between 1 and 101. The dataset has fewer tail samples to reflect real-world label imbalance settings. We use the same data splits provided by the dataset for training, validation, and testing.

We use a ResNet50 network [16] pretrained on ImageNet [11] as our deep regression model. We use a learning rate of 5 × 10 -4 with decay of 0.1 every 10 epochs and train for 30 epochs. We use a batch size of 32 for labeled samples and a batch size of 8 for unlabeled samples. We set w SC = 1 , w UC = 0 . 05 , w UR = 0 . 01 , and λ = 2 , which were chosen based on the validation set. To account for unstable training, all experiments were run separately 10 times using different random seeds. Mean and standard deviation of the 10 runs are reported. We use PyTorch for implementation and train on a single V100 Nvidia GPU.

4.3.1 Comparison with state-of-the-art alternatives and ablation studies

We compare results from using CLSS with alternative state-of-the-art semi-supervised methods and show results in Table 4. We use settings where 1/30, 1/25, 1/20, and 1/15 of the dataset is treated as labeled data and remaining samples are treated as unlabeled data.

We can see from the results that CLSS outperforms alternative methods for all settings. Overall, the experiments demonstrate that CLSS can also be applied effectively to natural image datasets. We also include ablation experiments in the supplementary materials to highlight the effect of different components.

MAE

↓

5 Conclusion

In this work, we propose a novel approach, CLSS, that allows unlabeled data to be used for contrastive learning on deep regression tasks. We make use of the observation that the feature similarity matrix of unlabeled samples also reflect label distance between samples, and that a robust ordinal ranking can be extracted from the matrix using spectral seriation. We derive theoretical bounds for error values in the similarity matrix for which the derived ordinal ranking remains correct, thereby demonstrating the robustness of our method. We validate our method empirically on a synthetic dataset and two realworld datasets and show that our method can outperform alternative state-of-the-art semi-supervised deep regression methods. Overall, CLSS is a useful technique for improving the performance of semi-supervised deep regression models.

References

Karim Armanious, Sherif Abdulatif, Wenbin Shi, Shashank Salian, Thomas Küstner, Daniel Weiskopf, Tobias Hepp, Sergios Gatidis, and Bin Yang. Age-net: An mri-based iterative framework for brain biological age estimation. IEEE Transactions on Medical Imaging , 40(7):1778-1791, 2021.

Jonathan E Atkins, Erik G Boman, and Bruce Hendrickson. A spectral algorithm for seriation and the consecutive ones problem. SIAM Journal on Computing , 28(1):297-310, 1998.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607. PMLR, 2020.

X Chen, S Xie, and K He. An empirical study of training self-supervised vision transformers. in 2021 ieee. In CVF International Conference on Computer Vision (ICCV) , pages 9620-9629.

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR , pages 2613-2622, 2021.

James H Cole and Katja Franke. Predicting age using neuroimaging: innovative brain ageing biomarkers. Trends in neurosciences , 40(12):681-690, 2017.

Weihang Dai, Xiaomeng Li, and Kwang-Ting Cheng. Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. Proceedings of the AAAI Conference on Artificial Intelligence , 37(6):7304-7313, Jun. 2023.

Weihang Dai, Xiaomeng Li, Wan Hang Keith Chiu, Michael D Kuo, and Kwang-Ting Cheng. Adaptive contrast for image regression in computer-aided disease assessment. IEEE Transactions on Medical Imaging , 41(5):1255-1268, 2021.

Weihang Dai, Xiaomeng Li, Xinpeng Ding, and Kwang-Ting Cheng. Cyclical self-supervision for semisupervised ejection fraction prediction from echocardiogram videos. IEEE Transactions on Medical Imaging , 2022.

Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah. Tclr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding , 219:103406, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Nikos Fazakis, Stamatis Karlos, Sotiris Kotsiantis, and Kyriakos Sgarbas. A multi-scheme semi-supervised regression approach. Pattern Recognition Letters , 125:758-765, 2019.

Katja Franke, Gabriel Ziegler, Stefan Klöppel, Christian Gaser, Alzheimer's Disease Neuroimaging Initiative, et al. Estimating the age of healthy subjects from t1-weighted mri scans using kernel methods: exploring the influence of various parameters. Neuroimage , 50(3):883-892, 2010.

Christian Gaser, Katja Franke, Stefan Klöppel, Nikolaos Koutsouleris, Heinrich Sauer, and Alzheimer's Disease Neuroimaging Initiative. Brainage in mild cognitive impaired patients: predicting the conversion to alzheimer's disease. PloS one , 8(6):e67346, 2013.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , pages 9729-9738, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Chen-I Hsieh, Kang Zheng, Chihung Lin, Ling Mei, Le Lu, Weijian Li, Fang-Ping Chen, Yirui Wang, Xiaoyun Zhou, Fakai Wang, et al. Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning. Nature communications , 12(1):5472, 2021.

Neal Jean, Sang Michael Xie, and Stefano Ermon. Semi-supervised deep kernel learning: Regression with unlabeled data by minimizing predictive variance. Advances in Neural Information Processing Systems , 31, 2018.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, and Pheng-Ann Heng. Semi-supervised skin lesion segmentation via transformation consistent self-ensembling model. In BMVC , 2018.

Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Transformationconsistent self-ensembling model for semisupervised medical image segmentation. IEEE Transactions on Neural Networks and Learning Systems , 32(2):523-534, 2020.

Yi Li, Yiduo Yu, Yiwen Zou, Tianqi Xiang, and Xiaomeng Li. Online easy example mining for weaklysupervised gland segmentation from histology images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part IV , pages 578-587. Springer, 2022.

Yiqun Lin, Huifeng Yao, Zezhong Li, Guoyan Zheng, and Xiaomeng Li. Calibrating label distribution for class-imbalanced barely-supervised knee segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part VIII , pages 109-118. Springer, 2022.

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence , 3(3):218-229, 2021.

Ankur Mallick, Chaitanya Dwivedi, Bhavya Kailkhura, Gauri Joshi, and T Yong-Jin Han. Deep kernels with probabilistic embeddings for small-data learning. In Uncertainty in Artificial Intelligence , pages 918-928. PMLR, 2021.

David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. Video-based ai for beat-to-beat assessment of cardiac function. Nature , 580(7802):252-256, 2020.

Marin Vlastelica Poganˇ ci´ c, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations , 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

Mohan Timilsina, Alejandro Figueroa, Mathieu d'Aquin, and Haixuan Yang. Semi-supervised regression using diffusion on graphs. Applied Soft Computing , 104:107188, 2021.

Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 6450-6459, 2018.

Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR , pages 3024-3033, 2021.

Sebastian J Wetzel, Roger G Melko, and Isaac Tamblyn. Twin neural network regression is a semisupervised regression algorithm. arXiv preprint arXiv:2106.06124 , 2021.

Lu Xu, Chen Hu, and Kuizhi Mei. Semi-supervised regression with manifold: A bayesian deep kernel learning approach. Neurocomputing , 497:76-85, 2022.

Shuo Xu, Xin An, Xiaodong Qiao, Lijun Zhu, and Lin Li. Semi-supervised least-squares support vector regression machines. Journal of information & computational science , 8(6):885-892, 2011.

Tengfei Xue, Fan Zhang, Leo R Zekelman, Chaoyi Zhang, Yuqian Chen, Suheyla Cetin-Karayumak, Steve Pieper, William M Wells, Yogesh Rathi, Nikos Makris, et al. A novel supervised contrastive regression framework for prediction of neurocognitive measures using multi-site harmonized diffusion mri tractography. arXiv preprint arXiv:2210.07411 , 2022.

Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning , pages 11842-11851. PMLR, 2021.

Huifeng Yao, Xiaowei Hu, and Xiaomeng Li. Enhancing pseudo label quality for semi-supervised domaingeneralized medical image segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3099-3107, 2022.

Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng. Uncertainty-aware selfensembling model for semi-supervised 3d left atrium segmentation. In MICCAI , pages 605-613. Springer, 2019.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. In The Eleventh International Conference on Learning Representations .

Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10623-10633, 2021.

Zhi-Hua Zhou, Ming Li, et al. Semi-supervised regression with co-training. In IJCAI , volume 5, pages 908-913, 2005.

Peer reviews:
comments: SUMMARY:
This paper tried to deal with semi-supervised deep regression problems via contrastive learning. To fully use the unlabeled data, they let the feature similarity between unlabeled samples be in agreement with the ranks. Therefore, the accurate ordinal relationship can be recovered through spectral seriation algorithms via feature similarity.

STRENGTHS:
1. The paper uses contrastive learning for unlabeled data by fully considering the ordinal relationship.
2. The paper adopted a robust ranking method named spectral seriation and boost the overall contrastive performance.
3.  Real-world application experiments on medical imaging processing demonstrate its effectiveness and significance.

WEAKNESSES:
1. It seems that the paper does not have obvious weaknesses, if some, the main weaknesses are in the model design Section. I do not understand how to optimize equation (7). 

2. Some descriptions need to be more clear. For example:

    Line 3, "only unlike for" -> "only, unlike for"?

    Line 28 said,  "This is not possible...". What does "This" refer to?

   Line 105, what is "Fiedler vector"?

3. Some typos:

    Line 105 "ofL".
    Equation (7), what is the meaning of  $R^{'} - R^{'}_{[i]}$?

QUESTIONS:
1. Why does the proposed method need to rank the supervision of unlabeled predictions with seriation rankings?
2. Have you considered other baseline methods? Like a simple SimClr  method for unlabeled data? 
3. Equation (7), what is the meaning of  $R^{'} - R^{'}_{[i]}$? Could you provide more explanations?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-23

comments: SUMMARY:
This paper extends contrastive regression methods to a semi-supervised setting using unlabeled data. They leverage the feature similarity matrix between these samples to infer ordinal relationships, a process guaranteed robust if the error is within defined bounds.

STRENGTHS:
This paper successfully extends contrastive regression to include unlabeled datasets, marking a significant step in semi-supervised learning. The authors' writing style is commendable clear, succinct, and easy to follow. Notably, the proposed method isn't just empirical; it is firmly grounded in theoretical evidence, which enhances its credibility and potential for practical application.

WEAKNESSES:
* Ablation studies are missing, ref L2,3
* Missing explanations on loss function design, for each component, what the purpose
* Section 2.3 is not clear, ref Q5

QUESTIONS:
* Why "contrastive learning that are only able to use labeled data" ? contrastive learning, e,g, Simclr, MOCO, etc are under unlabeled data
* How to achieve similarity matrix S?
* why did you set the loss function like formulation (9), for each part, would you give intuitive explanations?
* In eq (7,8,9), how 's the ranking function $l()$ look like? 
* For section 2.3, what are the intuitive explanations of Theorem 2,3, what insights can be taken away? what do the theorems trying to prove?

LIMITATIONS:
* Ablation studies on each loss component are missing
* Ablation studies on hyperparameters are missing
* Section 2.3 is not clear enough


I may have misunderstood, if you solve my question, I am willing to increase the score
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a sophisticated method for deep semi-supervised learning using contrastive loss function. The main idea is to estimate the ordinal relations among the unlabeled data samples by using the similarity matrix of the unlabeled data samples and then use these relations to improve the contrastive learning. To this end, the authors utilize the spectral seriation algorithm of [2]. The proposed method is tested on several datasets and the authors report better accuracies compared to the state-of-the-art.\n\nSTRENGTHS:\nThe main strengths of the paper can be summarized as follows:\n1) The paper is generally well-written with the exception of Related Work section. \n2) The proposed method is sound and supported with theoretical arguments. Estimating the ordinal rankings of the unlabeled data via similarity matrix makes sense and it seems the idea is working. I really liked the idea.\n3) The proposed method achieves the state-of-the-art accuracies on all tested datasets.\n\nWEAKNESSES:\nThe main limitations of the paper can be summarized as follows:\n1) Related work section is quite weak and its location in the manuscript is wrong. Please move it to Section 1 after Introduction. Also, there are some important related missing references. I listed some of them below. Especially, the one using graphs and Laplacian matrix is directly related to the proposed method is here since the ordinal rankings are found by using Laplacian matrix in the paper.\n2) Using cosine similarities only is a big limitation since it constraints all the feature space onto the boundary of a hypersphere. Please note that two samples very far from each other in the feature space may collapse to the similar points on the unit hypersphere. In contrastive loss, Euclidean distances are more preferred unless additional tricks are not done in the regression head (e.g., ArcFace, Cosface etc. normalize both features and classifier weights in their classification losses).\n3) I wonder if the authors use the loss function given in (9) directly. I do not see any constraint to enforce the feature samples to lie on the boundary of the unit hypersphere. In that case, using cosine distances is not a good choice.\n4) Please explain how the Fiedlar vector is computed.\n5) There are 3 parameters in the proposed loss function. Please describe how they are set.\nReferences\n[R1] Mohan Timilsina,  Alejandro Figueroa, Mathieu d’Aquin, Haixuan Yang. Semi-supervised regression using diffusion on graphs, Applied Soft Computing Journal, 2021.\n[R2] Jean et al., Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing\nPredictive Variance, Neurips, 2018.\n[R3] Xu et al., Semi-supervised regression with manifold: A Bayesian deep kernel learning approach, Neurocomputing, 2022.\n[R4] Fazakis et al., A multi-scheme semi-supervised regression approach, Pattern Recognition Letters, 2019.\n\nQUESTIONS:\n1) I wonder if the authors use the loss function given in (9) directly. I do not see any constraint to enforce the feature samples to lie on the boundary of the unit hypersphere. In that case, using cosine distances is not a good choice.\n2) Please explain how the Fiedlar vector is computed.\n\nLIMITATIONS:\nThe authors do not address any limitation, but the main limitation is using cosine distances in the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper presents a novel approach towards extending contrastive learning methods for deep regression in a semi-supervised setting. The authors address the challenge of using unlabeled data for contrastive learning in deep regression tasks by applying spectral seriation algorithms to infer the ordinal relationship between unlabeled samples. Empirically, they show that the proposed method improves performance on medical datasets.

STRENGTHS:
- The motivation for the paper is clear, and the idea is novel from what I can tell. 
- The authors provide a comprehensive robustness analysis of their method, with theoretical proofs showing its resilience to different types of noise.
- The paper is well-organized and logically structured, with clear explanations.

WEAKNESSES:
I understand that the motivation for this paper is mainly for medical datasets, but it would be great to test whether the proposed method also works beyond medical setting, as it does seem quite general.

QUESTIONS:
- What is the computational complexity of the proposed method? How does the training and inference time compare to other state-of-the-art methods?
 - How well would the proposed method perform on more diverse and complex datasets, especially beyond the medical setting?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06