Semi-Supervised Contrastive Learning for Deep Regression with Ordinal Rankings from Spectral Seriation
Authors: Weihang Dai
Yao DU
Hanru Bai
Kwang-Ting Cheng
Xiaomeng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Semi-supervised learning
deep regression
contrastive learning
TL;DR: We make use of the spectral seriation algorithm to extract ordinal rankings from the feature similarity matrix of unlabeled data. Rankings can then be used to supervise unlabeled samples.

Abstract: Contrastive learning methods can be applied to deep regression by enforcing label distance relationships in feature space. However, these methods are limited to labeled data only unlike for classification, where unlabeled data can be used for contrastive pretraining. In this work, we extend contrastive regression methods to allow unlabeled data to be used in a semi-supervised setting, thereby reducing the reliance on manual annotations. We observe that the feature similarity matrix between unlabeled samples still reflect inter-sample relationships, and that an accurate ordinal relationship can be recovered through spectral seriation algorithms if the level of error is within certain bounds. By using the recovered ordinal relationship for contrastive learning on unlabeled samples, we can allow more data to be used for feature representation learning, thereby achieve more robust results. The ordinal rankings can also be used to supervise predictions on unlabeled samples, which can serve as an additional training signal. We provide theoretical guarantees and empirical support through experiments on different datasets, demonstrating that our method can surpass existing state-of-the-art semi-supervised deep regression methods. To the best of our knowledge, this work is the first to explore using unlabeled data to perform contrastive learning for regression.

Semi-Supervised Contrastive Learning for Deep Regression with Ordinal Rankings from Spectral Seriation
Weihang Dai', Yao Du', Hanru Bai', Kwang-Ting Cheng', Xiaomeng Li
'The Hong Kong University of Science and Technology
HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Futian, Shenzhen FFudan University eexmli@ust.co
Abstract
Contrastive learning methods can be applied to deep regression by enforeing label distance relationships in feature space. However, these methods are limited to labeled data only unlike for classification, where unlabeled data can be used for contrastive pretraining. In this work, we extend contrastive regression methods to allow unlabeled data to be used in a semi-supervised setting, thereby reducing the reliance manual annotations. We observe that the feature similarity matrix between unlabeled samples still reflect inter-sample relationships, and that an accurate ordinal relationship can be recovered through spectral seriation algorithms if the level of error within certain bounds. By using the recovered ordinal relationship for contrastive learning unlabeled samples, to be used for feature representation learning, thereby achieve more robust results. also be used supervise predictions unlabeled samples,
allow more data
The ordinal rankings which can serve as an additional training signal. We provide theoretical guarantees and empirical support through experiments different datasets, demonstrating that our method can surpass existing state-of-the-art semi-supervised deep regression methods. To the best of knowledge, this work is the first explore using unlabeled data to perform contrastive learning for regression. Code is available at https::/githuut cww xmed- laalllalllssss
Introduction
Contrastive learning is an effective technique for improving feature representations in deep neural
networks (DNNs) 3 4 pairs based on feature similarity, where positive pairs are usually defined as augmented samples from the same input or samples from the same class. Early works such as SimCL. (3), MoCo [4] and SupCon [9], showed that pretraining DNNs through contrastive learning can lead to state-of-the-ar classification results due to improved feature representations. These techniques have since been
19 33 10]. These methods involve identifying positive and negative sample
extended to segmentation 33 vidco recognition [10,, multi-modality learning (28), as well as deep regression
42 37
great effect.
is possible to perform contrastive learning for classification in an unsupservised manner, which allows large amounts of unlabeled data to be used 3
28 Unsupervised contrastive learning is not
possible for deep regression tasks however, because order for feature representations to be effective for regression, they must reflect label distance relationships in feature space [8] 42 37). Contrastive regression loss functions ensure that sample pairs with smaller label distances have features that are
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Inferring rank from feature similarity
(b) Inferring rank from feature similarity with noise
Intiwoeted featunes Lobeled featuures
y
Feature Similarity
intabaled teaturm:
y
Feature
Similarity
pinviimn of uni untadieedd bamplle
-0.3
Noky abulet
0.4
2
-0.0
Libely protioenal
sample
C.3
y
Fealure
3
0.5
Feature
0.5
Casine
0.8
Simillanit
5
0.7
Similarity
0.7
0.5
0.5
(c)
Recovering ordinal rank from feature simllarity unlabeled samples
y
Feature
Foature
Similarity Matrix
Ordinal Ranking
Spectral Seriation
0
2
Costne
min (.))
S. R,)"
Similarity
R- Ranking of samples
Figure 1: Obtaining rankings from feature similarity. (a) Given labeled samples with well trained features, the ranking of an unlabeled sample can be easily inferred to be between the two with the highest similarity values. (b) The correct ranking can still be inferred with noisy features, as long as the noise levels are within certain thresholds. (c) Given feature similarity matrix of unlabeled samples, the correct ranking can still be obtained if noise levels are within certain bounds.
more similar, compared to those with greater label distance. Because of this, existing methods can only be used on fully labeled datasets, which can limiting factor when annotations are costly to obtain. This is especially true for medical imaging analysis, where deep regression is important for estimating real-number medical indicators but often require expert knowledge for manual annotation 8, 20 40
In this work, we extend contrastive learning for deep regression such that unlabeled data can also be used. We consider a semi-supervised setting where only a small portion of the training dataset labeled, and the majority of samples are unlabeled. We observe that by enforcing supervised contrastive learning on labeled samples, the feature similarity matrix of unlabeled samples will also learn to reflect label distance [8) Although the unlabeled feature similarity matrix will be inaccurate and noise corrupted, it is still possible to infer the relative ordering of samples if errors are within certain bounds. To give an intuitive example, it is easy to infer the relative ranking of an unlabeled sample based on its pair-wise feature similarity against labeled samples, assuming that features are well-trained (see Fig. [T. Even the presence of noisy features, it is still possible infer the correct ranking if most of the similarity values reliable (Fig. [pp). Similarly, given feature similarity matrix for a batch of unlabeled samples, is still possible to infer their relative ordering even with some degree of noise present (Fig. me). (e).
To this end, we propose novel semi-supervised contrastive regression method by making use of ordinal rankings recovered from the unlabeled feature similarity matrix. We make use of the spectral seriation algorithm proposed by Atkins er al. 2 for ranking recovery, which can then be used to construct distance matrix for supervising contrastive learning the unlabeled samples. The spectral seriation algorithm based on an error-minimization approach and allows for some degree of error correction when extracting rankings from the feature similarity matrix. Thus, the ordinal rankings can also be used to supervise predictions on unlabeled samples, which we demonstrate leads to further improvements for semi-supervised regression. We term our method Contrastive Learning with Spectral Seriation (CLSS). Fig. 2 illustrates our overall framework.
We provide theoretical proofs and empirically show that our method can achieve state-of sfate-of-the-art the-hart performance on multiple datasets. Our method is highly beneficial for critical applications such as medical imaging analysis, where annotations can be expensive to obtain. To the best of our knowledge, CLSS is the first to extend contrustive learning for deep regression to unlabeled data. To summarize, our main contributions are: (1) we propose CLSS, a novel semi remi-supervised contrastive learning method that is the first to use unlabeled data for contrastive regression; (2) we demonstrate that spectral seriation can be used to extract robust rankings from the feature similarity matrix of

---

Contrastive learning for deep regression
Existing works
Labeled
Label distance
similarity matrix
O
>)
SC (
Supervised
Deep encoder
Contrastive Loss
Labeled Data
Labeled
fcature
Layee
9
SR (P.y)
Supervised Regression Loss
Labeled Data
Unlabeled
Unlabeled feature
FC
g'
L (F', R')
Unsupervised Ranking Loss
Unlabeled Data
Spectral Seriation
min (R R))
Unsupervised Contrastive Loss
R'
CC
Unlabeled feature similarity
A- Ranking of samples
Distance ranking matrix (A
Our method, CLSS
Framework for Contrastive Learning with Spectral Seritation (CLSS). Unlike existing works for contrastive learning that only able to use labeled data, make use of spectral seriation to obtain ordinal rankings for unlabeled samples. This can then be used for constrative learning and ranking supervision for unlabeled samples.
Figure 2:
unlabeled samples for supervision; (3) we demonstrate that our method can outperform existing state-of-the-art alternatives different deep regression tasks using multiple datasets.
2
Related Work
Contrastive Learning
Contrastive learning for classification involves identifying positive and negative sample pairs based on feature similarity, which allows similar samples be grouped closer together in feature space 19 [5]. These methods can be performed supervised or unsupervised depending on how positive pairs are defined. SimCLR 3 and MoCo [15] defines positive pairs as augmented samples from the same input data, which means labels are not required. Supervised methods such as SupCon [19 define positive pairs as samples belonging the same class.
Contrastive learning for deep regression requires features to reflect label distance relationships in feature space. Dai et al. propose AdaCon, which uses adaptive margins in the SupCon loss function to encourage similarity values to reflect wiir-wise label distance [8] Xue et al. (37 find that determining positive and negative pairs based distance thresholds can be effective for regression tasks. Zhang el al. introduce ordinal entropy, which also encourages features to be spread out depending on their pair-wise label distance (22 Results from these methods show that improving feature learning is highly effective way to improve deep regression model performance. Because label information is required for these existing contrastive regression loss functions, they cannot be used with unlabeled data however. Our method, CLSS, is the first to extend contrastive learning for regression to allow unlabeled data to also be used for training.
Semi-Supervised Regression
Semi-supervised learning allows unlabeled data to be used with labeled data for model training. an effective way of reducing reliance on manual annotations and particularly valuable for
applications where labeled data is costly to obtain 99 22 23
39 21 Semi-supervised classification
methods are well studied 22, 41) but semi-supervised deep regression problems receive significantly less attention (7]. Early works such COREG [44) propose using co-trained KNN neural networks to enforce consistency between two models. Consistency based methods have also been explored Iterative approaches for generating pscudo-labels [12]. deep kernel learning methods and graph based methods B1 have been proposed for semi-supervised learning in more recent works. A major limitation of these methods is that they are primarily designed for structured tabular data however, and cannot trained end-to-end with a deep feature extractor for unstructured inputs.
in 36
34
25
35
18
Dai al. proposed UCVME 77, which enforces consistent uncertainty predictions between co- trained models, and can be used for unstructured inputs such as images and vidcos. Although this

---

method achieves good performance, it ignores potential improvements that can be made on the feature level, which is critical the performance of deep regression models 42 In this work, we are the first to address the challenge of performing contrastive learning for deep regression by making use of unlabeled data.
3 Method
3.1
Overview
Our novel method allows contrastive learning to be performed with unlabeled data under a semi- supervised framework. We denote
{(r,, 8:)) as the labeled dataset consisting of samples, where 2; is the imput data and yi is its corresponding label in IR. We denote D' { J as the unlabeled dataset consisting of input data only. We denote the feature extractor as f(() and the feature vector as for labeled data, such that 2; fz;)). Similarly, we denote features for unlabeled data as 2/. Contrastive learning is performed on the L.2 normalized feature vector, with some methods also performing feature projection onto a lower dimension [8 42 3 We denote the transformed, normalized feature vector for labeled data such that द 1 We use E' for unlabeled data.
The feature similarity matrix for labeled and unlabeled samples are denoted as S and S' respectively.
We denote the regression head as g. The overall deep regression model can be expressed as:
Y e((()) + +E; e~ N(0,
The model can be trained through supervised regression by minimizing mean squared error (MSE) loss, which we denote as LSR
S
( - -
Supervised contrastive learning can be performed on the normalized feature vectors for labeled samples using different contrastive regression losses, which denote _SC_ Spectral seriation can then be used to extract the ordinal rankings R' from S' for supervising contrastive learning and predictions on labeled samples. We describe method in detail below.
3.2
Using ordinal ranking from spectral seriation for supervision
3.2.1
Spectral seriation for retrieving ordinal ranking from similarity matrix
Contrastive learning for deep regression aims to ensure that feature representations reflect label distance relationships in feature space 8 42 37] If y, and yj are closer together than y and yk. than the distance between features 2; and 2j should closer together than 2; and 2k- Thus:
E žj112 < || Ex for 1y ツyト < lys mkl-
Alternatiely., because we use L2 normalized feature vectors.
7(,, 8,) > ( 3x) for 19: 3リ < 1ys - ykk
where is the cosine similarity function, and 7,, 2,) and y,,, εx) correspond to entries in similarity matrix S. By performing supervised contrastive learning on labeled samples, we can also expect entries in S' to also approximate Eq. 4 with noise, they have been trained to reflect distance relationships in feature space.
To retrieve the ordinal ranking R' of samples in an unlabeled batch, we can make use of the spectral seriation algorithm originally proposed by Atkins el al. (2]. The algorithm is designed to recover an ordinal ranking given correlation matrix, where higher correlation values indicate closer proximity in ranking. The algorithm is especially useful for when it is easy to determine the similarity between sample pairs but difficult to obtain a direct ordering. Cosine similarity is equivalent to correlation after L2 normalization. Therefore spectral seriation can also be used recover Ii' from S'.
The seriation problem can be formulated as a loss minimizing function:
arg min R
Σ St S R2)²
(5)

---

where Si Y2; 2,). Since sample pairs closer together in ranking have higher correlation values, minimizing this loss encourages these samples to have R, and R, that are closer together. As per spectral seriation, the solution to R' can be derived from the Fiedler vector, as stated in Theorem 1.
Theorem Given similarity matrix S' such that Shn > St for 19 Yıl < 1y- ykl, the ordinal ranking that best satisfies observed S' the ranking of the values the Fiedler vector of L, where is the Laplacian of S
Intuitively, the proof of this theorem can be obtained by approximating discrete rankings R' with real-number values r' and expressing Eq. in the form of:
Lr
(6)
r can be solved by computing the Fiedler vector, which is the eigenvector corresponding with the smallest non-zero eigenvalue. The relative rankings of r' then give us R'. We refer interested readers to [2] for detailed derivations.
3.2.2 Unlabeled contrastive learning with seriation rankings
Spectral seriation gives us rank H' of unlabeled samples within a batch. From this, we can obtain rankings the distances between sample pairs for some anchor sample i. The distance ranking can be used to supervise contrastive learning on unlabeled samples by ensuring that the feature similarity values are consistent with the rankings. We define the unlabeled contrastive learning loss with respect to some subset as:
B LUC Σ E (rk(S) rk(-IR' R(gg);
where :] denotes the ith row in matrix, [X] denotes the ith value vector, rk denotes the ranking operator, and Ć is the ranking similarity function. This loss function ensures that the ranking of the feature similarity values with anchor sample i are consistent with that of the distances between derived rankings obtained from seriation. For the differentiable ranking similarity loss function E, we directly use the differential combinatorial solver proposed in (27). which takes an additional parameter X. On high level, the function makes use of interpolation methods to allow combinatorial inputs to be differentiable, although detailed explanations can be found in the original work.
3.2.3 Ranking supervision of unlabeled predictions with seriation rankings
Given R' for unlabeled samples, we can also perform supervision on the prediction output ğ. The spectral seriation algorithm is inherently robust to error, which we show in section 3.3 which means pair-wise distance rankings from seriation are likely be more accurate than the predicted output. We define the unlabeled prediction ranking loss with respect to some subset B as:
CUR =
Erkk i/gl), -k(-|R2 Rígl): A);
(8)
where [i] denotes the ith value of vector. This serves as an additional supervision for unlabeled data.
3.2.4 Overall framework
The total loss function our method
C = LSR + SCC + wUCL + wonCUR
(9)
where LSR SS LCC and C represent the loss values of supervised regression, supervised contrastive loss, unsupervised contrastive loss, and unsupervised ranking loss. WSC, WVc, and wyn are the corresponding loss weights. CUC and C are calculated using unlabeled data with the ordinal rankings recovered through spectral seriation. Fig. 2 shows the overall framework.
5

---

3.3 Robustness analysis of spectral seriation
Because spectral seriation based on loss minimization approach, the algorithm itself is robust to noisy inputs. In this section, we provide theoretical proofs to formally show that spectral seriation is robust two different types of noise: noisy similarity matrices, and noisy feature representations. We derive approximate bounds to analytically demonstrate that spectral seriation can be a reliable method for recovering ordinal rankings from noisy similarity values of unlabeled samples. Additional derivations are also included in the supplementary materials for more technical readers.
3.3.1 Robustness noise in the similarity matrix
We make use of matrix perturbation theory to show that spectral seriation ranking robust noisy values in similarity matrix S'. We give upper bound for the noisy entries of the similarity matrix, and show that when the errors of the similarity matrix are bounded, our spectral ranking algorithm recovers the true ranking, thereby demonstrating the error correcting nature of spectral seriation.
The main result is presented in Theorem 2. We first present two related lemmas assist with the proof, where Lemma provides the perturbation bounds for eigenvalues of symmetric matrices, and Lemma 2 provides the upper bound of the Fiedler value.
Lemma Let A, B E Cax" be Hermitian matrices, X(A) (A.) A(B) {13 A1 d2 S An. 11 S 42 S ... S pms then:
A < B 1|2
Lemma À is the Fiedler value of the Laplacian matrix of the similarity matrix S', then:
A minisign (La).
Theorem For a similarity matrix S', suppose the error matrix of AS' When 2||A8*|| S 1 minisisn {E.z, 18(1]] the seriation obtained by the spectral ranking algorithm using S' is the same that obtained by the spectral ranking algorithm using S' + AS'.
The proof of Theorem 2 relies mainly on the definition of the Fielder vector, Lemma and Lemma 2. First order approximation is used approximate changes the Fielder vector after adding noise to the similarity matrix, thus allowing us to obtain upper bounds for the noise level. Detailed derivations included in the supplementary materials.
3.3.2 Robustness to noise in feature representations
We next consider the case where the feature representation for some given sample ž is noisy. The direct consequence of this is that all entries in rows and column of S' will be noisy. We can also use a similar approach to show that the spectral seriation algorithm can recover the correct ranking as long as the error values are within an upper bound.
Theorem For similarity matrix S', suppose rows and column i are corrupted due in- adequate feature representations being learnt for sample i. When |ASS),,)222 - ||AS -|AS +
maxisisn AS S minisisn (Eees 18ta]]
the seriation obtained by the spectral ranking algorithm using S' is the same as that obtained by the spectral ranking algorithm using S' + AS'.
n-1
The proof for Theorem can be obtained following similar approach Theorem 2. We include detailed derivations in the supplementary materials.
4 Experiments
We evaluate our proposed method using three different types of datasets to demonstrate its effec- tiveness as a general approach for semi-supervised deep regression. We use synthetic non-linear dataset for operator learning, a medical imaging dataset for brain age estimation from MRI seans, and natural image dataset for age estimation from photographs.

---

4.1 Synthetic dataset for non-linear operator learning
We use the non-lineat synthetic dataset generated by Lu et al. in problem 6 of [24 and train a neural network to estimate the operator function. The target is the stochastic partial differential equation:
-div(e ((:w)) vu(a;; w)) = f(x),
(10)
where w is stochastic, x (0. 1), and eb(aww) is a diffusion coefficient where b(z; w) follows
random Gaussian process. The Dirichlet boundary conditions are u(0) x(1) 0. and [(x) = 10.
The input data {ar { are outputs generated by (a;; w) and the target label {v) is the solution of u(:;; w). More details of the data generation process can be found in problem 6 of [24).
We use the same architecture and training scheme following [42], which consists of a two-layer fully connected neural network with 100 hidden units. We perform 10 separate training runs on 1,000 samples each and test our model on set of 100,000 samples. Mean and standard deviation of the 10 runs are reported. We use ordinal entropy 42 as the contrastive loss function LSC, which is imposed on the feature layer after L2 normalization. Training is performed with a learning rate of 1 x 10-3 We use the entire dataset as an input batch and train for 100,000 epochs. Smaller batches of 10 samples are used to calculate L LUC, and L to reduce computation time due to the quadratic scaling of the feature similarity matrix. We set WSCWWUC and WUR to x 10-3 and À to Implementation is done in PyTorch and training performed on a single V100 Nvidia GPU.
4.1.1 Comparison with state-of-the-as alternatives
We compare with state-of-the the- semi-supervised deep regression methods to demonstrate the effectiveness of our method, CLSS. We adapt conventional mean-teacher (30) (Mean-teacher) and cross psuedo-label [5] supervision (CPS) semi-supervised learning methods for deep regression using a single output value as the regression prediction. We also compare with the state-of-thh-ant method, UCVME, proposed in 77 For reference, we show results using a supervised naïve regression method using only labeled data. We show in Table results for different settings, where 1/5, 1/4, 1/3, and 1/2 of available labels are used, and the remaining samples are treated as unlabeled data. Additional implementation details are provided in the supplementary materials section.
Table I: Comparison with state-of-the-art methods on synthetic non- non-linear dataset.
MAE
Type
Method
1/5 labels
1/4 labels
1/3 labels
1/2 labels
Supervised
Regression
0.098 0.095
0.056 0.016
0.041 1 0.015
0.032 0.009
Mean-teacher (3
0.080 1 0.089
0.047 1 0.021
0.043 1 0.019
0.029 1 0.011
Semi-
CPS [5]
0.057 0.012
0.045 0.016
0.041 1 0.015
0.028 0.007
supervised
UCVME
0.040 + 0.008
0.033 + 0.008
0.027 1 0.007
0.028
+ 0.021
CLSS (Ours)
0.033 0.008
0.027 0.009
0.020 1 0.007
0.016 0.007
R t
Type
Method
1/5 labels
1/4 labels
1/3 labels
1/2 labels
Supervised
Regression
66.9% 39.4
83.8% 1 7.7
88.5% 1 8.0
90.9% 1 5.1
Mean-tcacher 30
69.4% 40.1
86.9% + 8.4
90.7% 7.6
92.5% 7.2
Semi-
CPS (5]
84.5% + 8.8
88.8% + 8.5
88.5% + 8.0
93.3% + 5.3
supervised
UCVME
92.2% 3.6
94.2%
+ 2.8
95.0% + 3.0
95.6% 4.3
CLSS (Ours)
96.4% % + 1.7
97.3% %
+ 2.4
98.4% % + 1.3
99.3% + 0.5
We can see that semi-supervised methods generally outperform naïve supervised regression. Our proposed method, CLSS, convincingly outperforms alternative methods however and consistently improves R² by 3-4% over the next best alternative across all settings. Computational and memory costs for each method are also provided in Section 13.1.5 of the supplementary materials for reference.
4.1.2 Ablation Study
To analyse the effect of different components in our methodology, namely the use of _ CUR, we perform training with the loss functions added separately to study their impact. We plot the results in Fig. 3 for easier visualization. We can see that each individual component leads to
LUC, and
7

---

significant contributions in improved performance across all settings. This provides further empirical support of the effectiveness of CLSS.
Rigresson
0.30
lingresiint 2
Regresioni 2
0.90
0.08
CL55 (Ours)
0.85
0.06
0.80
0.01
0.75
Regression
0.70
Regression- gM
0.02
0.85
Regression & giggh (158 (Ourg)
0.00
0.50
30% tion labeled
50%
60%
Froportion labeled
(a) MAE evaluation
(b) R evaluation
Figure 3: Ablation results from adding LSC LVC and LUR separately. Using LVC and LUR to supervise unlabeled samples consistently leads to better predictions.
4.1.3 Quality of ordinal rankings from spectral seriation
CLSS uses the ordinal rankings recovered from S' to supervise predictions unlabeled samples. This is based on the observation that the spectral seriation algorithm is robust to errors and more likely to provide accurate supervision. To validate this, compare with results using rankings derived from predictions ò instead of from spectral seriation. We formulate a new loss LDCP,
[8]
LUCP = = e (rk(S) rk(- -fa
(11)
and train the model using loss function L L + wsc WUCPC We do not include CUR since predictions i are already used to derive the ranking. Results are shown in Table
Table 2: Results using different rankings for supervision
MAE // R
Method
1/5 labels
1/4 labels
1/3 labels
1/2 labels
Regression++"
SC
+ CUCP
0.039 / 92.4%
0.030 95.8%
0.024 19688%
0.016 / 98.5%
CLSSS (Ours)
0.033 / 96.4%
0.027 97.3%
0.020 / 98.4%
0.016 99.3%
We can see that CLSS consistently performs better than using rankings derived from predictions j' This validates the observation that rankings obtained through spectral seriation more robust, leading to better supervision on unlabeled samples.
4.2 Validation on Brain Age estimation from MRI Scans
Semi-supervised deep regression problems are particularly valuable for medical applications, as real-number medical indicators are common for disease tracking 88 26 [7]. and labeled data are usually costly to obtain 20 21 19. We validate CLSS on the IXI brain MRI dataset for brain age estimation [13. Brain age estimation involves training a model learn relevant phenotypes from MRI scans associated with brain health and has important applications in detection of diseases such as Alzeimer's 's 14]. Typically, model is trained using brain MRIs from healthy patients to predict chronological age, and then used to make predictions for unhealthy patients to estimate their biological brain age for disease screening [ This process assumes that patients used for training have similar chronological and biological ages however, which requires data from healthy individuals to be used. By making use of unlabeled data, we can make training more robust since chronological age is not strictly enforced as the ground truth for these samples. This also reduces the reliance healthy patients for training.
The IXI dataset consists of 588 MRI brain scans with corresponding ages between 20 and 86 13
Out of these, we use 88 samples as the test set and 80 samples as the validation set. A 3D ResNet-18 32 is used as model. We use learning rate of x 10-3 with 0.1 decay every 10 epochs and
8

---

train for a total of 30 epochs. We use batch size of 16 for labeled samples and batch size of 8 for unlabeled samples. We set wsc 1, Wye 0.05, wy 0.01, and 2. which were chosen based on the validation set. To account for unstable training, all experiments were run separately 10 times using different random seeds. Mean and standard deviation of the 10 runs are reported. We use PyTorch for implementation and train on single V100 Nvidia GPU.
4.2.1 Comparison with state-of-the-art alternatives and ablation studies
We compare results from using CLSS with alternative tate-of-the-aa art sepi-super methods and show results in 3 We use settings where 1/5, 1/4, 1/3, and 1/2 of the dataset is treated as labeled data and remaining samples are treated as unlabeled data.
Table 3: Comparison with state-of-the-art methods on IXI brain age dataset.
MAE
Type
Method
1/5 labels
1/4 labels
1/3 labels
1/2 labels
Supervised
Regression
9.95 + 1.41
11.93 + 1.40
11.76 + 1.75
10.93 + 1.60
Mean-teacher [30]
11.23 2.31
10.27 = 1.57
10.52 3.12
12.01 # 2.03
Semi-
CPS [5]
10.23 + 1.41
10.27 + 1.19
9.64 + 1.27
9.69 + 1.01
supervised
UCVME
9.83 = 1.32
10.86 + 1.67
9.65 + 1.31
10.06 + 1.19
CLSS (Ours)
9.58 + 1.48
9.68 + 1.22
9.72 + 1.29
9.37 + 1.17
We can see from the results that our method, CLSS, performs the best under all semi-supervised settings except 1/3 labels. CLSS therefore can be used as an effective way of reducing reliance on labeled data for medical imaging analysis applications. Additional ablation experiments are included in the supplementary materials and demonstrate the importance of each component in CLSS.
4.3 Validation on Age-Estimation from photographs
We also validate our method on natural image dataset to provide further empirical support. We use the AgeDB-DIR dataset (38] for performing age-estimation from photographs, a common benehmark task for deep regression. Although photographs of people can easily be obtained online, accurate age labels are not always available due to privacy issues. This challenge can be addressed through semi-supervised deep regression methods to reduce reliance on labeled data. AgeDB-DIn consists of 16,488 images of people with ages ranging between 1 and 101. The dataset has fewer tail samples to reflect real-world label imbalance settings. We use the same data splits provided by the dataset for training, validation, and testing.
We use a ResNe150 network [16] pretrained on ImageNet ||| as our deep regression model. We use a learning rate of 5 x 10-4 with decay of 0.1 every 10 epochs and train for 30 epochs. We use batch size of 32 for labeled samples and batch size of 8 for unlabeled samples. We set wse = 1, wec 0.05, Wo R = 0.01, and À which were chosen based on the validation set. To account for unstable training, all experiments were run separately 10 times using different random seeds. Mean and standard deviation of the 10 runs are reported. We use PyTorch for implementation and train on a single V100 Nvidia GPU.
4.3.1 Comparison with state-of-the-art alternatives and ablation studies
We compare results from using CLSS with alternative state-of-the-art semi-supervised methods and show results in Table 4 We use settings where 1/30, 1/25, 1/20, and 1/15 of the dataset is treated as labeled data and remaining samples are treated as unlabeled data.
We can see from the results that CLSS outperforms alternative methods for all settings. Overall, the experiments demonstrate that CLSS can also be applied effectively to natural image datasets. We also include ablation experiments in the supplementary materials highlight the effect of different components.

---

Table 4: Comparison with state-of of the-hhr methods on AgeDB-Dil dataset.
MAE₁
Type
Method
1/30 labels
1/25 labels
1/20 labels
1/15 labels
Supervised
Regression
10.14 0.25
9.99 + 0.11
9.10 + 0.15
8.58 + 0.10
Mean-teacher (30
10.05 + 0.29
9.99 + 0.13
9.05 + 0.12
8.62 + 0.09
Semi-
CPS
S
9.99 + 0.12
9.83 + 0.10
8.99 + 0.14
8.47 + 0.08
supervised
Ours
9.95 + 0.18
9.59 0.12
8.88 + 0.09
8.45 + 0.11
Conclusion
In this work, we propose a novel approach, CLSS, that allows unlabeled data to be used for contrastive learning on deep regression tasks. We make use the observation that the feature similarity matrix of unlabeled samples also reflect label distance between samples, and that robust ordinal ranking can be extracted from the matrix using spectral seriation. We derive theoretical bounds for error values in the similarity matrix for which the derived ordinal ranking remains correct, thereby demonstrating the robustness our method. We validate our method empirically on a synthetic dataset and two real- world datasets and show that our method outperform alternative state-of-le state-of-the-arr -upervude deep regression methods. Overall, CLSS is a useful technique for improving the performance of semi-superriieed deep regression models.
6
Acknowledgements
This research is supported by grants from the National Natural Science Foundation of China/HKSAR Research Grants Council Joint Research Scheme under Grant N HKUST627/20, by the Project of Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone (HZQB-KCZ) -CZYB- 2020083), by the Hong Kong Innovation and Technology Commission (Project no. ITS/030/21 & Project no. PRP/041/22FX) and by Foshan HKUST Projects under FSUSIT1-HHUSSTOF and PSUST21-HKUSTIIE 1E.
References
[1] Karim Armanious, Sherif Abdulatif, Wenbin Shi, Shashank Salian, Thomas Küstner, Daniel Weiskopf, Tobias Hepp. Sergios Gatidis, and Bin Yang. Age-net: An mri-based iterative framework for brain biological age estimation. IEEE Transactions Medical Imaging, 40(7):1778-1791, 2021.
[2] Jonathan E Atkins, Erik G Boman, and Bruce Hendrickson. A spectral algorithm for seriation and the consecutive ones problem. SIAM Journal on Computing, 28(1):297-311 1998.
[3] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML, pages 1597-1607. PMLR, 2020.
[4] X Chen, Xie, and K In CVF International Conference on Computer Vision (ICCV), pages 9620-9629.
empirical study training supervue vision transformers. 2021 ieee.
[5] Xiaokang Chen, Yubui Yuan, Gang Zeng, and Jingdong Wang. Semi Semi-supervised semantic segmentation with cross pseudo supervision. CVPR, pages 2613 16222 2021.
[6] James II Cole and Katja Franke. Predicting age using neuroimaging: innovative brain ageing biomarkers. Trends in neurosciences, 40(12):681 690, 2017.
[7] Weihang Dai, Xiaomeng Li, and Kwang-Tine Cheng. Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. Pruceedings of the AAAI Conference on Artificial Intelligence, 37(6)7304-7733, Jun. 2023.
[8] Weihang Dai, Xiaomeng Li. Wan Hang Keith Chiu, Michael Kuo, and Kwang-Ting Cheng. Adaptive contrast for image regression in computer-aided disease assessment. IEEE Transactions on Medical Imaging, 1255:1255-1226 2021.
[9] Weihang Dai, Xiaomeng Li, Xinpeng Ding. and Kwang- Ting Cheng. Cyclical self-supervision for semi- supervised ejection fraction prediction from echocardiogram videos. IEEE Transactions on Medical Imaging. 2022.
10

---

[10] Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve. and Mubarak Shah. Telr: Temporal contrastive learning for video representation. Computer Vision and Image Understandiny, 219:103400,, 2022.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[12] Nikos Fazakis, Stamatis Karlos, Sotiris Kotsiantis, and Kyriakos Sgarbas. A multi-scheme semi-superiised regression approach. Puttern Recognition Letters, 125:758-765, 2019.
[13] Katja Franke, Gabriel Ziegler, Stefan Klöppel. Christian Gaser, Alzheimer's Disease Neuroimaging Initiative, et al. Estimating the age of healthy subjects from t1-weighted mri scans using kernel methods: exploring the influence various parameters. Neuroimage, 50(3):883-892, 2010.
[14] Christian Gaser, Katja Franke, Stefan Klöppel, Nikolaos Koutsouleris, Heinrich Sauer, and Alzheimer's Discase Ncuroimaging Initiative. Brainage in mild cognitive impaired patients: predicting the conversion to alzheimer's 's disease. PloS one, 8(6):e67344, 2013.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[17] Chen-I Hsich, Kang Zheng. Chihung Lin, Ling Mei, Le Lu, Weijian Li, Fang-Ping Chen, Yirui Wang. Xiaoyun Zhou, Fakai Wang, et al. Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning. Nature communications, 12(1))5472, 2021.
[18] Neal Jean, Sang Michael Xie, and Stefano Ermon. Semi-supervised deep kernel learning: Regression with unlabeled data by minimizing predictive variance. Advances in Neural Information Processing Systems, 31, 2018.
[19] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:1 166611 2020.
[20] Xiaomeng Li. Lequan Yu, Hao Chen, Chi-Wing Fu, and Pheng-Ann Heng. Semi-supervised skin lesion segmentation via transformation consistent self-cnsembling model. In BMVC, 2018.
[21] Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Transformation- consistent self-ensembling model for semisupervised medical image segmentation. IEEE Transactions on Newral Networks and Learning Systems, 32(2):523-555) 2020.
[22] Yi Li, Yiduo Yu, Yiwen Zou, Tianqi Xiang, and Xiaomeng Li. Online easy example mining for weakly- supervised gland segmentation from histology images. In Medical Image Computing and Computer Assisted inteventiinnMICCA 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part IV. pages 578 587. Springer, 2022.
[23] Yiqun Lin, Huifeng Yao, Zezhong Li, Guoyan Zheng, and Xiaomeng Li. Calibrating label distribution for class- imbalancmd barely supervised knee segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCAA/ 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part VIII. pages 109-118. 118. Springer, 2022.
[24] Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Kamiaaakis. Learning nonlinear operators via deeponet based on the universal approximation thcorem of operators. Nature machine intelligence, 3(3):218-22) 2021.
[25] Ankur Mallick, Chaitanya Dwivedi, Bhavya Kailkhura, Gauri Joshi, and T Yong-Jin Han. Deep kernels with probabilistic embeddings for small-data learning. In Uncertainty in Artificial Intelligence, pages 918-928. PMIR, 2021.
[26] David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):522-22 256, 2020.
[27] Marin Vlastelica Pogančić, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. Differentiation of blackbox combinatorial solvers. In International Conference Learning Representations, 2020.
11

---

[28] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[29] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang. Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:5966608, 2020.
[30] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight averaged consistency targets improve semi- eeme-vupe deep learning results. Advances in neural information processing systems, 30, 2017.
[31] Mohan Timilsina, Alejandro Figueroa, Mathicu d' Aquin, and Haixuan Yang. Semi-supervised regression using diffusion graphs. Applied Soft Computing, 104:107188, 2021.
[32] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEFE conference Computer Vision and Pottern Recognition, pages 6450-6459, 2018.
[33] Xinlong Wang. Rufeng Zhang. Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self self- superviedd visual pre training. CVPR, pages 3024 0003-303,2220 2021.
[34] Sebastian J Wetzel, Roger G Melko, and Isaac Tamblyn. Twin neural network regression is semi- supervised regression algorithm. arXiv preprint arXiv:2106.06/24 2021.
[35] Lu Xu, Chen Hu, and Kuizhi Mei. Supe-supe regression with manifold: A bayesian deep kernel learning approach. Neurocomputingg 497:76 -55776655220 2022.
[36] Shuo Xu, Xin An, Xiaodong Qiao, Lijun Zhu, and Lin Li. Semi-supervised lquare support vector regression machines. Journal of information computational science, 8(6):885 892, 2011.
[37] Tengfei Xue, Fan Zhang. Leo R Zekelman, Chaoyi Zhang. Yuqian Chen, Suheyla Cetin-Kant Karayumak, Steve Pieper. William M Wells, Yogesh Rathi, Nikos Makris, et al. A novel supervised contrastive regression framework for prediction of neurocognitive measures using multi-site harmonized diffusion mri tractography. arXiv preprint arXiv:2210.0741 1,77111120 2022.
[38] Yuzhe Yang. Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning, pages 11842-11851. PMIR, 2021.
[39] Huifeng Yao, Xiaowei Hu, and Xiaomeng Li. Enhancing pscudo label quality for semi- supi-supe domain- generalized medical image segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages
2022.
[40] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng. Uncertainty- self- ensembling model for semi-supervised 3d left atrium segmentation. In MICCAI, pages 605-613. Springer, 2019.
[41] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances Neural Information Processing Systems, 141111888889999 2021.
[42] Shihao Zhang, Linlin Yang. Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. In The Eleventh International Conference on Learning Representations.
[43] Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradlcy Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10623-10633, 2021.
[44] Zhi-Hua Zhou, Ming Li, et al. Semi-supervised regression with co-training. In IJCAI, volume 5, pages 908-913, 2005.
regressioo
12


Reviews:
[{"review_id": "z1o0LQj2hQ", "reviewer": "Reviewer_jwFg", "summary": "The paper presents a novel approach towards extending contrastive learning methods for deep regression in a semi-supervised setting. The authors address the challenge of using unlabeled data for contrastive learning in deep regression tasks by applying spectral seriation algorithms to infer the ordinal relationship between unlabeled samples. Empirically, they show that the proposed method improves performance on medical datasets.", "questions": "- What is the computational complexity of the proposed method? How does the training and inference time compare to other state-of-the-art methods?\n - How well would the proposed method perform on more diverse and complex datasets, especially beyond the medical setting?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation for the paper is clear, and the idea is novel from what I can tell. \n- The authors provide a comprehensive robustness analysis of their method, with theoretical proofs showing its resilience to different types of noise.\n- The paper is well-organized and logically structured, with clear explanations.", "weaknesses": "I understand that the motivation for this paper is mainly for medical datasets, but it would be great to test whether the proposed method also works beyond medical setting, as it does seem quite general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments! We are glad you found our idea novel and well supported theoretically. We address remaining questions below:\n\\\n&nbsp;\n\n### W1) Application beyond medical settings \n\nSee response to Q2\n\\\n&nbsp;\n\n### Q1) Computational complexity \n\nOur proposed method, CLSS, does not introduce significant computational complexity compared to state-of-the-art methods. CLSS requires calculating eigenvalues and eigenvectors for Laplacian matrix $L$ (Eq. 6), but this can be done efficiently with existing computational tools and algorithms. Test-time inference is also more efficient than state-of-the-art semi-supervised methods because we only require predictions from **one** model, instead of taking the average from **two** co-trained models. \n\nWe show some reference times below for the synthetic dataset used in Section 3.1. We report the time taken in seconds for one iteration of training and inference for different methods. We use the same batch sizes for all methods as stated in the manuscript for fair comparison. \n\n\n| |Training (seconds per iteration)|Testing (seconds per iteration)|\n| ----------- | ----------- | ----------- |\n| Regression | 0.2015 | 0.0013 |\n| Regression + Contrastive Loss | 0.2167 | 0.0012 |\n| Mean-teacher | 0.2145 | 0.0012 |\n| CPS | 0.2022 | 0.0018 |\n| UCVME| 0.2487 | 0.0043 |\n| CLSS (Ours)| 0.2310| 0.0013|\n\n\nOur method CLSS has competitive inference times for testing (0.0013). Training time is also faster than UCVME (0.2310 vs 0.2487), the best performing alternative, since UCVME performs variational inference during training. \n\nCLSS also has smaller model size compared to state-of-the-art semi-supervised approaches. CLSS only uses **one** model, whilst alternative methods rely on **two** co-trained models, thereby doubling the memory required. \n\n| |Number of parameters|\n| ----------- | ----------- |\n| Regression | 34,401 |\n| Regression + Contrastive Loss | 34,401 |\n| Mean-teacher | 68,802 |\n| CPS | 68,802 |\n| UCVME| 69,004 |\n| CLSS (Ours)| 34,401|\n\nOverall, our method is computationally efficient and memory efficient. We will include these details in our revision.\n\\\n&nbsp;\n### Q2) Performance on general datasets\n\nTo clarify, we performed experiments on **three different types of datasets** in our manuscript: \n1. a synthetic dataset for solving partial differential equations (PDEs) through operator learning (Section 3.1)\n2. a medical dataset for brain age estimation from MRI scans (Section 3.2)\n3. a natural image dataset for age estimation from photographs (Section 3.3)\n\nThis is also stated at the start of Section 3. \n\nIn the manuscript, we emphasize the benefits of our method for medical imaging analysis because semi-supervised deep regression is particularly valuable for such realistic problem settings. However, our experiments demonstrate that CLSS **can similarly be applied to different problem types**, such solving PDEs and regression benchmark tasks like age estimation.\n\nWe will make sure that this is stated more explicitly in our revision to avoid confusion.\n\n"}, {"review_id": "6WxR4JoZup", "reviewer": "Reviewer_H1Yh", "summary": "In this paper, the authors propose a sophisticated method for deep semi-supervised learning using contrastive loss function. The main idea is to estimate the ordinal relations among the unlabeled data samples by using the similarity matrix of the unlabeled data samples and then use these relations to improve the contrastive learning. To this end, the authors utilize the spectral seriation algorithm of [2]. The proposed method is tested on several datasets and the authors report better accuracies compared to the state-of-the-art.", "questions": "1) I wonder if the authors use the loss function given in (9) directly. I do not see any constraint to enforce the feature samples to lie on the boundary of the unit hypersphere. In that case, using cosine distances is not a good choice.\n2) Please explain how the Fiedlar vector is computed.\n", "limitations": "The authors do not address any limitation, but the main limitation is using cosine distances in the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strengths of the paper can be summarized as follows:\n1) The paper is generally well-written with the exception of Related Work section. \n2) The proposed method is sound and supported with theoretical arguments. Estimating the ordinal rankings of the unlabeled data via similarity matrix makes sense and it seems the idea is working. I really liked the idea.\n3) The proposed method achieves the state-of-the-art accuracies on all tested datasets.\n", "weaknesses": "The main limitations of the paper can be summarized as follows:\n1) Related work section is quite weak and its location in the manuscript is wrong. Please move it to Section 1 after Introduction. Also, there are some important related missing references. I listed some of them below. Especially, the one using graphs and Laplacian matrix is directly related to the proposed method is here since the ordinal rankings are found by using Laplacian matrix in the paper.\n2) Using cosine similarities only is a big limitation since it constraints all the feature space onto the boundary of a hypersphere. Please note that two samples very far from each other in the feature space may collapse to the similar points on the unit hypersphere. In contrastive loss, Euclidean distances are more preferred unless additional tricks are not done in the regression head (e.g., ArcFace, Cosface etc. normalize both features and classifier weights in their classification losses).\n3) I wonder if the authors use the loss function given in (9) directly. I do not see any constraint to enforce the feature samples to lie on the boundary of the unit hypersphere. In that case, using cosine distances is not a good choice.\n4) Please explain how the Fiedlar vector is computed.\n5) There are 3 parameters in the proposed loss function. Please describe how they are set.\nReferences\n[R1] Mohan Timilsina,  Alejandro Figueroa, Mathieu d’Aquin, Haixuan Yang. Semi-supervised regression using diffusion on graphs, Applied Soft Computing Journal, 2021.\n[R2] Jean et al., Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing\nPredictive Variance, Neurips, 2018.\n[R3] Xu et al., Semi-supervised regression with manifold: A Bayesian deep kernel learning approach, Neurocomputing, 2022.\n[R4] Fazakis et al., A multi-scheme semi-supervised regression approach, Pattern Recognition Letters, 2019.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments! We are glad you liked our idea! We address remaining concerns below:\n\\\n&nbsp;\n\n### W1) Improvements to Related Work section\n\nThank you for your helpful feedback. These will be considered in our revision. \n\\\n&nbsp;\n\n### W2) Use of cosine similarity \n\nIt is true that using cosine similarity limits features to a unit hypersphere after normalization. However, existing state-of-the-art contrastive learning methods for classification (e.g. SimCLR [1], MOCO [2], SupCon [3], etc.) and regression (AdaCon [4], Ordinal Entropy [5]), all make use of L2 normalized features and cosine similarity for contrastive learning. \n\nBecause the focus of this work is on **how to utilize additional signals from unlabeled samples** for semi-supervised contrastive regression, we build on top of these existing contrastive regression approaches. We do not explore improved contrastive learning loss functions that do not make use of cosine similarity in this work. \n\nSpecifically, our method (CLSS) utilizes Ordinal Entropy [5] for contrastive loss, which is implemented after applying L2 normalization on the feature layer (see lines 73, 75, 80, and 184). Regression predictions are obtained using a fully-connected regression layer directly after the features without L2 normalization. \n\\\n&nbsp;\n\n### W3) Loss function in Eq. 9\n\nThe loss function in Eq. 9 is applied directly. To clarify, we do not **explicitly enforce** the features to lie on a unit hypersphere. Instead, we apply contrastive loss to the feature layer **after first applying L2 normalization**. This is also the standard approach used in existing state-of-the-art methods for contrastive learning [1-5]. In our method, we use Ordinal Entropy [5] for contrastive learning, which applies L2 normalization on the feature layer before applying the loss function (see lines 73, 75, 80, and 184). The fully-connected regression layer is directly applied to features without normalization to obtain the regression prediction. We will state this more clearly in our revision. \n\\\n&nbsp;\n\n### W4) Computation of Fiedlar vector\n\nThe Fiedlar vector is defined as the eigenvector corresponding to the smallest non-zero eigenvalue of a matrix [6]. It can be computed through standard eigenvalue decomposition. We will state this more clearly in our revision. \n\\\n&nbsp;\n\n### W5) Parameters for loss function\n\nThe three parameters for the loss function in Eq. 9 ($w_{SC}, w_{UC}, w_{UR}$) are determined by performing a coarse search on the validation set. We include hyperparameter ablation results below for reference. Parameters are individually adjusted whilst keeping the remaining ones at optimum value:\n\n\n**Synthetic dataset, 1/4 labels (Section 3.1)**\n\n\nOptimum at $w_{SC}=w_{UC}=w_{UR}=0.001$; MAE 0.027\n| |MAE|\n|---|---|\n|$w_{SC}=0.01$|0.033|\n|$w_{SC}=0.0001$|0.030|\n|$w_{UC}=0.01$|0.032|\n|$w_{UC}=0.0001$|0.031|\n|$w_{UR}=0.01$|0.052|\n|$w_{UR}=0.0001$|0.030|\n\n\n**Brain age estimation, 1/2 labels (Section 3.2)**\n\n\nOptimum at $w_{SC}=1,w_{UC}=0.05,w_{UR}=0.01$; MAE 9.37\n| |MAE|\n|---|---|\n|$w_{SC}=5$|12.73|\n|$w_{SC}=0.2$|9.42|\n|$w_{UC}=0.5$|10.24|\n|$w_{UC}=0.005$|9.45|\n|$w_{UR}=0.1$|9.49|\n|$w_{UR}=0.001$|10.61|\n\n\n**Age estimation, 1/25 labels (Section 3.3)**\n\n\nOptimum at $w_{SC}=1,w_{UC}=0.05,w_{UR}=0.01$; MAE 9.59\n| |MAE|\n|---|---|\n|$w_{SC}=5$|9.73|\n|$w_{SC}=0.2$|9.91|\n|$w_{UC}=0.5$|10.70|\n|$w_{UC}=0.005$|9.64|\n|$w_{UR}=0.1$|9.97|\n|$w_{UR}=0.001$|9.65|\n\n\\\n&nbsp;\n\\\n&nbsp;\n\n### Q1) Loss function in Eq. 9\n\nSee response to W3.\n\\\n&nbsp;\n\n### Q2) Computation of Fiedlar vector\n\nSee response to W4.\n\\\n&nbsp;\n\n[1] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" International conference on machine learning. PMLR, 2020.\n\n\n[2] He, Kaiming, et al. \"Momentum contrast for unsupervised visual representation learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n\n[3] Khosla, Prannay, et al. \"Supervised contrastive learning.\" Advances in neural information processing systems 33 (2020): 18661-18673.\n\n\n[4] Dai, Weihang, et al. \"Adaptive contrast for image regression in computer-aided disease assessment.\" IEEE Transactions on Medical Imaging 41.5 (2021): 1255-1268.\n\n\n[5] Zhang, Shihao, et al. \"Improving Deep Regression with Ordinal Entropy.\" arXiv preprint arXiv:2301.08915 (2023).\n\n\n[6] Atkins, Jonathan E., Erik G. Boman, and Bruce Hendrickson. \"A spectral algorithm for seriation and the consecutive ones problem.\" SIAM Journal on Computing 28.1 (1998): 297-310.\n\n\n\n\n\n\n\n"}, {"review_id": "bqxII6haFe", "reviewer": "Reviewer_kMK1", "summary": " This paper extends contrastive regression methods to a semi-supervised setting using unlabeled data. They leverage the feature similarity matrix between these samples to infer ordinal relationships, a process guaranteed robust if the error is within defined bounds.", "questions": "* Why \"contrastive learning that are only able to use labeled data\" ? contrastive learning, e,g, Simclr, MOCO, etc are under unlabeled data\n* How to achieve similarity matrix S?\n* why did you set the loss function like formulation (9), for each part, would you give intuitive explanations?\n* In eq (7,8,9), how 's the ranking function $l()$ look like? \n* For section 2.3, what are the intuitive explanations of Theorem 2,3, what insights can be taken away? what do the theorems trying to prove?", "limitations": "* Ablation studies on each loss component are missing\n* Ablation studies on hyperparameters are missing\n* Section 2.3 is not clear enough\n\n\nI may have misunderstood, if you solve my question, I am willing to increase the score\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper successfully extends contrastive regression to include unlabeled datasets, marking a significant step in semi-supervised learning. The authors' writing style is commendable clear, succinct, and easy to follow. Notably, the proposed method isn't just empirical; it is firmly grounded in theoretical evidence, which enhances its credibility and potential for practical application.\n", "weaknesses": "* Ablation studies are missing, ref L2,3\n* Missing explanations on loss function design, for each component, what the purpose\n* Section 2.3 is not clear, ref Q5\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging feedback! We address questions below.\n\\\n&nbsp;\n\n### Q1) Labels for contrastive learning\n\n\nTo clarify, it is possible to perform contrastive learning using unlabeled data **for classification** (e.g. SimCLR, MOCO). \n\n\nIt is **NOT possible** for existing contrastive learning methods to use unlabeled data **for regression**. \n\n\nThis is because regression labels **reflect order and distance relationships between samples** [1-2], but for classification, we only need to distinguish between different classes. Effective feature representations for regression therefore **need to reflect label distance relationships** unlike for classification. \n\nIntuitively, for samples $i,j,k$, if we have $y_i=5$, $y_j=6$, and $y_k=20$, we expect features for sample $i$ should be more similar to sample $j$ compared to sample $k$ since their labels are closer together. To enforce this, **it is necessary to know the labels**. \n\nIn our work, we propose a novel method that **extracts order relationships from unlabeled samples as well** through spectral seriation for contrastive learning. \n\\\n&nbsp;\n\n### Q2) Calculating similarity matrix $S$\n\n\n$S$ is obtained by calculating cosine similarity between features within a batch. \n\n\nWe use $\\tilde{z}$ to denote L2 normalized features. $\\tilde{z}$ has dimensions $B \\times F$ where $B$ is the batch size and $F$ is the feature length. \n\n$S$ is equal to $\\tilde{z} \\tilde{z}^T$ and has dimensions $B \\times B$.\n\\\n&nbsp;\n\n### Q3) Explanation of loss function (Eq. 9)\n\n\nEq. 9: $\\mathcal{L} = \\mathcal{L}^{SR} + w_{SC} \\mathcal{L}^{SC}  + w_{UC} \\mathcal{L}^{UC} + w_{UR} \\mathcal{L}^{UR}$\n\n\n$\\mathcal{L}^{SR}$ is the supervised regression loss (Eq. 2). \n\n\n$\\mathcal{L}^{SC}$ is the supervised contrastive loss for regression. We use Ordinal Entropy [2] in our experiments (line 184). This ensures that the model learns features that are consistent with the distance relationships between sample labels. Pairs with labels closer together will have features that are more similar compared to pairs with labels further apart. \n\n\n$\\mathcal{L}^{UC}$ is the unsupervised contrastive loss for unlabeled samples (Eq. 7). Because labeled features are trained using contrastive learning, the similarity matrix for unlabeled features will also reflect order and distance relationships between them, but with noise. We can use spectral seriation to extract the ordinal ranking between unlabeled samples from this noisy similarity matrix. \n\n\nBy obtaining the ordinal ranking of unlabeled samples, we can infer the relative relationship of similarity values for feature pairs. Intuitively, if sample $i$ is ranked closer to sample $j$ than to sample $k$, then the cosine similarity between feature pairs ${i,j}$ should be larger then ${i,k}$. These relationships are enforced through $\\mathcal{L}^{UC}$, thus allowing us to perform contrastive learning using unlabeled samples. \n\n\n$\\mathcal{L}^{UR}$ is the unsupervised ranking loss, which we impose on predictions of unlabeled samples (Eq. 8). Because spectral seriation is robust to noise, the recovered ordinal rankings should also be useful for supervising prediction outputs. This also enforces consistency in the order of features and predictions. We enforce the same pairwise relationships inferred in $\\mathcal{L}^{UC}$ on predictions using $\\mathcal{L}^{UR}$. \n\\\n&nbsp;\n\n### Q4) Ranking function $\\ell$ in Eq. 7-9\n\n\nWe directly use the differential combinatorial solver proposed in [3] for $\\ell$. On a high level, $\\ell$ uses interpolation methods to allow discrete combinatorial inputs (e.g. rankings), to be differentiable. We do not provide details since it is better explained in the original work. However, we will include an intuitive explanation in our revision. \n\\\n&nbsp;\n\n### Q5) Explanation of Section 2.3\n\nOne advantage of spectral seriation is that it is based on error minimization and is therefore robust to noise. To understand this intuitively, we can imagine a loss surface and its optimum point at the surface minimum. The optimum point will remain the same even if the surface is perturbed, as long as the distortions are within certain limits. \n\n\nTheorems 2 and 3 illustrates this formally through deriving theoretical bounds for noise tolerance. We demonstrate robustness to two different kinds of noise: noise randomly distributed across the entire surface (Theorem 2), and noise from a single poorly learned feature (Theorem 3). \n\\\n&nbsp;\n\n### L1) Loss component ablation\n\n\nComponent ablation results on the synthetic dataset are illustrated in Figure 3. Results for brain age estimation and age estimation from photos are given in supplementary materials Table S2 and S3. \n\\\n&nbsp;\n\n### L2) Hyperparameter ablation\n\nWe perform hyperparameter ablation by adjusting parameters individually and keeping others at optimum:\n\n\n**Synthetic dataset, 1/4 labels (Section 3.1)**\n\n\nOptimum at $w_{SC}=w_{UC}=w_{UR}=0.001$; MAE 0.027\n| |MAE|\n|---|---|\n|$w_{SC}=0.01$|0.033|\n|$w_{SC}=0.0001$|0.030|\n|$w_{UC}=0.01$|0.032|\n|$w_{UC}=0.0001$|0.031|\n|$w_{UR}=0.01$|0.052|\n|$w_{UR}=0.0001$|0.030|\n\n\n**Brain age estimation, 1/2 labels (Section 3.2)**\n\n\nOptimum at $w_{SC}=1,w_{UC}=0.05,w_{UR}=0.01$; MAE 9.37\n| |MAE|\n|---|---|\n|$w_{SC}=5$|12.73|\n|$w_{SC}=0.2$|9.42|\n|$w_{UC}=0.5$|10.24|\n|$w_{UC}=0.005$|9.45|\n|$w_{UR}=0.1$|9.49|\n|$w_{UR}=0.001$|10.61|\n\n\n**Age estimation, 1/25 labels (Section 3.3)**\n\n\nOptimum at $w_{SC}=1,w_{UC}=0.05,w_{UR}=0.01$; MAE 9.59\n| |MAE|\n|---|---|\n|$w_{SC}=5$|9.73|\n|$w_{SC}=0.2$|9.91|\n|$w_{UC}=0.5$|10.70|\n|$w_{UC}=0.005$|9.64|\n|$w_{UR}=0.1$|9.97|\n|$w_{UR}=0.001$|9.65|\n\n### L3) \nSee Q5. \n\\\n&nbsp;\n\n[1] W, Dai et al. \"Adaptive contrast for image regression in computer-aided disease assessment.\" IEEE Transactions on Medical Imaging 41.5 (2021): 1255-1268.\n\n[2] S. Zhang et al. \"Improving Deep Regression with Ordinal Entropy.\" arXiv preprint arXiv:2301.08915 (2023).\n\n[3] M.V. Pogančić, et al. \"Differentiation of blackbox combinatorial solvers.\" ICLR. 2019."}, {"review_id": "jfF0qlNCvN", "reviewer": "Reviewer_w9AQ", "summary": "This paper tried to deal with semi-supervised deep regression problems via contrastive learning. To fully use the unlabeled data, they let the feature similarity between unlabeled samples be in agreement with the ranks. Therefore, the accurate ordinal relationship can be recovered through spectral seriation algorithms via feature similarity.", "questions": "1. Why does the proposed method need to rank the supervision of unlabeled predictions with seriation rankings?\n2. Have you considered other baseline methods? Like a simple SimClr  method for unlabeled data? \n3. Equation (7), what is the meaning of  $R^{'} - R^{'}_{[i]}$? Could you provide more explanations?\n\n", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper uses contrastive learning for unlabeled data by fully considering the ordinal relationship.\n2. The paper adopted a robust ranking method named spectral seriation and boost the overall contrastive performance.\n3.  Real-world application experiments on medical imaging processing demonstrate its effectiveness and significance.", "weaknesses": "1. It seems that the paper does not have obvious weaknesses, if some, the main weaknesses are in the model design Section. I do not understand how to optimize equation (7). \n\n2. Some descriptions need to be more clear. For example:\n\n    Line 3, \"only unlike for\" -> \"only, unlike for\"?\n\n    Line 28 said,  \"This is not possible...\". What does \"This\" refer to?\n\n   Line 105, what is \"Fiedler vector\"?\n\n3. Some typos:\n\n    Line 105 \"ofL\".\n    Equation (7), what is the meaning of  $R^{'} - R^{'}_{[i]}$? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments! We address remaining concerns below.\n\\\n&nbsp;\n\n### W1) Optimizing Eq. 7\n\n\nWe make use of the differentiable combinatorial solver proposed in [1] to optimize Eq. 7. The loss function $\\ell$ enforces predictions for discrete combinatorial values, such as rankings, to follow some ground-truth order. A high level explanation of $\\ell$ is that it makes use of interpolation methods to allow the discrete combinatorial predictions to be differentiable. We do not provide detailed descriptions of this function since we directly use the implementation from [1], and it is better explained in the original work. We will include an intuitive explanation in our revision however. \n\\\n&nbsp;\n\n### W2) Clearer descriptions\n\n>Line 28 said, \"This is not possible...\". What does \"This\" refer to?\n\n\"This\" refers to using unlabeled data only for unsupervised contrastive learning. \n\n>Line 105, what is \"Fiedler vector\"?\n\nThe Fiedler vector is defined as the eigenvector corresponding to the smallest non-zero eigenvalue of a matrix [2]. \n\n\nWe will make the necessary clarifications in the revision. \n\\\n&nbsp;\n\n### W3) Meaning of $R' - R'_{[i]}$\n\n$R' - R'_{[i]}$ is a vector obtained by subtracting every element in the ranking vector $R'$ by the $i$th element of $R'$. \n\n\nFor example, if \n\n\n$R' = [1,5,2,3,4]$, \n\n\n$R' - R'_{[3]}$ is calculated by subtracting every element in $R'$ by the third entry of $R'$, i.e. 2. \n\n\nThis gives us $[-1,3,0,1,2]$. \n\n\nThis is used to supervise the relative relationships between similarity values in matrix $S'$. Intuitively, if the ranking of sample $i$ is closer to sample $j$ compared to sample $k$, then the cosine similarity between samples ${i,j}$ should also be greater than samples ${i,k}$.  $R' - R'_{[i]}$ calculates the ranking difference between samples relative to sample $i$. \n\\\n&nbsp;\n\n### Q1) Supervising unlabeled predictions with seriation rankings\n\n\nWe note that the rankings obtained through spectral seriation is robust to noise and has error correction properties. Theoretical bounds are derived in Theorems 2 and 3. \n\n\nBecause of this, using the recovered rankings to supervise output predictions should have additional benefits compared to without supervision. Furthermore, this also ensures that the feature relationships learnt through contrastive learning are consistent with the predicted outputs on unlabeled samples. \n\n\nWe empirically verify that including this constraint leads to additional improvements through ablation experiments in Figure 3 and Tables S2 and S3 of the supplementary materials.\n\\\n&nbsp;\n\n### Q2) Using SimCLR for unlabeled samples\n\nContrastive learning methods for classification generally do not work well when applied to regression, since they are unable to consider label distance and order relationships. The loss function treats all negative pairs equivalently without regard to its label value, which often results in ineffective features being learnt. This has been verified in existing works such as [3-4]. \n\nWe can also confirm that applying SimCLR to unlabeled samples only also does not work well, as it disrupts features learnt by the contrastive regression loss function on labeled samples. We show results when applied to our three tasks.  \n\n\n**Synthetic dataset, 1/4 labels (Section 3.1)**\n| |MAE|\n|---|---|\n|SimCLR on unlabeled samples|0.059|\n|CLSS (ours) |0.027|\n\n**Brain age estimation, 1/2 labels (Section 3.2)**\n| |MAE|\n|---|---|\n|SimCLR on unlabeled samples|10.95|\n|CLSS (ours) |9.37|\n\n\n**Age estimation, 1/25 labels (Section 3.3)**\n\n\n| |MAE|\n|---|---|\n|SimCLR on unlabeled samples|9.91|\n|CLSS (ours) |9.59|\n\n\n\\\n&nbsp;\n\n### Q3) Meaning of $R' - R'_{[i]}$\n\nSee response to W3.\n\\\n&nbsp;\n\n\n[1] Pogančić, Marin Vlastelica, et al. \"Differentiation of blackbox combinatorial solvers.\" ICLR. 2019.\n\n[2] Atkins, Jonathan E., Erik G. Boman, and Bruce Hendrickson. \"A spectral algorithm for seriation and the consecutive ones problem.\" SIAM Journal on Computing 28.1 (1998): 297-310.\n\n[3] Dai, Weihang, et al. \"Adaptive contrast for image regression in computer-aided disease assessment.\" IEEE Transactions on Medical Imaging 41.5 (2021): 1255-1268.\n\n[4] Zhang, Shihao, et al. \"Improving Deep Regression with Ordinal Entropy.\" arXiv preprint arXiv:2301.08915 (2023).\n"}]

Author rebuttal:
We thank the reviewers for taking the time to provide their thoughtful comments and feedback. We are glad that in general, the reviewers found our method novel and well supported theoretically. We individually address remaining questions and concerns below. These will also be included in our final revision to help strengthen the manuscript. 

Meta-review: Summary: A solid paper that addresses contrastive learning for regression that self-supervises by enforcing rankings of similarity to align with the direct rankings of the cosine similarity and predicted y-labels. The reason why the former helps is detailed in Section 2.3. which shows how seriation can denoise noisy similarity matrices.

The paper is well-written. Some of the experimental results do suggest that the method beats baselines, though many lie in the error bars of previous methods. The rebuttal was on point and addressed the reviewers' questions well.
Decision: Accept (poster)