SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use the same architecture and training scheme following [42], which consists of a two-layer fully connected neural network with 100 hidden units. We perform 10 separate training runs on 1,000 samples each and test our model on a set of 100,000 samples. Mean and standard deviation of the 10 runs are reported. We use ordinal entropy [42] as the contrastive loss function  $\mathcal{L}^{SC}$ , which is imposed on the feature layer after L2 normalization. Training is performed with a learning rate of  $1 \times 10^{-3}$ . We use the entire dataset as an input batch and train for 100,000 epochs. Smaller batches of 10 samples are used to calculate  $\mathcal{L}^{SC}$ ,  $\mathcal{L}^{UC}$ , and  $\mathcal{L}^{UR}$  to reduce computation time due to the quadratic scaling of the feature similarity matrix. We set  $w_{SC}, w_{UC}$ , and  $w_{UR}$  to  $1 \times 10^{-3}$  and  $\lambda$  to 2. Implementation is done in PyTorch and training is performed on a single V100 Nvidia GPU.
4.1.1 Comparison with state-of-the-art alternatives
We compare with state-of-the-art semi-supervised deep regression methods to demonstrate the effectiveness of our method, CLSS. We adapt conventional mean-teacher [30] (Mean-teacher) and cross psuedo-label [5] supervision (CPS) semi-supervised learning methods for deep regression using a single output value as the regression prediction. We also compare with the state-of-the-art method, UCVME, proposed in [7]. For reference, we show results using a supervised naive regression method using only labeled data. We show in Table 1 results for different settings, where 1/5, 1/4, 1/3, and 1/2 of available labels are used, and the remaining samples are treated as unlabeled data. Additional implementation details are provided in the supplementary materials section.
Table 1: Comparison with state-of-the-art methods on synthetic non-linear dataset.
MAE↓
Type	Method	1/5 labels	1/4 labels	1/3 labels	1/2 labels
Supervised	Regression	0.098 ± 0.095	0.056 ± 0.016	0.041 ± 0.015	0.032 ± 0.009
Semi-supervised	Mean-teacher [30]	0.080 ± 0.089	0.047 ± 0.021	0.043 ± 0.019	0.029 ± 0.011
CPS [5]	0.057 ± 0.012	0.045 ± 0.016	0.041 ± 0.015	0.028 ± 0.007
UCVME [7]	0.040 ± 0.008	0.033 ± 0.008	0.027 ± 0.007	0.028 ± 0.021
CLSS (Ours)	0.033 ± 0.008	0.027 ± 0.009	0.020 ± 0.007	0.016 ± 0.007
$\mathbf{R}^2 \uparrow$
Type	Method	1/5 labels	1/4 labels	1/3 labels	1/2 labels
Supervised	Regression	66.9% ± 39.4	83.8% ± 7.7	88.5% ± 8.0	90.9% ± 5.1
Semi-supervised	Mean-teacher [30]	69.4% ± 40.1	86.9% ± 8.4	90.7% ± 7.6	92.5% ± 7.2
CPS [5]	84.5% ± 8.8	88.8% ± 8.5	88.5% ± 8.0	93.3% ± 5.3
UCVME [7]	92.2% ± 3.6	94.2% ± 2.8	95.0% ± 3.0	95.6% ± 4.3
CLSS (Ours)	96.4% ± 1.7	97.3% ± 2.4	98.4% ± 1.3	99.3% ± 0.5
We can see that semi-supervised methods generally outperform naive supervised regression. Our proposed method, CLSS, convincingly outperforms alternative methods however and consistently improves  $\mathbf{R}^2$  by  $3 - 4\%$  over the next best alternative across all settings. Computational and memory costs for each method are also provided in Section S3.1.5 of the supplementary materials for reference.
To analyse the effect of different components in our methodology, namely the use of  $\mathcal{L}^{SC}$ ,  $\mathcal{L}^{UC}$ , and  $\mathcal{L}^{UR}$ , we perform training with the loss functions added separately to study their impact. We plot the results in Fig. 3 for easier visualization. We can see that each individual component leads to
significant contributions in improved performance across all settings. This provides further empirical support of the effectiveness of CLSS.
(a) MAE evaluation
(b)  $R^2$  evaluation
Figure 3: Ablation results from adding  $\mathcal{L}^{SC}$ ,  $\mathcal{L}^{UC}$  and  $\mathcal{L}^{UR}$  separately. Using  $\mathcal{L}^{UC}$  and  $\mathcal{L}^{UR}$  to supervise unlabeled samples consistently leads to better predictions.
4.1.3 Quality of ordinal rankings from spectral seriation
CLSS uses the ordinal rankings recovered from  $S'$  to supervise predictions on unlabeled samples. This is based on the observation that the spectral seriation algorithm is robust to errors and more likely to provide accurate supervision. To validate this, we compare with results using rankings derived from predictions  $\hat{y}$  instead of from spectral seriation. We formulate a new loss  $\mathcal{L}^{UCP}$ :
$$
\mathcal {L} ^ {U C P} = \sum_ {i = 1} ^ {| \mathcal {B} |} \ell \left(\mathbf {r} \mathbf {k} \left(\mathcal {S} _ {[ i,: ]} ^ {\prime}\right), \mathbf {r} \mathbf {k} \left(- \left| \hat {y} ^ {\prime} - \hat {y} _ {[ i ]} ^ {\prime} \right|\right)\right), \tag {11}
$$
and train the model using loss function  $\mathcal{L} = \mathcal{L}^{SR} + w_{SC}\mathcal{L}^{SC} + w_{UCP}\mathcal{L}^{UCP}$ . We do not include  $\mathcal{L}^{UR}$  since predictions  $\hat{y}$  are already used to derive the ranking. Results are shown in Table 2.
Table 2: Results using different rankings for supervision
Method	MAE ↓ / R²↑
1/5 labels	1/4 labels	1/3 labels	1/2 labels
Regression+LSC+LUCP	0.039 / 92.4%	0.030 / 95.8%	0.024 / 96.8%	0.016 / 98.5%
CLSS (Ours)	0.033 / 96.4%	0.027 / 97.3%	0.020 / 98.4%	0.016 / 99.3%
We can see that CLSS consistently performs better than using rankings derived from predictions  $\hat{y}^{\prime}$ . This validates the observation that rankings obtained through spectral seriation are more robust, leading to better supervision on unlabeled samples.
4.2 Validation on Brain Age estimation from MRI Scans
Semi-supervised deep regression problems are particularly valuable for medical applications, as real-number medical indicators are common for disease tracking [8, 26, 17], and labeled data are usually costly to obtain [20, 21, 9]. We validate CLSS on the IXI brain MRI dataset for brain age estimation [13]. Brain age estimation involves training a model to learn relevant phenotypes from MRI scans associated with brain health and has important applications in detection of diseases such as Alzheimer's [6, 14]. Typically, a model is trained using brain MRIs from healthy patients to predict chronological age, and then used to make predictions for unhealthy patients to estimate their biological brain age for disease screening [1]. This process assumes that patients used for training have similar chronological and biological ages however, which requires data from healthy individuals to be used. By making use of unlabeled data, we can make training more robust since chronological age is not strictly enforced as the ground truth for these samples. This also reduces the reliance on healthy patients for training.
The IXI dataset consists of 588 MRI brain scans with corresponding ages between 20 and 86 [13]. Out of these, we use 88 samples as the test set and 80 samples as the validation set. A 3D ResNet-18 [32] is used as our model. We use a learning rate of  $1 \times 10^{-3}$  with 0.1 decay every 10 epochs and
train for a total of 30 epochs. We use a batch size of 16 for labeled samples and a batch size of 8 for unlabeled samples. We set  $w_{SC} = 1$ ,  $w_{UC} = 0.05$ ,  $w_{UR} = 0.01$ , and  $\lambda = 2$ , which were chosen based on the validation set. To account for unstable training, all experiments were run separately 10 times using different random seeds. Mean and standard deviation of the 10 runs are reported. We use PyTorch for implementation and train on a single V100 Nvidia GPU.
4.2.1 Comparison with state-of-the-art alternatives and ablation studies
We compare results from using CLSS with alternative state-of-the-art semi-supervised methods and show results in Table 3. We use settings where  $1/5$ ,  $1/4$ ,  $1/3$ , and  $1/2$  of the dataset is treated as labeled data and remaining samples are treated as unlabeled data.
Table 3: Comparison with state-of-the-art methods on IXI brain age dataset. MAE↓
Type	Method	1/5 labels	1/4 labels	1/3 labels	1/2 labels
Supervised	Regression	9.95 ± 1.41	11.93 ± 1.40	11.76 ± 1.75	10.93 ± 1.60
Semi-supervised	Mean-teacher [30]	11.23 ± 2.31	10.27 ± 1.57	10.52 ± 3.12	12.01 ± 2.03
CPS [5]	10.23 ± 1.41	10.27 ± 1.19	9.64 ± 1.27	9.69 ± 1.01
UCVME [7]	9.83 ± 1.32	10.86 ± 1.67	9.65 ± 1.31	10.06 ± 1.19
CLSS (Ours)	9.58 ± 1.48	9.68 ± 1.22	9.72 ± 1.29	9.37 ± 1.17
We can see from the results that our method, CLSS, performs the best under all semi-supervised settings except 1/3 labels. CLSS therefore can be used as an effective way of reducing reliance on labeled data for medical imaging analysis applications. Additional ablation experiments are included in the supplementary materials and demonstrate the importance of each component in CLSS.
4.3 Validation on Age-Estimation from photographs
We also validate our method on a natural image dataset to provide further empirical support. We use the AgeDB-DIR dataset [38] for performing age-estimation from photographs, a common benchmark task for deep regression. Although photographs of people can easily be obtained online, accurate age labels are not always available due to privacy issues. This challenge can be addressed through semi-supervised deep regression methods to reduce reliance on labeled data. AgeDB-DIR consists of 16,488 images of people with ages ranging between 1 and 101. The dataset has fewer tail samples to reflect real-world label imbalance settings. We use the same data splits provided by the dataset for training, validation, and testing.
We use a ResNet50 network [16] pretrained on ImageNet [11] as our deep regression model. We use a learning rate of  $5 \times 10^{-4}$  with decay of 0.1 every 10 epochs and train for 30 epochs. We use a batch size of 32 for labeled samples and a batch size of 8 for unlabeled samples. We set  $w_{SC} = 1$ ,  $w_{UC} = 0.05$ ,  $w_{UR} = 0.01$ , and  $\lambda = 2$ , which were chosen based on the validation set. To account for unstable training, all experiments were run separately 10 times using different random seeds. Mean and standard deviation of the 10 runs are reported. We use PyTorch for implementation and train on a single V100 Nvidia GPU.
4.3.1 Comparison with state-of-the-art alternatives and ablation studies
We compare results from using CLSS with alternative state-of-the-art semi-supervised methods and show results in Table 4. We use settings where 1/30, 1/25, 1/20, and 1/15 of the dataset is treated as labeled data and remaining samples are treated as unlabeled data.
We can see from the results that CLSS outperforms alternative methods for all settings. Overall, the experiments demonstrate that CLSS can also be applied effectively to natural image datasets. We also include ablation experiments in the supplementary materials to highlight the effect of different components.
Table 4: Comparison with state-of-the-art methods on AgeDB-DIR dataset. MAE↓
Type	Method	1/30 labels	1/25 labels	1/20 labels	1/15 labels
Supervised	Regression	10.14 ± 0.25	9.99 ± 0.11	9.10 ± 0.15	8.58 ± 0.10
Semi-supervised	Mean-teacher [30]	10.05 ± 0.29	9.99 ± 0.13	9.05 ± 0.12	8.62 ± 0.09
CPS [5]	9.99 ± 0.12	9.83 ± 0.10	8.99 ± 0.14	8.47 ± 0.08
Ours	9.95 ± 0.18	9.59 ± 0.12	8.88 ± 0.09	8.45 ± 0.11
In this work, we propose a novel approach, CLSS, that allows unlabeled data to be used for contrastive learning on deep regression tasks. We make use of the observation that the feature similarity matrix of unlabeled samples also reflect label distance between samples, and that a robust ordinal ranking can be extracted from the matrix using spectral seriation. We derive theoretical bounds for error values in the similarity matrix for which the derived ordinal ranking remains correct, thereby demonstrating the robustness of our method. We validate our method empirically on a synthetic dataset and two real-world datasets and show that our method can outperform alternative state-of-the-art semi-supervised deep regression methods. Overall, CLSS is a useful technique for improving the performance of semi-supervised deep regression models.
This research is supported by grants from the National Natural Science Foundation of China/HKSAR Research Grants Council Joint Research Scheme under Grant N_HKUST627/20, by the Project of Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone (HZQB-KCZYB-2020083), by the Hong Kong Innovation and Technology Commission (Project no. ITS/030/21 & Project no. PRP/041/22FX), and by Foshan HKUST Projects under FSUST21-HKUST10E and FSUST21-HKUST11E.
[1] Karim Armanious, Sherif Abdulatif, Wenbin Shi, Shashank Salian, Thomas Kustner, Daniel Weiskopf, Tobias Hepp, Sergios Gatidis, and Bin Yang. Age-net: An mri-based iterative framework for brain biological age estimation. IEEE Transactions on Medical Imaging, 40(7):1778-1791, 2021.
[2] Jonathan E Atkins, Erik G Boman, and Bruce Hendrickson. A spectral algorithm for seriation and the consecutive ones problem. SIAM Journal on Computing, 28(1):297-310, 1998.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
[4] X Chen, S Xie, and K He. An empirical study of training self-supervised vision transformers. in 2021 IEEE. In CVF International Conference on Computer Vision (ICCV), pages 9620-9629.
[5] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR, pages 2613-2622, 2021.
[6] James H Cole and Katja Franke. Predicting age using neuroimaging: innovative brain ageing biomarkers. Trends in neurosciences, 40(12):681-690, 2017.
[7] Weihang Dai, Xiaomeng Li, and Kwang-Ting Cheng. Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):7304-7313, Jun. 2023.
[8] Weihang Dai, Xiaomeng Li, Wan Hang Keith Chiu, Michael D Kuo, and Kwang-Ting Cheng. Adaptive contrast for image regression in computer-aided disease assessment. IEEE Transactions on Medical Imaging, 41(5):1255–1268, 2021.
[9] Weihang Dai, Xiaomeng Li, Xinpeng Ding, and Kwang-Ting Cheng. Cyclical self-supervision for semi-supervised ejection fraction prediction from echocardiogram videos. IEEE Transactions on Medical Imaging, 2022.