SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we study the standard deviations of the best results on each dataset with 5 different random seeds. Table 9 shows their average prediction errors and standard deviations. These results are aligned with the results we reported in the main paper.
Table 9: Average prediction errors and standard deviations of the best results on each dataset.
AgeDB: RNC(L1)	TUAB: RNC(DLDL-V2)	MPIIFaceGaze: RNC(OR)	SkyFinder: RNC(DLDL-V2)
6.19 ± 0.08	7.00 ± 0.10	5.24 ± 0.13	2.87 ± 0.04
G.3 Impact of Data Augmentation
In this section, we study the impact of data augmentation on RNC. The following ablations are considered:
-  $\mathrm{RNC}(L_1)$  (without augmentation): Data augmentation is removed and  $\mathcal{L}_{\mathrm{RNC}}$  is computed over  $N$  feature embeddings.
-  $\mathrm{RNC}(L_1)$  (one-view augmentation): Data augmentation is only performed once for each data point and  $\mathcal{L}_{\mathrm{RNC}}$  is computed over  $N$  feature embeddings.
-  $\mathrm{RNC}(L_1)$  (two-view augmentation): Data augmentation is performed twice to create a two-view batch and  $\mathcal{L}_{\mathrm{RNC}}$  is computed over  $2N$  feature embeddings.
-  $L_{1}$  (without augmentation): Data augmentation is removed and  $L_{1}$  is computed over  $N$  samples.
-  $L_{1}$  (with augmentation): Data augmentation is performed for each datapoint and  $L_{1}$  is computed over  $N$  samples.
Table 10 reports the results on each dataset, which demonstrate that removing data augmentation will result in a decrease in performance for both  $L_{1}$  and  $\mathrm{RNC}(L_1)$ , and our method outperforms the baseline with or without data augmentation.
Here we further discuss the different roles played by data augmentation in (unsupervised) contrastive learning methods, end-to-end regression learning methods, and our method respectively:
- For (unsupervised) contrastive learning methods, data augmentation is essential for creating the pretext task, which is to distinguish whether the augmented views belong to the same identity or not. Therefore, removing the data augmentation will result in a complete collapse of the model.
- For end-to-end regression learning methods, data augmentation helps the model generalize better and become more robust to unseen variations. However, without augmentation, the model can still perform reasonably well. The improvement brought by the augmentation is often correlated to how much the data augmentation can compensate for the gap between training data and testing data.
- For RNC, data augmentation is also not necessary since our loss contrasts samples according to label distance rather than the identity. Thus, creating augmented views is not crucial to our method. The role of data augmentation for our method is similar to its role in regression learning methods, namely, improving the generalization ability. The experiment results also confirm this: removing data augmentation will lead to a similar drop in performance for both  $\mathrm{RNC}(L_1)$  and  $L_{1}$ .
Table 10: Impact of data augmentation. Data augmentation is not essential for RNC's superior performance.
Method	Augmentation	AgeDB	TUAB	MPIIFaceGaze	SkyFinder
L1	×	9.53	10.24	6.93	3.14
RNC(L1)	×	8.96	9.88	6.31	2.97
L1	✓	6.63	7.46	5.97	2.95
RNC(L1)	One-view	6.40	7.29	5.46	2.89
RNC(L1)	Two-view	6.14	6.97	5.27	2.86
G.4 Two-Stage Training Scheme for End-to-End Regression Methods
RNC employs a two-stage training scheme, where the encoder is trained with  $\mathcal{L}_{\mathrm{RNC}}$  in the first stage and a predictor is trained using a regression loss on top of the encoder in the second stage. In Table 11, we also train a predictor using  $L_{1}$  loss on top of the encoder learned by the competing end-to-end regression methods and compare with the performance of  $\mathrm{RNC}(L_1)$  on the AgeDB dataset. The results show that the two-stage training scheme does not increase the performance of end-to-end regression methods. In fact, it can even be detrimental to performance. This is because these methods are not designed for representation learning, and their loss functions are calculated w.r.t. the final model predictions. As a result, there is no guarantee for the representation learned by these methods. This finding validates that the benefit of RNC is due to the proposed  $\mathcal{L}_{\mathrm{RNC}}$  and not the training scheme.
Table 11: Two-stage training results for end-to-end regression methods on AgeDB. The two-stage training scheme does not increase or even harms the performance of end-to-end regression methods.
Method	End-to-End	Two-Stage
L1	6.63	6.68
MSE	6.57	6.57
HUBER	6.54	6.63
DEX [36]	7.29	7.42
DLDL-v2 [14]	6.60	7.28
OR [33]	6.40	6.72
CORN [40]	6.72	6.94
RNC(L1)	-	6.14
G.5 Training Efficiency
We compute the average wall-clock running time (in seconds) per training epoch on 8 NVIDIA Titan RTX GPUs for RNC and compare it with SupCON [25] on all four datasets, as shown in Table 12. Results indicate that the training efficiency of RNC is comparable to SupCON.
Table 12: Average wall-clock running time (in seconds) per training epoch for RNC and SUPCON [25] on all datasets. The training efficiency of RNC is comparable to SUPCON.
Method	AgeDB	TUAB	MPIIFaceGaze	SkyFinder
SUPCON [25]	23.1	25.3	69.1	55.6
RNC	26.2	27.3	75.4	61.8
RATIO	1.13	1.08	1.09	1.11