SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} - \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \frac {\exp (s _ {i , j})}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp (s _ {i , k})} \\ \geq - n _ {i, m} \log \left(\frac {1}{n _ {i , m}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)}\right) = n _ {i, m} \log n _ {i, m}. \tag {5} \\ \end{array}
$$
Thus, by plugging Eqn. (5) into Eqn. (4), we have
$$
\mathcal {L} _ {\mathrm {R N C}} > \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} n _ {i, m} \log n _ {i, m} = L ^ {\star}. \tag {6}
$$
We will show  $\forall \epsilon > 0$ , there is a set of feature embeddings where
$$
\left\{ \begin{array}{l} s _ {i, j} > s _ {i, k} + \gamma \text {i f} d _ {i, j} <   d _ {i, k} \\ s _ {i, j} = s _ {i, k} \text {i f} d _ {i, j} = d _ {i, k} \end{array} \right.
$$
and  $\gamma \coloneqq \log \frac{2N}{\min_{i\in[2N],m\in[M_i]}n_{i,m\epsilon}}$ ,  $\forall i\in [2N],j,k\in [2N]\backslash \{i\}$ , such that  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon$ .
For such a set of feature embeddings,  $\forall i\in [2N],m\in [M_i],j\in \{j\in [2N]\backslash \{i\} \mid d_{i,j} = D_{i,m}\}$
$$
- \log \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} = \log n _ {i, m} \tag {7}
$$
since  $s_{i,k} = s_{i,j}$  for all  $k$  such that  $d_{i,k} = D_{i,m} = d_{i,j}$ , and
$$
\begin{array}{l} \log \left(1 + \frac {\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} > D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}\right) \tag {8} \\ <   \log \left(1 + \frac {2 N \exp (- \gamma)}{n _ {i , m}}\right) <   \frac {2 N \exp (- \gamma)}{n _ {i , m}} \leq \epsilon \\ \end{array}
$$
since  $s_{i,k} - s_{i,j} < -\gamma$  for all  $k$  such that  $d_{i,k} > D_{i,m} = d_{i,j}$  and  $s_{i,k} - s_{i,j} = 0$  for all  $k$  such that  $d_{i,k} = D_{i,m} = d_{i,j}$ .
From Eqn. (4) we have
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {R N C}} = - \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} \\ + \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \left(1 + \frac {\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} > D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}\right), \tag {9} \\ \end{array}
$$
By plugging Eqn. (7) and Eqn. (8) into Eqn. (9) we have
$$
\mathcal {L} _ {\mathrm {R N C}} <   \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} n _ {i, m} \log n _ {i, m} + \epsilon = L ^ {\star} + \epsilon \tag {10}
$$
We will show  $\forall 0 < \delta < 1$ , there is a
$$
\epsilon = \frac {1}{2 N (2 N - 1)} \min  \left(\min  _ {i \in [ 2 N ], m \in [ M _ {i} ]} \log \left(1 + \frac {1}{n _ {i , m} \exp (\delta + \frac {1}{\delta})}\right), 2 \log \frac {1 + \exp (\delta)}{2} - \delta\right) > 0,
$$
such that when  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon$ , the feature embeddings are  $\delta$ -ordered.
We first show that  $|s_{i,j} - s_{i,k}| < \delta$  if  $d_{i,j} = d_{i,k}, \forall i \in [2N]$ ,  $j, k \in [2N] \backslash \{i\}$  when  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon$ . From Eqn. (4) we have
$$
\mathcal {L} _ {\mathrm {R N C}} > - \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)}. \tag {11}
$$
Let  $p_{i,m} := \underset{j\in [2N]\setminus \{i\}}{\arg \min} s_{i,j}$ ,  $q_{i,m} := \underset{j\in [2N]\setminus \{i\}}{\arg \max} s_{i,j}$ ,  $\zeta_{i,m} := s_{i,p_{i,m}}$ ,  $\eta_{i,m} :=$
$s_{i,q_i,m} - s_{i,p_i,m}, \forall i \in [2N], m \in [M_i]$ , by splitting out the maximum term and the minimum term we have
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {R N C}} > - \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \left\{\log \frac {\exp \left(\zeta_ {i , m}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} \right. \\ + \log \frac {\exp \left(\zeta_ {i , m} + \eta_ {i , m}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} + \log \frac {\exp \left(\sum_ {j \in [ 2 N ] \backslash \{i , p _ {i , m} , q _ {i , m} \} , d _ {i , j} = D _ {i , m}} s _ {i , j}\right)}{\left(\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)\right) ^ {n _ {i , m} - 2}} \Bigg \}. \tag {12} \\ \end{array}
$$
Let  $\theta_{i,m} := \frac{1}{n_{i,m} - 2}\sum_{j\in [2N]\setminus \{i,p_{i,m},q_{i,m}\},d_{i,j} = D_{i,m}}\exp (s_{i,j} - \zeta_{i,m})$  we have
$$
- \log \frac {\exp \left(\zeta_ {i , m}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} = \log (1 + \exp \left(\eta_ {i, m}\right) + \left(n _ {i, m} - 2\right) \theta_ {i, m}) \tag {13}
$$
$$
- \log \frac {\exp \left(\zeta_ {i , m} + \eta_ {i , m}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} = \log (1 + \exp (\eta_ {i, m}) + (n _ {i, m} - 2) \theta_ {i, m}) - \eta_ {i, m}. \tag {14}
$$
Then, from Jensen's inequality, we know
$$
\exp \left(\sum_ {j \in [ 2 N ] \backslash \{i, p _ {i, m}, q _ {i, m} \}, d _ {i, j} = D _ {i, m}} s _ {i, j}\right) \leq \left(\frac {1}{n _ {i , m} - 2} \sum_ {j \in [ 2 N ] \backslash \{i, p _ {i, m}, q _ {i, m} \}, d _ {i, j} = D _ {i, m}} \exp \left(s _ {i, j}\right)\right) ^ {n _ {i, m} - 2}, \tag {15}
$$
$$
- \log \frac {\exp \left(\sum_ {j \in [ 2 N ] \backslash \{i , p _ {i , m} , q _ {i , m} \} , d _ {i , j} = D _ {i , m}} s _ {i , j}\right)}{\left(\sum_ {k \in [ 2 N ] \backslash \{i \}, d _ {i , k} = D _ {i , m}} \exp (s _ {i , k})\right) ^ {n _ {i , m} - 2}} \geq (n _ {i, m} - 2) \log (1 + \exp (\eta_ {i, m}) + (n _ {i, m} - 2) \theta_ {i, m}) - (n _ {i, m} - 2) \log (\theta_ {i, m}) \tag {16}
$$
By plugging Eqn. (13), Eqn. (14) and Eqn. (16) into Eqn. (12), we have
$$
\mathcal {L} _ {\mathrm {R N C}} > \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \left(n _ {i, m} \log \left(1 + \exp \left(\eta_ {i, m}\right) + \left(n _ {i, m} - 2\right) \theta_ {i, m}\right) - \eta_ {i, m} - \left(n _ {i, m} - 2\right) \log \left(\theta_ {i, m}\right)\right). \tag {17}
$$
Let  $h(\theta) \coloneqq n_{i,m} \log (1 + \exp (\eta_{i,m}) + (n_{i,m} - 2)\theta) - \eta_{i,m} - (n_{i,m} - 2) \log (\theta)$ . From derivative analysis we know  $h(\theta)$  decreases monotonically when  $\theta \in \left[1, \frac{1 + \exp(\eta_{i,m})}{2}\right]$  and increases monotonically when  $\theta \in \left[\frac{1 + \exp(\eta_{i,m})}{2}, \exp(\eta_{i,m})\right]$ , thus
$$
h (\theta) \geq h \left(\frac {1 + \exp \left(\eta_ {i , m}\right)}{2}\right) = n _ {i, m} \log n _ {i, m} + 2 \log \frac {1 + \exp \left(\eta_ {i , m}\right)}{2} - \eta_ {i, m}. \tag {18}
$$
By plugging Eqn. (18) into Eqn. (17), we have
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {R N C}} > \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \binom {n _ {i, m} \log n _ {i, m} + 2 \log \frac {1 + \exp \left(\eta_ {i , m}\right)}{2} - \eta_ {i, m}} {2 N - M} \tag {19} \\ = L ^ {\star} + \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \left(2 \log \frac {1 + \exp (\eta_ {i , m})}{2} - \eta_ {i, m}\right). \\ \end{array}
$$
Then, since  $\eta_{i,m} \geq 0$ , we have  $2\log \frac{1 + \exp(\eta_{i,m})}{2} - \eta_{i,m} \geq 0$ . Thus,  $\forall i \in [2N]$ ,  $m \in [M_i]$
$$
\mathcal {L} _ {\mathrm {R N C}} > L ^ {\star} + \frac {1}{2 N (2 N - 1)} \left(2 \log \frac {1 + \exp \left(\eta_ {i , m}\right)}{2} - \eta_ {i, m}\right). \tag {20}
$$
If  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon \leq L^{\star} + \frac{1}{2N(2N - 1)}\left(2\log \frac{1 + \exp(\delta)}{2} -\delta\right)$ , then
$$
2 \log \frac {1 + \exp \left(\eta_ {i , m}\right)}{2} - \eta_ {i, m} <   2 \log \frac {1 + \exp (\delta)}{2} - \delta . \tag {21}
$$
Since  $y(x) = 2\log \frac{1 + \exp(x)}{2} - x$  increases monotonically when  $x > 0$ , we have  $\eta_{i,m} < \delta$ . Hence,  $\forall i \in [2N]$ ,  $j, k \in [2N] \setminus \{i\}$ , if  $d_{i,j} = d_{i,k} = D_{i,m}$ ,  $|s_{i,j} - s_{i,k}| \leq \eta_{i,m} < \delta$ .
Next, we show  $s_{i,j} > s_{i,k} + \delta$  if  $d_{i,j} < d_{i,k}$  when  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon$ .
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {R N C}} = - \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)} \\ + \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \left(1 + \frac {\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} > D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}\right), \tag {22} \\ \end{array}
$$
and combining it with Eqn. (5) we have
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {R N C}} \geq L ^ {\star} + \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \left(1 + \frac {\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} > D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}\right) \\ > L ^ {\star} + \frac {1}{2 N (2 N - 1)} \log \left(1 + \frac {\exp \left(s _ {i , k} - s _ {i , j}\right)}{\sum_ {l \in [ 2 N ] \backslash \{i \}, d _ {i , l} = d _ {i , j}} \exp \left(s _ {i , l} - s _ {i , j}\right)}\right), \tag {23} \\ \end{array}
$$
$$
\forall i \in [ 2 N ], j \in [ 2 N ] \backslash \{i \}, k \in \{k \in [ 2 N ] \backslash \{i \} \mid d _ {i, j} <   d _ {i, k} \}.
$$
When  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon$ , we already have  $|s_{i,l} - s_{i,j}| < \delta$ ,  $\forall d_{i,l} = d_{i,j}$ , which derives  $s_{i,l} - s_{i,j} < \delta$  and thus  $\exp(s_{i,l} - s_{i,j}) < \exp(\delta)$ . By putting this into Eqn. (22), we have  $\forall i \in [2N]$ ,  $j \in [2N] \setminus \{i\}$ ,  $k \in \{k \in [2N] \setminus \{i\} \mid d_{i,j} < d_{i,k}\}$ ,
$$
\mathcal {L} _ {\mathrm {R N C}} > L ^ {\star} + \frac {1}{2 N (2 N - 1)} \log \left(1 + \frac {\exp \left(s _ {i , k} - s _ {i , j}\right)}{n _ {i , r _ {i , j}} \exp (\delta)}\right), \tag {24}
$$
where  $r_{i,j} \in [M_i]$  is the index such that  $D_{i,r_{i,j}} = d_{i,j}$ .
Further, given  $\mathcal{L}_{\mathrm{RNC}} < L^{\star} + \epsilon < L^{\star} + \frac{1}{2N(2N - 1)}\log \left(1 + \frac{1}{n_{i,r_{i,j}}\exp(\delta + \frac{1}{\delta})}\right)$ , we have
$$
\log \left(1 + \frac {\exp \left(s _ {i , k} - s _ {i , j}\right)}{n _ {i , r _ {i , j}} \exp (\delta)}\right) <   \log \left(1 + \frac {1}{n _ {i , r _ {i , j}} \exp \left(\delta + \frac {1}{\delta}\right)}\right) \tag {25}
$$
which derives  $s_{i,j} > s_{i,k} + \frac{1}{\delta}, \forall i \in [2N]$ ,  $j \in [2N] \backslash \{i\}$ ,  $k \in \{k \in [2N] \backslash \{i\} \mid d_{i,j} < d_{i,k}\}$ .
Finally,  $\forall i\in [2N],j,k\in [2N]\backslash \{i\}$ $s_{i,j} < s_{i,k} - \frac{1}{\delta}$  if  $d_{i,j} > d_{i,k}$  directly follows from  $s_{i,j}>$ $s_{i,k} + \frac{1}{\delta}$  if  $d_{i,j} < d_{i,k}$
B Additional Theoretical Analysis
In this section, we present an analysis based on Rademacher Complexity [39] to substantiate that  $\delta$ -ordered feature embedding results in a better generalization bound.
A regression learning task can be formulated as finding a hypothesis  $h$  to predict the target  $y$  from the input  $x$ . Suppose there are  $m$  data points in the training set  $S = \{(x_k, y_k)\}_{k=1}^m$ . Let  $\mathcal{H}_1$  be the class of all possible functions from the input space to the target space.
If a  $\delta$ -ordered feature embedding is guaranteed with an encoder  $f$  mapping  $x_{k}$  to  $v_{k}$ , the set of candidate hypotheses can be reduced to all "  $\delta$ -monotonic" functions  $h(x) = g(f(x))$ , which satisfy  $d(g(v_{i}),g(v_{j})) < d(g(v_{i}),g(v_{k}))$  for  $s_{i,j} > s_{i,k} + \frac{1}{\delta}$ ,  $d(g(v_{i}),g(v_{j})) = d(g(v_{i}),g(v_{k}))$  for  $|s_{i,j} - s_{i,k}| < \delta$ , and  $d(g(v_{i}),g(v_{j})) > d(g(v_{i}),g(v_{k}))$  for  $s_{i,j} < s_{i,k} - \frac{1}{\delta}$  for any  $i,j$  and  $k$ . We denote the class of all "  $\delta$ -monotonic" functions by  $\mathcal{H}_2$ . Note that both  $\mathcal{H}_1$  and  $\mathcal{H}_2$  contain the optimal hypothesis  $h^*$ , which satisfies  $\forall x,y,h^{*}(x) = y$ .
Further, for a hypothesis set  $\mathcal{H}_i$ , let  $\mathcal{A}_i = \{(l((x_1,y_1);h)),\dots,l((x_m,y_m);h)):h\in \mathcal{H}_i\}$  be the loss set for each hypothesis in  $\mathcal{H}_i$  with respect to the training set  $\mathcal{S}$ , where  $l$  is the loss function. Let  $c_{i}$  be the upper bound of  $|l((x,y);h))|$  for all  $x,y$  and  $h\in \mathcal{H}_i$ . We introduce the Rademacher Complexity [39] of  $\mathcal{A}_i$ , denoted as  $R(\mathcal{A}_i)$ . Then, the generalization bound based on Rademacher Complexity says that with a high probability (at least  $1 - \epsilon$ ), the gap between the empirical risk (i.e., training error) and the expected risk (i.e., test error) is upper bounded by  $2R(\mathcal{A}_i) + 4c_i\sqrt{\frac{2\ln(4 / \epsilon)}{m}}$ .
Since  $\mathcal{H}_2\subset \mathcal{H}_1$ , we have  $\mathcal{A}_2\subset \mathcal{A}_1$  and  $c_{2}\leq c_{1}$ , and from the monotonicity of Rademacher Complexity we have  $R(\mathcal{A}_2)\leq R(\mathcal{A}_1)$ . Hence, with a  $\delta$ -ordered feature embedding, the upper bound on the gap between the training error and the test error will be reduced, which leads to better regression performance.
Table 7: Visualizations of original and augmented data samples on all datasets.
Dataset	Original	Augmented
AgeDB
TUAB	21 channels
MPIIFaceGaze
SkyFinder
Five real-world datasets are used in the experiments:
- AgeDB [32, 44] is a dataset for predicting age from face images. It contains 16,488 in-the-wild images of celebrities and the corresponding age labels. The age range is between 0 and 101. It is split into a 12,208-image training set, a 2140-image validation set, and a 2140-image test set.
- TUAB [34, 11] is a dataset for brain-age estimation from EEG resting-state signals. The dataset comes from EEG exams at the Temple University Hospital in Philadelphia. Following Engemann et al. [11], we use only the non-pathological subjects, so that we may consider their chronological age as their brain-age label. The dataset includes 1,385 21-channel EEG signals sampled at  $200\mathrm{Hz}$  from individuals whose age ranges from 0 to 95. It is split into a 1,246-subject training set and a 139-subject test set.
- MPIIFaceGaze [51, 52] is a dataset for gaze direction estimation from face images. It contains 213,659 face images collected from 15 participants during their natural everyday laptop use. We subsample and split it into a 33,000-image training set, a 6,000-image validation set, and a 6,000-image test set with no overlapping participants. The gaze direction is described as a 2-dimensional vector with the pitch angle in the first dimension and the yaw angle in the second dimension. The range of the pitch angle is  $-40^{\circ}$  to  $10^{\circ}$  and the range of the yaw angle is  $-45^{\circ}$  to  $45^{\circ}$ .
- SkyFinder [31, 7] is a dataset for temperature prediction from outdoor webcam images. It contains 35,417 images captured by 44 cameras around 11am on each day under a wide range of weather and illumination conditions. The temperature range is  $-20^{\circ}\mathrm{C}$  to  $49^{\circ}\mathrm{C}$ . It is split into a 28,373-image training set, a 3,522-image validation set, and a 3,522-image test set.
- IMDB-WIKI [36, 44] is a large dataset for predicting age from face images, which contains 523,051 celebrity images and the corresponding age labels. The age range is between 0 and 186 (some images are unlabeled). We use this dataset to test our method's resilience to reduced training data, performance on transfer learning, and ability to generalize to unseen targets. We subsample the dataset to create a variable size training set, and keep the validation set and test set unchanged with 11,022 images in each.
D Details of Data Augmentation
Table 7 shows examples of original and augmented data samples on each dataset. The data augmentations used on each dataset are listed below:
- For AgeDB and SkyFinder, random crop and resize (with random horizontal flip), color distortions are used as data augmentation;
- For TUAB, random crop is used as data augmentation;
- For MPIIFaceGaze, random crop and resize (without random horizontal flip), color distortions are used as data augmentation.
E Details of Competing Methods
E.1 End-to-End Regression Methods
We compared with seven end-to-end regression methods:
-  $L_{1}$ , MSE and HUBER have the model directly predict the target value and train the model with an error-based loss function, where  $L_{1}$  uses the mean absolute error, MSE uses the mean squared error and HUBER uses an MSE term when the error is below a threshold and an  $L_{1}$  term otherwise.
- DEX [36] and DLDL-v2 [14] divide the regression range of each label dimension into several bins and learn the probability distribution over the bins. DEX [36] optimizes a cross-entropy loss between the predicted distribution and the one-hot ground-truth labels, while DLDL-v2 [14] jointly optimizes a KL loss between the predicted distribution and a normal distribution centered at the ground-truth value, as well as an  $L_{1}$  loss between the expectation of the predicted distribution and the ground-truth value. During inference, they output the expectation of the predicted distribution for each label dimension.
- OR [33] and CORN [40] design multiple ordered thresholds for each label dimension, and learn a binary classifier for each threshold. OR [33] optimizes a binary cross-entropy loss for each binary classifier to learn whether the target value is larger than each threshold, while CORN [40] learns whether the target value is larger than each threshold conditioning on it is larger than the previous threshold. During inference, they aggregate all binary classification results to produce the final results.
For the classification-based baselines, the regression range is divided into small bins, and each bin is considered as a class. Details for each dataset are as follows:
- For AgeDB, the age range is  $0 \sim 101$  and the bin size is set to 1;
- For TUAB, the brain-age range is  $0 \sim 95$  and the bin size is set to 1;
- For MPIIFaceGaze, the target range is  $-40 \sim 10$  (°) for the pitch angle and  $-45 \sim 45$  (°) for the yaw angle, and the bin size is set to 0.5 for the pitch angle and is set to 1 for the yaw angle;
- For SkyFinder, the temperature range is  $-20 \sim 49$  ( $^\circ\mathrm{C}$ ) and the bin size is set to 1.
E.2 State-of-the-art Representation Learning Methods
We compared with three state-of-the-art representation learning methods:
- SIMCLR [4] is a contrastive learning method that learns representations by aligning positive pairs and repulsing negative pairs. Positive pairs are defined as different augmented views from the same data input, while negative pairs are defined as augmented views from different data inputs.
- DINO [3] is a self-supervised representation learning method using self-distillation. It passes two different augmented views from the same data input to both the student and the teacher networks and maximizes the similarity between the output features of the student network and those of the teacher network. The gradients are propagated only through the student network and the teacher parameters are updated with an exponential moving average of the student parameters.
- SupCon [25] extends SimCLR [4] to the fully-supervised setting, where positive pairs are defined as augmented data samples from the same class and negative pairs are defined as augmented data samples from different classes. To adapt SupCon to the regression task, we follow the standard routine of classification-based regression methods to divide the regression range into small bins and regard each bin as a class.
E.3 State-of-the-art Regression Learning Methods
We also compared with five state-of-the-art regression learning methods:
- LDS+FDS [44] is the state-of-the-art method on the AgeDB dataset. It addresses the data imbalance issue by performing distribution smoothing for both labels and features.
- L2CS-NET [1] is the state-of-the-art method on the MPIIFaceGaze dataset. It regresses each gaze angle separately and applies both a cross-entropy loss and an MSE loss on the predictions.
- LDE [7] is the state-of-the-art method on the SkyFinder dataset. It converts temperature estimation to a classification task, and the class label is encoded by a Gaussian distribution centered at the ground-truth label.
- RANKSIM [17] is a state-of-the-art regression method that proposes a regularization loss to match the sorted list of a given sample's neighbors in the feature space with the sorted list of its neighbors in the label space.
- ORDINAL ENTROPY [50] is a state-of-the-art regression method that proposes a regularization loss to encourage higher-entropy feature spaces while maintaining ordinal relationships.
F Details of Experiment Settings
All experiments are trained using 8 NVIDIA Titan RTX GPUs. We use the SGD optimizer and cosine learning rate annealing [29] for training. The batch size is set to 256. For one-stage methods and encoder training of two-stage methods, we select the best learning rates and weight decays for each dataset by grid search, with a grid of learning rates from  $\{0.01, 0.05, 0.1, 0.2, 0.5, 1.0\}$  and weight decays from  $\{10^{-6}, 10^{-5}, 10^{-4}, 10^{-3}\}$ . For the predictor training of two-stage methods, we adopt the same search setting as above except for adding no weight decay to the search choices of weight decays. For temperature parameter  $\tau$ , we search from  $\{0.1, 0.2, 0.5, 1.0, 2.0, 5.0\}$  and select the best, which is 2.0. We train all one-stage methods and the encoder of two-stage methods for 400 epochs, and the linear regressor of two-stage methods for 100 epochs.
G Additional Experiments and Analyses
G.1 Impact of Model Architectures
In the main paper, we use ResNet-18 as the default encoder backbone for three visual datasets (AgeDB, MPIIFaceGaze and SkyFinder). In this section, we study the impact of backbone architectures on the experiment results. As Table 8 reports, the results of using ResNet-50 as the encoder backbone are consistent with the results using ResNet-18 in Table 2, indicating that our method is compatible with different model architectures.
Table 8: Evaluation results using ResNet-50 as the encoder backbone for visual datasets. The results are consistent with the results using ResNet-18 as the encoder backbone.
Metrics	AgeDB	MPIIFaceGaze	SkyFinder
MAE↓	R2↑	Angular↓	R2↑	MAE↓	R2↑
L1	6.49	0.830	5.74	0.748	2.88	0.863
RNC(L1)	6.10	0.851	5.16	0.819	2.78	0.877
(+0.39)	(+0.021)	(+0.58)	(+0.071)	(+0.10)	(+0.014)
G.2 Standard Deviations of Results