SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Cancer Diagnosis. Given the huge costs of misprediction for high-stakes applications such as cancer diagnosis, instead of a single point prediction it is useful to predict a contiguous set. For instance, prediction set of [stage 2, stage 3] gives a better notion of severity than a non-contiguous set such as [no cancer, stage 3] which might be discordant or a point prediction with low accuracy.
- Dynamic Product Search Filters. Customers new to any e-commerce platform often experience heavy cognitive load in specifying their requirements via search filters (e.g., budget, product dimensions). Identifying a small highly likely set of options based on their typical profile or immediate session history would significantly enhance the usability of the search filters and improve the customer experience.
- Personalised Fit Recommendations. Shopping apparel and shoes at any e-commerce platform is often tedious due to the limited support for fit-based recommendations. Often, customers find that the recommended products do not have options for their size and are forced to use search filters, which need to be repeatedly specified for each query. Additionally, customers also tend to order multiple products in the same size (bracketing) that results in a high return rate and excessive shipping costs for the platform. Automatic identification of the likely size ranges of a customer would improve the accuracy of recommendations and reduce shopping effort as well as return rates.
- Personalised Budget Recommendations, Since budget ranges have a natural ordering, automatic personalisation of product and brand recommendations for customers based on their preferred budget ranges is another area that can leverage COPOC to improve customer satisfaction.
- Abuse Incident Audits. E-commerce abuse incidents are often categorised along severity levels that have a natural ordering. Typically, human auditors are required to audit the abuse incidents, but current models do not often distinguish between a high chance of low severity incident vs. moderate chance of high severity incident. Conformal predictions can help streamline the audit workflows to better focus on the high severity incidents and optimise the overall outcomes both for e-commerce platform and the customers through expedited resolution.
Lemma 1. Given a fitted unimodal model  $\hat{P}_{Y|X}$ , for any test  $\mathbf{x}$  and  $\alpha \in (0,1]$ , prediction sets  $\hat{S}_{D,\alpha}(\mathbf{x})$  constructed using Eqn.6 or 7 has at least one solution which is contiguous (amongst multiple possibilities). When  $\hat{P}_{Y|X}$  is strictly unimodal, all the solutions are contiguous.
Proof: Let  $\hat{S}_{D,\alpha}(\mathbf{x})$  be the prediction set with the shortest span (i.e., difference between highest and lowest included labels) as per Eqn.6 or 7.
Let  $l + 1$  and  $u$  denote the smallest and largest indices of the labels included in  $\hat{S}_{D,\alpha}(\mathbf{x})$  so that the span is given by  $u - l$ .
Assuming  $\hat{S}_{D,\alpha}(\mathbf{x})$  is non-contiguous implies that there exists at least one  $k^{skip}$  such that  $(l + 1) < k^{skip} < u$  and  $c_{k^{skip}} \notin \hat{S}_{D,\alpha}(\mathbf{x})$ . Let  $\hat{p}_k(\mathbf{x}) = \hat{P}_{Y|X}(Y = c_k|X = \mathbf{x})$ . Since  $\hat{p}(\mathbf{x})$  is unimodal, there are two possible scenarios depending on where  $k^{skip}$  relies relative to the mode  $c_{\hat{m}}$  of  $\hat{p}$ :
-  $k^{skip} < \hat{m}$ : In this case, we have  $\hat{p}_{k^{skip}} \geq \hat{p}_{l+1}$  since  $\hat{p}$  is non-decreasing before the mode
-  $k^{skip} \geq \hat{m}$ : In this case, we have  $\hat{p}_{k^{skip}} \geq \hat{p}_u$  since  $\hat{p}$  is non-increasing after the mode
Thus,  $\hat{p}_{k^{skip}}\geq \min (\hat{p}_{l + 1},\hat{p}_u)$
Case 1: LAC- PS Construction follows Eqn. 6: For this case, we have,
$$
\hat {S} _ {D, \alpha} (\mathbf {x}) = \left\{c _ {k} \in \mathcal {C} \mid \hat {p} _ {k} \geq \hat {q} _ {D _ {\text {c a l}}} (\alpha) \right\}, \tag {6}
$$
where  $\hat{q}_{D_{cal}}(\alpha)$  is the bias-adjusted  $(\alpha)^{th}$  quantile of the model score of the true label. Since  $c_{l+1}$  and  $c_u$  are included in  $\hat{S}_{D,\alpha}(\mathbf{x})$ , it follows that both  $\hat{p}_{l+1} \geq \hat{q}_{D_{cal}}(\alpha)$  and  $\hat{p}_u \geq \hat{q}_{D_{cal}}(\alpha)$ .
Since  $\hat{p}_{k^{skip}} \geq \min(\hat{p}_{l+1}, \hat{p}_u)$ , it follows that  $\hat{p}_{k^{skip}} \geq \hat{q}_{D_{cal}}(\alpha)$  as well implying that  $c_{k^{skip}} \in \hat{S}_{D,\alpha}(\mathbf{x})$  which leads to a contradiction. Hence, the shortest span  $\hat{S}_{D,\alpha}(\mathbf{x})$  has to be contiguous.
Case 2: APS- PS Construction follows Eqn. 7: For this case, we have,
$$
\hat {S} _ {\mathcal {D}, \alpha} (\mathbf {x}) = \left\{c _ {\pi_ {1}}, c _ {\pi_ {2}} \dots c _ {\pi_ {j}} \right\} \text {w h e r e} j = \sup  \left\{j!: \sum_ {k = 1} ^ {j ^ {\prime}} \hat {p} _ {\pi_ {k}} <   \hat {q} _ {D _ {c a l}} (\alpha) \right\} + 1, \tag {7}
$$
where  $\pi$  is a permutation of  $\{1,\dots ,K\}$  that sorts  $\hat{p}_k$  in the descending order from most likely to least likely and  $\hat{q}_{D_{cal}}(\alpha)$  is the bias-adjusted  $(1 - \alpha)^{th}$  quantile of the APS conformity scores as defined for Eqn. 7.
Let  $\hat{P}_{sum}(S) = \sum_{c_k\in S}\hat{p}_k$  denote the (fitted) probability mass within the prediction set  $S$ . Due to the unimodality of  $\hat{p}$ , it follows that one of the boundary labels  $c_{u}$  and  $c_{l + 1}$  have the minimum probability among those included in the set  $\hat{S}_{\mathcal{D},\alpha}(\mathbf{x})$ . Without loss of generality, let us assume  $\hat{p}_u$  is one of the minima (since the same argument can be applied for the case where  $(l + 1)$  is among the minima).
From the construction, we have,  $\hat{P}_{sum}(\hat{S}_{\mathcal{D},\alpha}(\mathbf{x})) \geq \hat{q}_{D_{cal}}(\alpha)$  and  $\hat{P}_{sum}(\hat{S}_{\mathcal{D},\alpha}(\mathbf{x}) \backslash \{c_u\}) < \hat{q}_{D_{cal}}(\alpha)$ . Consider the sets  $S_1 = \hat{S}_{\mathcal{D},\alpha}(\mathbf{x}) \cup \{c_{k^{skip}}\} \setminus \{c_u\}$  and  $S_2 = S_1 \setminus \{c_{k^{min}}\}$  where  $k^{min}$  is the largest index satisfying  $k^{min} = \arg \min_{k|c_k \in S_1} [\hat{p}_k]$ . Since  $\hat{p}_{k^{skip}} \geq \min(\hat{p}_{l+1}, \hat{p}_u)$ , it follows that  $\hat{P}_{sum}(S_1) \geq \hat{q}_{D_{cal}}(\alpha)$ . Further, from the definition of  $j$  as the size of largest top  $k$  set with probability mass as defined in Eqn. 7, it follows that  $\hat{P}_{sum}(S_2) < \hat{q}_{D_{cal}}(\alpha)$ .
Therefore, the set  $S_{1}$  is a valid APS prediction set as well with span  $(k^{min} - l) < (u - l)$ , which leads to a contradiction. Thus, the shortest span  $\hat{S}_{D,\alpha}(\mathbf{x})$  has to be contiguous for this case as well. Thus, in both cases, there exists at least one solution, i.e., shortest span prediction set, which is contiguous for both the constructions.
For the case, where  $\hat{p}$  is strictly unimodal, from the constructio Eqn.6 or 7, the prediction sets have to contain the top  $k$  most likely classes for some  $k$  which results in contiguity in case of strict unimodality.
Theorem 1. For any  $\mathbf{x} \in \mathcal{X}$ , let  $p_k(\mathbf{x}) = P_{Y|X}(Y = c_k|X = \mathbf{x})$  and  $\hat{p}_k(\mathbf{x}) = \hat{P}_{Y|X}(Y = c_k|X = \mathbf{x})$  denote the true and fitted model class probabilities that are always unimodal. Let  $\sigma_k(\mathbf{x}) = \sum_{k' = 1}^k p_{k'}(\mathbf{x})$  and  $\hat{\sigma}_k(\mathbf{x}) = \sum_{k' = 1}^k \hat{p}_{k'}(\mathbf{x})$  denote the corresponding cumulative distribution functions. If  $|\sigma_k(\mathbf{x}) - \hat{\sigma}_k(\mathbf{x})| \leq \delta$ ,  $[k]_1^K$  for a constant  $\delta$ , then for any  $\alpha \in (0,1]$ ,  $\forall \mathbf{x} \in D_{test}$ , the APS and oracle prediction sets from Eqn.7 and Eqn.4 satisfy  $|\hat{S}_{D,\alpha}(\mathbf{x})| \leq |S_{\alpha - 4\delta - \frac{1}{n + 1}}^{oracle}(\mathbf{x})|$  where  $n$  is the size of the calibration set.
Proof. To establish the result, we prove that the following two statements hold true under the assumption on the CDFs of  $P_{Y|X}$  and  $\hat{P}_{Y|X}$ :
(a)  $|\hat{S}_{D,\alpha}(\mathbf{x})| \leq |S_{1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta}^{Oracl e}(\mathbf{x})|$
(b)  $\left| S_{1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta}^{\text{oracle}}(\mathbf{x}) \right| \leq \left| S_{\alpha - 4\delta - \frac{1}{n + 1}}^{\text{oracle}} \right|$
Part (a): From Eqn. 4 and Lemma 1, we observe that the unimodality of  $\hat{p}(\mathbf{x})$  and  $p(\mathbf{x})$  leads to the oracle prediction set being contiguous and also the existence of a contiguous APS prediction set.
Since all the APS solution sets as per Eqn 7 have the same cardinality, we use  $\hat{S}_{D,\alpha}(\mathbf{x})$  to denote the contiguous solution.
Let  $\hat{S}_{D,\alpha}(\mathbf{x}) = \{c_{\hat{l} +1},\dots ,c_{\hat{u}}\}$ ,  $0\leq \hat{l} < \hat{u}\leq K$  and  $S_{1 - \hat{q}_{D_{cal}}}^{oracle}(\alpha) - 2\delta}(\mathbf{x}) = \{c_{l^* +1},\dots ,c_{u^*}\}$ ,  $0\leq l^{*} < u^{*}\leq K$ . From the definition of the sets and the contiguity, we observe that the probability mass of  $\hat{S}_{D,\alpha}(\mathbf{x})$  w.r.t.  $\hat{p}$  equals  $(\hat{\sigma}_{\hat{u}}(\mathbf{x}) - \hat{\sigma}_{\hat{l}}(\mathbf{x}))\geq \hat{q}_{D_{cal}}(\alpha)$  while that of  $S_{1 - \hat{q}_{D_{cal}}}^{oracle}(\alpha) - 2\delta}(\mathbf{x})$  w.r.t  $p$  equals  $(\sigma_{u^{*}}(\mathbf{x}) - \sigma_{l^{*}}(\mathbf{x}))\geq 1 - (1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta) = \hat{q}_{D_{cal}}(\alpha) + 2\delta$ .
Using the divergence bound on the two CDFs, i.e.,  $|\sigma_k(\mathbf{x}) - \hat{\sigma}_k(\mathbf{x})| \leq \delta$ ,  $[k]_1^K$ , we have
$$
\begin{array}{l} \left(\hat {\sigma} _ {u ^ {*}} (\mathbf {x}) - \hat {\sigma} _ {l ^ {*}} (\mathbf {x})\right) \geq \left(\sigma_ {u ^ {*}} (\mathbf {x}) - \delta\right) - \left(\left(\sigma_ {l ^ {*}} (\mathbf {x}) + \delta\right) \right. \\ = \sigma_ {u ^ {*}} (\mathbf {x}) - \sigma_ {l ^ {*}} (\mathbf {x}) - 2 \delta \\ \ge \hat {q} _ {D _ {c a l}} (\alpha) + 2 \delta - 2 \delta \\ = \hat {q} _ {D _ {c a l}} (\alpha). \\ \end{array}
$$
Since  $\hat{S}_{D,\alpha}(\mathbf{x})$  is the minimal contiguous set with probability mass greater than or equal to  $\hat{q}_{D_{cal}}(\alpha)$  as per  $\hat{p}$  in Eqn 7, we have
$$
| \hat {S} _ {D, \alpha} (\mathbf {x}) | = (\hat {u} - \hat {l}) \leq (u ^ {*} - l ^ {*}) = | S _ {1 - \hat {q} _ {D _ {c a l}} (\alpha) - 2 \delta} ^ {o r a c l e} (\mathbf {x}) |.
$$
Part (b): Denoting the minimal contiguous APS prediction set by  $\hat{S}_{D,\alpha}(\mathbf{x})$  as before, we have  $(\hat{\sigma}_{\hat{u}}(\mathbf{x}) - \hat{\sigma}_{\hat{l}}(\mathbf{x})) \geq \hat{q}_{D_{cal}}(\alpha)$ . Considering the divergence bound on the two CDFs, i.e.,  $|\sigma_k(\mathbf{x}) - \hat{\sigma}_k(\mathbf{x})| \leq \delta$ ,  $[k]_1^K$ , we have  $(\hat{\sigma}_{\hat{u}}(\mathbf{x}) - \hat{\sigma}_{\hat{l}}(\mathbf{x})) \leq (\sigma_{\hat{u}}(\mathbf{x}) + \delta) - ((\sigma_{\hat{l}}(\mathbf{x}) - \delta) = \sigma_{\hat{u}}(\mathbf{x}) - \sigma_{\hat{l}}(\mathbf{x}) + 2\delta$ .
Hence, for all  $\mathbf{x}$ , we have
$$
\begin{array}{l} \left(\hat {\sigma} _ {\hat {u}} (\mathbf {x}) - \hat {\sigma} _ {\hat {l}} (\mathbf {x})\right) \geq \hat {q} _ {D _ {c a l}} (\alpha) \\ \Leftrightarrow \quad \sigma_ {\hat {u}} (\mathbf {x}) - \sigma_ {\hat {l}} (\mathbf {x}) + 2 \delta \geq \hat {q} _ {D _ {c a l}} (\alpha) \\ \Leftrightarrow \quad \sigma_ {\hat {u}} (\mathbf {x}) - \sigma_ {\hat {J}} (\mathbf {x}) \geq \hat {q} _ {D _ {c a l}} (\alpha) - 2 \delta \\ \end{array}
$$
Since this holds for all  $\mathbf{x}$ , the marginal coverage  $P[Y\in \hat{S}_{D,\alpha}(X)]\geq \hat{q}_{D_{cal}}(\alpha) - 2\delta$
From Theorem 3, we also have an upper bound on the marginal coverage for test samples, i.e.,  $P[Y \in \hat{S}_{D,\alpha}(X)] \leq 1 - \alpha + \frac{1}{n+1}$  where  $n$  is the size of the calibration set.
$$
\begin{array}{l} 1 - \alpha + \frac {1}{n + 1} \geq P [ Y \in \hat {S} _ {D, \alpha} (X) ] \\ \Leftrightarrow \quad 1 - \alpha + \frac {1}{n + 1} \geq \hat {q} _ {D _ {c a l}} (\alpha) - 2 \delta \\ \Leftrightarrow \quad 1 - \hat {q} _ {D _ {c a l}} (\alpha) - 2 \delta \geq \alpha - 4 \delta - \frac {1}{n + 1} \\ \end{array}
$$
From the above inequality and the definition of the oracle prediction set, we observe that
$$
| S _ {1 - \hat {q} _ {D _ {c a l}} (\alpha) - 2 \delta} ^ {o r a c l e} (\mathbf {x}) | \leq | S _ {\alpha - 4 \delta - \frac {1}{n + 1}} ^ {o r a c l e} |.
$$
Combining the results in part (a) and (b), we have
$$
| \hat {S} _ {D, \alpha} (\mathbf {x}) | \leq | S _ {\alpha - 4 \delta - \frac {1}{n + 1}} ^ {o r a c l e} |.
$$
As the size of the calibration set increases, the term  $\frac{1}{n + 1}$  vanishes and as the divergence  $\delta$  decreases, then the cardinality of the APS set converges to that of the oracle set.
Theorem 2. Let  $\eta : \mathcal{X} \to R^K$ ,  $\phi : R \to R^+$  and  $\psi^E : R \to R^-$  such that  $\psi^E(r) = \psi^E(-r)$ ,  $\forall r \in R$  and its restriction to  $R^+$  is a strictly monotonically decreasing bijective function. (a) Then, the model output constructed as per Eqn. 5 is always unimodal, i.e.,  $\hat{p}(\mathbf{x}) \in \mathcal{U}$ ,  $\forall \mathbf{x} \in \mathcal{X}$ . (b) Further, given any  $\hat{p}(\mathbf{x}) \in \mathcal{U}$  for  $\mathbf{x} \in \mathcal{X}$ , there exists a function  $\eta(\mathbf{x}) \in R^K$  that satisfies Eqn. 5 if  $\phi(\cdot)$  is surjective on  $R^+$ .
Proof. We begin by restating the construction:
$$
\eta (\mathbf {x}) = \mathbf {f} (\mathbf {x}, \theta); v _ {1} = \eta_ {1} (\mathbf {x}); v _ {k} = \phi (\eta_ {k} (\mathbf {x})), [ k ] _ {2} ^ {K},
$$
$$
r _ {1} = v _ {1}; r _ {k} = r _ {k - 1} + v _ {k}, [ k ] _ {2} ^ {K}; z _ {k} = \psi^ {E} (r _ {k}); \hat {p} _ {k} = \frac {\exp (z _ {k})}{\sum_ {k = 1} ^ {K} \exp (z _ {k})}, [ k ] _ {1} ^ {K}.
$$
Figure 7: Construction of our DNN
**Part a:** Following the above construction, for any  $\mathbf{x} \in \mathcal{X}$ , since  $\phi : R \to R^{+}$ , the DNN output  $v_{k} \geq 0$ ,  $[k]_2^K$ . Hence, the cumulative sum sequence  $\mathbf{r}$  is non-decreasing, i.e.,  $r_1 \leq r_2 \leq \dots \leq r_K$ .
There can be 3 possible scenarios:
Scenario 1.  $r_1 \leq r_2 \dots \leq r_K \leq 0$ : In this case,  $[z_k = \psi^E(r_k)]_{k=1}^K$  is also a non-decreasing sequence and so is  $[\hat{p}_k]_{k=1}^K$ . Here  $[\hat{p}_k]_{k=1}^K$  is unimodal sequence with mode at  $c_K$ .
Scenario 2.  $0 \leq r_{1} \leq r_{2} \dots <= r_{K}$ : In this case,  $[z_{k} = \psi^{E}(r_{k})]_{k=1}^{K}$  is also a non-increasing sequence and so is  $[\hat{p}_{k}]_{k=1}^{K}$ . Here  $[\hat{p}_{k}]_{k=1}^{K}$  is unimodal sequence with mode at  $c_{1}$ .
Scenario 3.  $r_1 \leq r_2 \dots \leq r_m \leq 0 \leq r_{m+1} \leq \dots \leq r_K$  for some  $m$ . In this case,  $[z_k = \psi^E(r_k)]_{k=1}^K$  is non-decreasing till  $m$  and non-increasing from  $m+1$  onwards, which makes it unimodal. The mode is either  $m$  or  $m+1$  or both depending on the magnitudes  $|r_m|$  and  $|r_{m+1}|$ . The probability distribution  $[\hat{p}_k]_{k=1}^K$  follows the same pattern and is unimodal as well.
Part b: Let us assume  $\hat{p}(\mathbf{x}) \in \mathcal{U}$  is any arbitrary unimodal distribution conditioned on  $x$  with class probabilities  $\hat{p}_k \leq \hat{p}_{k+1}$ ,  $[k]_1^{m-1}$  and  $\hat{p}_k \geq \hat{p}_{k+1}$ ,  $[k]_m^K$ , where  $m$  is the highest indexed (in case of multiple) mode of the unimodal distribution. We can then obtain  $z_k = \log(\hat{p}_k)$ ,  $[k]_1^K$  and construct the sequence  $r_k = (\psi^E)^{-1}(z_k)$  where  $r_k \in R^-$  for  $1 \leq k \leq (m-1)$  and  $r_k \in R^+$  for  $m \leq k \leq K$ . Since  $\psi : R^+ \to R^-$  is a strictly monotonically decreasing bijective function and  $\psi^E$  is its even extension, the sequence  $[r_k]_{k=1}^K$  is well-defined. Further, since  $[z_k]_{k=1}^K$  is a unimodal sequence,  $[r_k]_{k=1}^K$  is monotonically increasing with  $r_{m-1} \leq 0 \leq r_m$ . Then, we can obtain the vector  $\mathbf{v}$  such that  $v_k = r_k - r_{k-1} \geq 0$ ,  $[k]_2^K$  and  $v_1 = r_1$ . When  $\phi(\cdot)$  is a surjective function on  $R^+$ , we can define  $\eta_k(\mathbf{x}) = (\phi)^{-1}(v_k)$ ,  $[k]_2^K$  and  $\eta_1(\mathbf{x}) = v_1$ . There will always be a valid  $\eta(\mathbf{x})$ , which ensures that construction can generate the original  $\hat{p}(\mathbf{x})$ .
B.1 APS Coverage guarantees
Theorem 3. [APS [35]] If samples  $(x_{i},y_{i})$ $x_{i}\in \mathcal{X},y_{i}\in \mathcal{Y}$  are exchangeable  $\forall 1\leq i\leq n$  and all samples from  $D_{train}$ ,  $D_{cal}$  are invariant to permutations, and conformity scores are almost surely distinct, then APS algorithm gives tight marginal coverage given by:
$$
1 - \alpha \leq P [ Y _ {t e s t} \in \hat {S} _ {D, \alpha} (X _ {t e s t}) ] \leq 1 - \alpha + \frac {1}{| D _ {c a l} | + 1}
$$
C.1 Benchmark Image Datasets and Implementation Details
We now provide a brief description of the four public datasets and the modeling details. For each of these datasets, we split the data into train, calibration, and test sets. We use calibration set to calibrate APS and report mean and standard deviation (std. error) on the test set across 5 independent splits. Note that for all experiments to avoid over-fitting, data augmentation, i.e., random horizontal flipping and random cropping for each training image, was applied in our experiments. The predictions was obtained with a central crop during testing. COPOC was implemented with  $\phi = |x|$  and  $\psi = -|x|$ .
Age Estimation - Adience [13]: The task associated with this dataset is to predict the age for a given facial image. This dataset contains 26580 Flickr photos of 2284 subjects. The age is annotated with eight groups:  $0 - 2, 4 - 6, 8 - 13, 15 - 20, 25 - 32, 38 - 43, 48 - 53$ , and over 60 years old. From the nature of the class labels, it is evident that classes are not equally spaced categories. Hence, previous works which assumed it to be equi-spaced (SORD [12] for instance) are suboptimal. For feature extractor backbone, we use ImageNet pre-trained VGG-16 network since most competing methods [23, 12] used this model. For our usage we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2. We trained models for 50 epochs with a batch size of 64. For optimization, Adam optimizer was utilized with a learning rate of 0.0001, with decay rate of 0.2.
Historical Colour Image Dating - HCI [30]: The historical color image dataset is collected for the task of estimating the age of historical color photos. Each image is annotated with its associated decade, where five decades from the 1930s to 1970s are considered. There are 265 images for each category. Following [23] we utilized VGG-16 as the backbone, which was initialized with the ImageNet pre-trained weights for a fair comparison. We trained models for 50 epochs with Adam optimizer with a learning rate of 0.0001, with decay rate of 0.2. For COPOC, we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2.
Retina-MNIST [44]: RetinaMNIST is based on the DeepDRiD24 challenge, which provides a dataset of 1600 retina fundus images. The task is ordinal classification for 5-level grading of diabetic retinopathy severity. We use a similar feature extractor network as used in [44] along with a final unimodality constrained layer at end. The network was trained with same settings as [44].
Image Aesthetic Estimation [37]: The Aesthetics dataset consists of 15687 Flickr image belonging to four different nominal categories: animals, urban, people, and nature. All The pictures are annotated by 5 different graders in 5 aesthetic categories in an orderly manner: 1) "unacceptable" pictures with extremely low quality, 2) "flawed" low quality images (slightly blurred, over/underexposed), and with no artistic value; 3) "ordinary" images without technical flaws (well framed, in focus), but no artistic value; 4) "professional" images (flawless framing, lightning), and 5) "exceptional", very appealing images, showing outstanding quality. The ground truth label for each image is set to be the median among all of its gradings. Following [23, 12] we use ImageNet pre-trained VGG-16 as the backbone for feature extraction. For our usage, we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2. We only report aggregate metric across all the categories for this data.
Additional Implementation Details: On the public benchmark datasets where the official best hyperparameters are available for baseline methods (For instance in Adience, HCI and Aesthetic dataset best settings for POE and SORD, and for Binomial best settings on Adience were available) from the corresponding authors work or code, we directly use those settings. We were able to replicate the results (MAE and Accuracy) on these datasets as reported by them. For all other cases (namely AVDL, Uni-Loss), we optimize for MAE in hyperparameters search since that is the
most common metric used for all ordinal classification tasks across competing benchmarks. We cross-validate over the following grid:
- learning rate  $\in \{1e - 2, 1e - 3, 1e - 4\}$  with decay rate of 0.2.
- weight decay  $\in \{0, 1e - 3, 1e - 2, 1e - 1\}$ .
- dropout rate  $\in$  {0.1, 0.25, 0.5, 0.75}.
- Adam optimizer with default settings.
Few additional algorithm specific hyperparameters that needed tuning were:
- For POE, there are two extra hyperparameters of  $\alpha$  and  $\beta$  in its distance-aware loss function in embedding space which we search over  $\{1e - 3, 1e - 4, 1e - 5, 1e - 6\}$  as suggested by the authors.
- SORD describes three type of distance metric losses - absolute inter class distance, squared distance and its log variant. We search over these loss functions too.
- AVDL requires choosing the initial variance (of the Gaussian) of all images which we search from  $\{0.25, 0.5, 1, 2\}$  similar to their work.
- Uni-loss has hyperparameter that controls the weightage between unimodality and mean-variance component of its loss function. We search over  $\{10, 100, 500, 1000, 5000\}$ .