SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Prasenjit Dey
prasendx@amazon.com
Srujana Merugu
smerugu@amazon.com
Sivaramakrishnan Kaveri
kavers@amazon.com
Amazon, India
Ordinal classification (OC), i.e., labeling instances along classes with a natural ordering, is common in multiple applications such as disease severity labeling and size or budget based recommendations. Often in practical scenarios, it is desirable to obtain a small set of likely classes with a guaranteed high chance of including the true class. Recent works on conformal prediction (CP) address this problem for the classification setting with non-ordered labels but the resulting prediction sets (PS) are often non-contiguous and unsuitable for ordinal classification. In this work, we propose a framework to adapt existing CP methods to generate contiguous sets with guaranteed coverage and minimal cardinality. Our framework employs a novel non-parametric approach for modeling unimodal distributions. Empirical results on both synthetic and real-world datasets demonstrate that our method outperforms SOTA baselines by  $4\%$  on Accuracy@K and  $8\%$  on PS size.
A large number of practical applications involve ordinal classification (OC), i.e., labeling of instances along a set of ordered discrete classes. These include applications such as determining severity level of abuse incidents or disease diagnosis for efficient follow up, predicting customer age-range, apparel size, and budget for improving recommendations and reducing customer cognitive effort by offering adaptive search filters. Often, in such cases, it is desirable to identify a small set of classes (called prediction set or PS) for a given input instance such that the true class is amongst the identified set with a high probability (called coverage level) instead of a single point estimate of the most likely class. This allows more flexibility in choosing operating points that balance precision-recall trade-off. For instance, customers looking for shoes are often recommended products that are not even available in the customer's typical size. An OC model that predicts the customer size would be well suited to improve the recommendations offered. However, limiting the recommendations to a single most likely size is too restrictive because the model accuracy is often low due to variations in size over time and across brands. Instead, it is preferable to identify a small set of product sizes with say  $>90\%$  coverage and use that for automatically filtering the recommendations as shown in Fig. 1.
Conformal prediction (CP)[35, 1, 36, 9] is an emerging area that addresses the problem of predicting a set of likely outputs along with a measure of the confidence or coverage level. While existing CP methods offer ease of deployment and provable coverage guarantees, most of these methods deal with unordered classes and output prediction sets that are non-contiguous and unsuitable for OC settings where the underlying distribution is likely unimodal. For example, for any customer, based on their foot size, a good fit is possible only with a small set of proximal shoe sizes resulting in a unimodal class distribution. Recommending a non-contiguous set of sizes [4, 8, 9] is not a desirable customer experience. On the other hand, expanding the PS to a minimal contiguous superset leads to
Figure 1: A comparison on different model fit on an underlying unimodal distribution. CP on Unconstrained fit results in non-contiguous sets (red). While CP on unimodal fits results in contiguous set (green), a better fitted model can result in the ideal minimal set (blue) which we seek.
a significant expansion of the set size (e.g.,  $[4,8,9] \to [4,5,6,7,8,9]$ ) diminishing its application utility and diluting coverage guarantees. A similar contiguity requirement arises in applications such as cancer stage diagnosis where predicting non-contiguous labels (stage-1, stage-4) would be discordant compared to contiguous labels. Note that there do exist OC scenarios such as prediction of day of week for an event where unimodality might not be valid as we discuss in Section 6.
While ordinal classification [18, 7] has been well-studied in the past, most earlier works [28, 8] do not assume unimodality. Recent works [4, 22, 42, 12, 15, 20] demonstrate strong benefits of assuming the underlying conditional distribution to be unimodal for OC tasks. However, these techniques primarily focus on point estimates and not conformal predictions. Furthermore, these methods often either impose a restrictive structure on the model [33] or assume parametric models (e.g., Binomial[4], Gaussian[42]), which limits their representation ability and utility for real-world applications. Given the multitude of practical applications, there is a strong need for a principled approach that combines ideas from ordinal classification and conformal predictions to construct minimal contiguous prediction sets when the underlying true class distribution is unimodal.
In this work, we focus on ordinal classification where the true class distribution is unimodal. We build on existing work, Adaptive Prediction Set (APS) [35], to obtain minimal contiguous prediction sets with provable coverage guarantees. Below we summarize our key contributions:
1. For ordinal classification where the true class distribution is unimodal, we show that any model constrained to output a unimodal distribution can be used with APS algorithm to yield contiguous PS with a guaranteed coverage level. Further, we prove a tight upper bound for the cardinality of this set in terms of the optimal set and the divergence between the fitted model and the true distribution.
2. We provide a novel construction to adapt any DNN-based architecture to always yield a unimodal distribution over ordinal classes while ensuring that any arbitrary unimodal class distribution can be approximated.
3. We study the efficacy of our approach for Conformal Predictions for OC (COPOC) via controlled ablation studies on synthetic datasets which demonstrates the utility of unimodal construction in improving learnability relative for small data sizes, and its versatility compared to other methods that make parametric distributional assumptions.
4. We perform extensive evaluation on benchmark datasets for facial age estimation, HCI classification, image aesthetic score estimation and biomedical image classification with results indicating that our method outperforms SOTA baselines by  $4\%$  on Accuracy@K and  $8\%$  on PS size.
2 Contiguous Conformal Prediction Set Problem
Notation:  $\mathbf{P}[\mathbf{E}]$  denotes the probability of event  $\mathbf{E}$ .  $[i]_a^b$  denotes conditioning on integer index  $\forall i$ ,  $a \leq i \leq b$  within a range while  $\mathbf{x} = [x_i]_{i=a}^b$  denotes the vector or sequence  $[x_a, \dots, x_b]$ .
Let  $\mathcal{X}$  denote input space and  $\mathcal{C} = \{c_k\}_{k=1}^K$  denote an ordered set of  $K$  classes. Consider the case where the random variables  $X, Y$  are drawn from  $\mathcal{X} \times \mathcal{C}$  following a joint distribution  $P_{X,Y}$  such that the true underlying conditional distribution  $P_{Y|X}$  is unimodal. In other words, for any  $\mathbf{x} \in \mathcal{X}$ , denoting  $p_k = P_{Y|X}(Y = c_k | X = x)$ , there exists a class  $c_m \in \mathcal{C}$  such that  $p_k \leq p_{k+1}$ ,  $[k]_1^{m-1}$  and
$p_k \geq p_{k+1}, [k]_m^K$ . Given a training dataset  $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  with samples drawn exchangeably (e.g., i.i.d.) from  $P_{X,Y}$  and a desired coverage level  $(1 - \alpha) \in (0,1)$  for any unseen input test instance  $(X_{test}, Y_{test}) \in D_{test}$  drawn from the same distribution, the objective is to determine the minimal non-empty contiguous prediction set (PS)  $\hat{S}_{\mathcal{D},\alpha}(X_{test}) = \{c_{l+1}, \dots, c_u\}$ ,  $0 \leq l < u \leq K$  satisfying marginal coverage with finite samples above a specified level:
$$
\mathbf {P} \left[ Y _ {\text {t e s t}} \in \hat {S} _ {\mathcal {D}, \alpha} \left(X _ {\text {t e s t}}\right) \right] \geq (1 - \alpha). \tag {1}
$$
The above probability is taken over all  $(X_{test},Y_{test})\in D_{test}$  data points. While marginal coverage is achievable, it differs from conditional coverage:  $\mathbf{P}[Y_{test}\in \hat{S}_{\mathcal{D},\alpha}(X_{test})|X_{test} = \mathbf{x}_{test}]\geq (1 - \alpha)$ , which is a stronger notion. In practice, it is desirable for  $\hat{S}_{\mathcal{D},\alpha}$  to approximate conditional coverage and achieve it asymptotically in the limit of large sample sizes.
Conformal Prediction. Originating in [40, 41], conformal prediction is a statistical approach for generating predictive sets with marginal coverage guarantees (Eqn.1) with finite samples. Earlier methods for calibrating probabilities (e.g., via Platt scaling [32] or temperature scaling [17]) and assembling a prediction set by including the most likely classes till the covered probability mass exceeds a specified coverage level do not achieve the desired theoretical guarantees since the calibration is often erroneous[29]. Most recent CP methods use split conformal prediction that enables easy deployment with any arbitrary predictor [31, 21]. APS [35], [9], and [36] introduce techniques aimed at achieving coverage that is similar across regions of feature space whereas [19] propose methods to achieve equal coverage for each class. Of these, Least Ambiguous Set-valued Classifier (LAC) [36] provides provably smallest average PS size when probability estimates are approximately correct while APS[35] has been shown to be superior in practice. For a primer on conformal methods please refer [2]. RAPS[1] presents a regularized version of APS for Imagenet. Although relatively new, CP methods are being widely applied for regression, vision, NLP and time-series forecasting [34, 38, 3, 14]. Our work adapts existing CP methods to unimodal ordinal classification setting with an additional contiguity requirement. Ordinal-APS [25] and [43] share the same motivation but propose a greedy CP method that results in larger prediction sets compared to COPOC (Table 4).
Ordinal Classification. Early works on ordinal classification [18, 7, 45] are based on regular classification or regression, ignoring ordinal relationship and non-uniform separation among labels. Some DNN-based methods model cumulative distribution (CDF) up to each ordered class using multiple binary classifiers [8, 39, 6, 28] but the output class probabilities are not necessarily unimodal and are also not guaranteed to form a valid CDF. Label smoothing methods [15, 16, 24, 12, 20, 42] convert hard target labels into unimodal prior distributions to be used as the reference for the training loss but these methods are often sub-optimal since the assumed priors might not reflect the true distribution, classes might not be equi-spaced categories and additionally test predictions might not necessarily be unimodal. [22, 5] learns a non-parametric unimodal distribution as a constraint optimization problem in the loss function which is not only difficult to optimize but also does not guarantee unimodality on test data. In [4, 10], unimodality is guaranteed by assuming a Binomial or Poisson distribution but has limited representation ability due to a single degree-of-freedom which does not capture heteroscedastic noise. Keeping in view the potential benefits of accurate unimodal distribution learning for the ordinal classification setting, our work explores a non-parametric flexible DNN-based approach that guarantees unimodality of the predicted class distribution even on test data.
4 Solution Approach
To generate conformal predictions for ordinal classification, we consider two questions: (i) what is the primary drawback of the existing CP methods for classification with respect to ordered labels setting? (ii) how do we address this drawback?
Most CP methods for classification [35, 36, 9] are based on a split calibration approach where the training data  $D$  is divided into two sets. The first set  $D_{train}$  is used to learn a probabilistic model  $\hat{P}_{Y|X}(\cdot)$  using a blackbox learning algorithm while the second set  $D_{cal}$  is used to determine the optimal conformity score threshold for a desired coverage level, which is then used for conformal inference. Below we discuss two common CP methods:
Least Ambiguous Set Values Classifier (LAC) [36]. Here, the conformal PS for a test instance  $\mathbf{x}_{test}$  is constructed using a calibration rule of the form,
$$
\hat {S} _ {D, \alpha} (\mathbf {x} _ {\text {t e s t}}) = \{c \in \mathcal {C} | \hat {p} _ {k} (\mathbf {x} _ {\text {t e s t}}) \geq \hat {q} _ {D _ {c a l}} (\alpha) \}, \tag {2}
$$
where  $\hat{p}_k(\mathbf{x}) = \hat{P}_{Y|X}(Y = c_k|X = \mathbf{x})$  is the class probability from the model and  $\hat{q}_{D_{cal}}(\alpha)$  is the score threshold defined as the bias-adjusted  $(\alpha)^{th}$  quantile of the model score of the true label.
Adaptive Prediction Set (APS) [35]. In APS, for every instance  $(\mathbf{x},y)\in D_{cal}$ , we compute a conformity score defined as the cumulative probability mass required to include the true label corresponding to that instance, i.e.,  $s(\mathbf{x}) = \sum_{k = 1}^{T}\hat{p}_{\pi_k}(\mathbf{x})$ , where  $T$  is chosen such that  $c_{\pi_T} = y$  and  $\pi$  is the permutation of  $\{1,\dots ,K\}$  that sorts  $\hat{P}_{Y|X}(.|\mathbf{x})$  in the descending order from most likely to least likely. Given a desired coverage  $(1 - \alpha)$ , we compute a suitable score threshold  $\hat{q}_{D_{cal}}(\alpha) = \text{Quantile} (\{s(\mathbf{x})|\mathbf{x}\in D_{cal}\},\frac{\lceil(n + 1)(1 - \alpha)\rceil}{n})$ , where  $n = |D_{cal}|$ . The conformal PS in APS is constructed using:
$$
\hat {S} _ {\mathcal {D}, \alpha} \left(\mathbf {x} _ {t e s t}\right) = \left\{c _ {\pi_ {1}}, c _ {\pi_ {2}} \dots c _ {\pi_ {j}} \right\} \text {w h e r e} j = \sup  \left\{j ^ {\prime}: \sum_ {k = 1} ^ {j ^ {\prime}} \hat {p} _ {\pi_ {k}} \left(\mathbf {x} _ {t e s t}\right) <   \hat {q} _ {D _ {c a l}} (\alpha) \right\} + 1. \tag {3}
$$
More details on the coverage guarantees of APS are in Appendix B. While the above methods yield minimal sets with provable guarantees on marginal coverage, the primary drawback is that the resulting prediction sets are not necessarily contiguous, which is an important requirement for ordinal classification when underlying true distribution is unimodal.
A naive solution is to consider the minimal contiguous set that covers the PS output by CP method. For example, if CP outputs  $\{2, 4, 6\}$ , we can produce a minimal contiguous set by including "in-between" classes i.e.,  $\{2, 3, 4, 5, 6\}$ . However, this would end up having a much larger cardinality, which invalidates the tight coverage bounds (refer Appendix B) (i.e., the new set will have a coverage much higher than the upper bound guaranteed for that method). Hence, we consider an alternative approach based on the observation that if the blackbox learning algorithm in the first step is constrained to output a unimodal distribution, then the classes with probability above a threshold will cluster around the mode. The conformal calibration rules in Eqn. 2 and 3, which are designed to choose classes in the order of class probability will thus result in contiguous sets. Using this insight, we propose a solution depicted in Fig. 2 that consists of two steps: (a) design a learning algorithm that can accurately model training data such that the output class distribution is always unimodal, (b) adapt an appropriate CP algorithm based on Eqn. 2 or Eqn. 3 to identify the PS. While this high level approach can be used with any of the CP method, in our current work, we consider the Adaptive Predictive Set (APS) approach [35] since it provides tight guarantees on marginal coverage and has been empirically shown to achieve better conditional coverage than LAC [36] and smaller set size than CQC [9]. We now describe our solution beginning with a study of the properties of the resulting PS (Sec. 4.1) followed by construction of the unimodal classifier (Sec. 4.2).
4.1 Conformal Prediction Sets from Unimodal Models
We now study the properties of PS corresponding to the COPOC approach outlined in Fig. 2 where for every input instance  $X$ , both the true underlying distribution  $P_{Y|X}$  and the learned classification model  $\hat{P}_{Y|X}$  are unimodal. In other words, there exists a class  $c_{\hat{m}} \in \mathcal{C}$  such that  $\hat{p}_k \leq \hat{p}_{k+1}$ ,  $[k]_1^{\hat{m}-1}$ , and  $\hat{p}_k \geq \hat{p}_{k+1}$ ,  $[k]_m^K$ . For a given  $\mathbf{x}_{test}$  and coverage level  $(1-\alpha)$ , let  $S_{\alpha}^{oracle}(\mathbf{x}_{test})$  denote the minimal contiguous PS that can be constructed by an oracle that knows the underlying  $P_{Y|X}$ , i.e.,
$$
S _ {\alpha} ^ {o r a c l e} \left(\mathbf {x} _ {t e s t}\right) = \underset {S = \{c _ {l + 1}, \dots , c _ {u} \}, 0 \leq l <   u \leq K, \mathbf {P} \left[ y _ {t e s t} \in S \mid \mathbf {x} _ {t e s t} \right] \geq (1 - \alpha)} {\arg \min } [ u - l ]. \tag {4}
$$
where  $p_k = P_{Y|X}(Y = c_k|X = \mathbf{x}_{test})$ . This is the set with the least cardinality satisfying contiguity and conditional coverage. In case of multiple solutions, the oracle can pick any one at random. In practice, we do not have access to the true  $P_{Y|X}$  and  $\hat{S}_{D,\alpha}(\mathbf{x}_{test})$  has to be constructed from the approximate distribution  $\hat{P}_{Y|X}$  using CP methods such as APS for valid coverage guarantees. Lemma 1 establishes the contiguity of the prediction sets resulting from LAC and APS. Thm. 1 bounds the cardinality of the APS PS relative to the oracle sets. Further, when  $\hat{P}_{Y|X}$  is a consistent estimator of
Figure 2: Framework of COPOC.
Figure 3: Unimodal DNN Construction.
the true  $P_{Y|X}$ , the APS conformity score threshold asymptotically converges to  $\alpha$  and the cardinality of  $\hat{S}_{D,\alpha}(\mathbf{x}_{test})$  converges to that of the oracle [35].
Lemma 1.  ${}^{1}$  Given a fitted unimodal model  ${\widehat{P}}_{Y \mid  X}$  ,for any  ${\mathbf{x}}_{\text{test }}$  and  $\alpha  \in  \left( {0,1}\right\rbrack$  ,prediction sets  ${\widehat{S}}_{D,\alpha }\left( {\mathbf{x}}_{\text{test }}\right)$  constructed using Eqn.2 or 3 have at least one solution which is contiguous (amongst multiple possibilities). When  ${\widehat{P}}_{Y \mid  X}$  is strictly unimodal,all the solutions are contiguous.
Proof Sketch: By design, the prediction sets from Eqn.2 or 3 have to contain the top  $k$  most likely classes for some  $k$  which results in contiguity in case of strict unimodality. When the unimodality is not strict, non-contiguous sets may satisfy the required constraints due to classes with equal probabilities at the boundaries, but the shortest span solution is contiguous.
Theorem 1. For any  $\mathbf{x} \in \mathcal{X}$ , let  $p_k(\mathbf{x}) = P_{Y|X}(Y = c_k|X = \mathbf{x})$  and  $\hat{p}_k(\mathbf{x}) = \hat{P}_{Y|X}(Y = c_k|X = \mathbf{x})$  denote the true and fitted model class probabilities that are always unimodal. Let  $\sigma_k(\mathbf{x}) = \sum_{k' = 1}^{k} p_{k'}(\mathbf{x})$  and  $\hat{\sigma}_k(\mathbf{x}) = \sum_{k' = 1}^{k} \hat{p}_{k'}(\mathbf{x})$  denote the corresponding cumulative distribution functions. If  $|\sigma_k(\mathbf{x}) - \hat{\sigma}_k(\mathbf{x})| \leq \delta$ ,  $[k]_1^K$  for a constant  $\delta$ , then for any  $\alpha \in (0,1]$ ,  $\forall \mathbf{x} \in D_{test}$ , the APS and oracle prediction sets from Eqn.3 and Eqn.4 satisfy  $|\hat{S}_{D,\alpha}(\mathbf{x})| \leq |S_{\alpha - 4\delta - \frac{1}{n + 1}}^{\text{oracle}}(\mathbf{x})|$  where  $n$  is the size of the calibration set.
Proof Sketch.: To establish the result, we prove the following two statements: (a)  $|\hat{S}_{D,\alpha}(x)| \leq |S_{1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta}^{O r a c l e}(x)$ , and (b)  $|S_{1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta}^{O r a c l e}(\mathbf{x})| \leq |S_{\alpha -4\delta -\frac{1}{n + 1}}^{O r a c l e}|$ . From Eqn. 4 and Lemma 1, we observe that the unimodality of  $\hat{p} (\mathbf{x})$  and  $p(\mathbf{x})$  leads to the oracle prediction sets and at least one of the APS prediction sets being contiguous. Let  $\hat{S}_{D,\alpha}(\mathbf{x}) = \{c_{\hat{l} +1},\dots ,c_{\hat{u}}\} ,0\leq \hat{l} < \hat{u}\leq K$  and  $S_{1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta}^{O r a c l e}(\mathbf{x}) = \{c_{l^* +1},\dots ,c_{u^*}\} ,0\leq l^* < u^*\leq K$ . From the definitions and contiguity, we observe that the probability mass of  $\hat{S}_{D,\alpha}(\mathbf{x})$  w.r.t.  $\hat{p}$  equals  $(\hat{\sigma}_{\hat{u}}(\mathbf{x}) - \hat{\sigma}_{\hat{l}}(\mathbf{x}))\geq \hat{q}_{D_{cal}}(\alpha)$  while that of  $S_{1 - \hat{q}_{D_{cal}}(\alpha) - 2\delta}^{O r a c l e}(\mathbf{x})$  w.r.t  $p$  equals  $(\sigma_{u^*}(\mathbf{x}) - \sigma_{l^*}(\mathbf{x}))\geq \hat{q}_{D_{cal}}(\alpha) + 2\delta$ . Using the divergence bound on the two CDFs and the fact that  $\hat{S}_{D,\alpha}(\mathbf{x})$  is the minimal contiguous set with probability mass  $\geq \hat{q}_{D_{cal}}(\alpha)$  as per  $\hat{p}$  yields (a). Using the divergence bound, we also observe that the marginal coverage,  $P[y\in \hat{S}_{D,\alpha}(\mathbf{x})]\geq \hat{q}_{D_{cal}}(\alpha) - 2\delta$ . Combining this from the APS coverage guarantees (Theorem 3 in Appendix B) yields part (b).
4.2 Non-Parametric Unimodal Distribution Learning
We now consider the problem of learning a classification model that is guaranteed to be unimodal and also expressive enough to accurately model any unimodal distribution. Given universal approximation properties of DNNs with respect to probability distributions [26], these form a natural choice as a multi-class classification learning algorithm. For a given  $\mathbf{x}$ , let  $\mathbf{z}(\mathbf{x}) = \mathbf{f}(\mathbf{x};\theta)$  denote the output of last fully connected layer of a DNN model  $\mathbf{f}(\cdot)$ , where  $\theta$  is the model parameters and  $\mathbf{z}(\mathbf{x})$  a  $K$ -dimensional vector where  $K = |\mathcal{C}|$ . Softmax operation is applied to  $\mathbf{z}(\mathbf{x})$  to obtain class probability distribution. Predicted probability of class  $c_k \in \mathcal{C}$  for  $\mathbf{x}$  is given by  $\hat{p}_k(\mathbf{x}) = \frac{\exp(z_k(\mathbf{x}))}{\sum_{k=1}^{K} \exp(z_k(\mathbf{x}))}$ . Training is usually performed by minimizing cross-entropy (CE) loss between
Detailed proofs are in Appendix B.
this predicted distribution and the ground truth. Since CE loss does not assume any constraints on underlying distribution, it can, in principle, model any arbitrary distribution asymptotically with large enough data and model size. When the true underlying distribution  $P_{Y|X}$  is unimodal, this standard learning approach results in a classifier  $\hat{P}_{Y|X}$  that approximates  $P_{Y|X}$  and is itself unimodal assuming large enough data and model size. However, in practice with limited high-dimensional data and limited model capacity, the standard training is not guaranteed to reach the true "global minima" [11] or even adhere to the unimodality constraints that lead to contiguous prediction sets. Hence, there is a need for introducing the right inductive bias that truncates the hypothesis class to be searched while ensuring it is expressive enough to include the true hypothesis so as to allow for better learnability and generalization [27]. For our scenario, the set of unimodal distributions is the most natural hypothesis class. While previous works [4, 42] consider a set of parameterised unimodal distributions (e.g., based on Gaussian or Binomial distributions), we propose a construction that ensures the hypothesis class exactly maps to the set of unimodal distributions on the  $K$ -simplex.