SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Results on Image Benchmark Datasets: Mean and std. error is reported for 5 trials. Best mean results bolded.
MAE	Acc@1	Acc@2	Acc@3	|PSI|	CV%
HCI	V-CE	0.68 ± 0.03	54.3 ± 2.6	75.3 ± 3.1	88.9 ± 1.6	3.28 ± 0.14	24.4 ± 1.2
POE	0.66 ± 0.05	56.5 ± 1.8	76.5 ± 2.5	89.0 ± 2.1	3.1 ± 0.18	9.8 ± 1.2
SORD	0.65 ± 0.06	56.2 ± 2.8	77.1 ± 2.9	89.8 ± 2.6	2.96 ± 0.19	2.7 ± 1.1
AVDL	0.64 ± 0.08	56.8 ± 1.5	77.9 ± 2.4	89.8 ± 1.05	2.98 ± 0.11	2.1 ± 1.4
Binomial	0.68 ± 0.05	54.5 ± 1.2	75.8 ± 2.6	88.8 ± 1.8	3.01 ± 0.16	0
Binomial-temp	0.66 ± 0.04	55.5 ± 1.8	78 ± 2.2	90.1 ± 2.1	2.90 ± 0.11	0
Uni-loss	0.67 ± 0.09	54.5 ± 3.1	74.8 ± 2.5	88.1 ± 2.5	3.05 ± 0.38	5.1 ± 1.9
COPOC	0.65 ± 0.04	56.1 ± 2.0	79.8 ± 1.6	91.7 ± 2.8	2.66 ± 0.13	0
Adience	V-CE	0.57 ± 0.07	58.1 ± 1.6	80.8 ± 1.6	91.4 ± 2.3	4.82 ± 0.24	21.4 ± 2.2
POE	0.48 ± 0.05	60.5 ± 1.5	84.1 ± 2.0	93.9 ± 2.3	4.16 ± 0.18	12.8 ± 1.2
SORD	0.48 ± 0.06	59.9 ± 3.8	85.2 ± 2.9	94.3 ± 1.6	2.86 ± 0.09	3.7 ± 1.1
AVDL	0.49 ± 0.03	60.1 ± 2.5	85.3 ± 3.1	94.0 ± 1.1	2.95 ± 0.15	4.1 ± 0.9
Binomial	0.5 ± 0.04	60.0 ± 1.2	86 ± 1.8	95.4 ± 1.9	2.5 ± 0.06	0
Binomial-temp	0.48 ± 0.04	60.5 ± 2.1	86.4 ± 1.2	95.6 ± 1.3	2.45 ± 0.05	0
Uni-loss	0.64 ± 0.14	51.5 ± 7.9	80.8 ± 5.8	89.4 ± 3.5	3.14 ± 0.26	8.3 ± 2.3
COPOC	0.49 ± 0.04	61.0 ± 1.9	86 ± 1.5	96.1 ± 2.2	2.26 ± .06	0
Aesthetic	V-CE	0.29 ± 0.01	71.4 ± 1.6	94.6 ± 2.0	97.8 ± 0.8	1.96 ± 0.2	7.9 ± 0.2
POE	0.28 ± 0.05	72.1 ± 1.5	94.1 ± 1.1	98.0 ± 0.1	1.85 ± 0.11	7.85 ± 0.9
SORD	0.29 ± 0.02	72.0 ± 1.7	95.2 ± 1.9	98.3 ± 0.2	1.78 ± 0.09	0
AVDL	0.28 ± 0.03	72.2 ± 1.5	95.2 ± 1.8	98.5 ± 0.1	1.75 ± 0.05	0.3 ± 0.1
Binomial	0.31 ± 0.01	69.5 ± 0.7	93.1 ± 2.8	96.0 ± 0.9	1.83 ± 0.06	0
Binomial-temp	0.32 ± 0.04	69 ± 1.7	93.0 ± 1.6	96.2 ± 0.1	1.89 ± 0.09	0
Uni-loss	0.37 ± 0.14	66.8 ± 5.0	92.0 ± 3.8	97.4 ± 1.5	1.94 ± 0.24	2.1 ± 0.8
COPOC	0.28 ± 0.04	72.0 ± 1.3	95.9 ± 1.0	99.0 ± 0.2	1.70 ± .06	0
Retina-MNIST	V-CE	0.73 ± 0.02	52.2 ± 0.6	72.2 ± 0.1	86.0 ± 0.5	3.6 ± 0.08	9.8 ± 2.4
POE	0.73 ± 0.02	52.4 ± 0.4	72.5 ± 0.6	86.4 ± 0.8	3.4 ± 0.05	6.4 ± 2.8
SORD	0.71 ± 0.01	53.5 ± 0.3	70.5 ± 0.6	84.5 ± 0.9	3.2 ± 0.03	3.9 ± 1.1
AVDL	0.72 ± 0.02	53.0 ± 0.2	71.0 ± 0.4	84.6 ± 0.9	3.24 ± 0.04	3.8 ± 1.2
Binomial	0.71 ± 0.01	52.7 ± 0.2	69.7 ± 0.6	83.7 ± 0.8	3.33 ± 0.02	0
Binomial-temp	0.70 ± 0.02	53.0 ± 0.2	70.5 ± 0.5	84.0 ± 0.4	3.3 ± 0.02	0
Uni-loss	0.74 ± 0.05	52.0 ± 1.1	72.5 ± 0.6	84.5 ± 1.5	3.25 ± 0.1	4.2 ± 1.1
COPOC	0.71 ± 0.01	53.5 ± 0.2	72.5 ± 0.6	87.0 ± 0.3	3.03 ± 0.01	0
Result Discussion : We highlight the key takeaways from Table 6.
1. COPOC performs at par with SOTA baselines in terms of MAE and  $Acc@1$ .
2. Benefit of COPOC comes with improved gains in  $Acc@2$  and  $Acc@3$ . Apart from COPOC, there is no single method that performs consistently across the 4 datasets in terms of these metrics. For instance in HCI and Adience, Binomial-temp comes closest to COPOC, but on Aesthetic, both variants of Binomial severely under-perform whereas AVDL and SORD perform quite well and comes the closest to COPOC. In contrast, on Retina-MNIST, non-parametric models such as V-CE, POE, Uni-loss have  $Acc@k$  close to COPOC and beat other parametric models significantly. This shows that parametric distribution assumption in any underlying model fits the data well when the data is actually drawn from a similar distribution. Since most methods depend largely on the validity of the assumptions, the relatively unconstrained parameter-free nature of COPOC is more robust and allows it to consistently outperform across datasets.
3. There is a strong correlation between  $CV\%$  and PS size. This is expected because higher  $CV\%$  indicates more number of cases for which we had to predict a minimal contiguous super set, thus inflating the size of PS. Better unimodal fit by underlying model is bound to have lesser  $CV\%$  and and thus, shorter sets. Hence, COPOC again outperforms all other baselines across datasets in term of |PS| consistently. Although Binomial model variants has  $0CV\%$  due to its construction, it still produces larger sets than COPOC as seen in HCI and Adience. This can be because COPOC results in better unimodal fit which is also idicated by higher Acc@K.
4. Enforcing unimodality in training scheme in terms of soft-labels (SORD,AVDL) or in loss function (Uni-loss) or in embedding space (POE) does not necessarily translate to a unimodal distribution in test samples which is indicated by high  $CV\%$ .
5. Although  $V - CE$  in principle should have been able to model any underlying distribution, on high dimensional real-world datasets it fails miserably. This shows the need for injecting prior "bias" into training network like COPOC which aids the model in reaching the optima.
6. Uni-loss has issues with model convergence as it shows high variance across metrics for all datasets. This could be because its sensitive to  $\lambda$  hyperparameter that control the weightage between unimodality and mean-variance component of its loss function which is difficult to tune.
7. Datasets with higher accuracy results in shorter PS size in general, which is expected. For instance Aesthetic has lower PS size across methods compared to HCI or Retina-MNIST both having same number of class labels.
C.2 Implementation details of experiments on synthetic Data
For all the results on synthetic datasets presented in Sec.5.3, we employ same DNN network across all the methods for fair comparison. To be precise, we use 6 layer DNN architecture having 128 hidden dimensions with a dropout of 0.2. We use the same training paradigm as before - Adam optimizer with learning rate of 0.001 and batch size of 512 trained for 500 epochs ensuring convergence. We divide the data into  $70\%$  train and  $30\%$  test splits. We train our model 10 times for each independent split of the data. For each test set, we again randomly split into calibration for APS and evaluate |PSI| on final-test and repeat this 100 times to ensure convergence of PS. We use  $\phi = |x|$  and  $\psi = -|x|$  for COPOC. We report mean and standard error in Table 2.
C.3 Ablation study on the choice of  $\phi (\cdot)$  and  $\psi^E (\cdot)$  for COPOC
Although there can be many possible choices for  $\phi(\cdot)$  and  $\psi^{E}(\cdot)$  in the COPOC construction Eqn. 5, in practice not all choices leads to good model convergence. In this section, we perform a comparison of few common choices and present results in Table 7. We train the model for synthetic data D4 as described in Sec. 5.3. We train the model using CE loss with same model capacity and training paradigm as described in Appendix C.2. We report train CE loss and since we have access to true underlying distribution for D4 we report KL. Div. to measure goodness of model fit. Below we summarise few observations:
Table 7: Ablation study on implementation choice of  $\phi(\cdot)$  and  $\psi(\cdot)$  for COPOC. We report mean results across 10 trials.
Train loss	KL Div.
φ = ReLU, ψ = -|x|	3.89	0.24
φ = Softplus, ψ = -|x|	3.11	0.19
φ = x2, ψ = -|x|	3.48	0.2
φ = |x|, ψ = -|x|	1.64	0.04
φ = |x|, ψ = -x2	2.20	0.13
φ = |x|, ψ = -|x|0.5	1.89	0.1
1.  $\phi = R e L U$  maps most of  $[v_{k}]_{2}^{K}$  to zeroes which results in flat probability distribution for most of the data points while  $\phi = S o f t p l u s$  instead maps most  $[v_{k}]_{2}^{K}$  to very small values which again results in almost flat distribution for most points. With  $\phi = x^{2}$  we observed unusually large values for  $[v_{k}]_{2}^{K}$  resulting in unstable training.  $\phi = |x|$  gives a good balance as each  $[\eta_k]_2^K$  gets linearly mapped to  $[v_{k}]_{2}^{K}$ .
2.  $\psi = -|x|^2$  tends to over-emphasize higher probability classes in the model fitting while  $\psi = -|x|^{0.5}$  under-emphasizes them. Again since  $\psi = -|x|$  does a linear transformation of  $r_k$  on either side of the origin it gives a good balanced estimate of  $z_k$ .
C.4 Illustrative Comparison Between Unimodal and Non-Unimodal Model Fit
Figure 8: Predicted probability distributions of (naive) unconstrained DNN trained with vanilla Cross entropy loss (V-CE) (Top) and COPOC (bottom) on three datapoints from the public benchmark Adience-Age Estimation task dataset [13], which is annotated with 8 class labels with age groups as: 0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, and over 60 years old. Ground truth class is highlighted in red. We present a few difficult data instances where both models fail to predict the correct class as the top one. However COPOC with its unimodality bias is able to predict the correct class within top 2 or 3 predictions leading to better Accuracy@K compared to naive DNN.