Improving Deep Regression with Ordinal Entropy
Authors: Shihao Zhang
Linlin Yang
Michael Bi Mi
Xiaoxu Zheng
Angela Yao
Venue: ICLR 2023 poster (2023)
Keywords: regression
classification
entropy
depth estimation
counting
age estimation

Abstract: In computer vision, it is often observed that formulating regression problems as a classification task yields better performance. We investigate this curious phenomenon and provide a derivation to show that classification, with the cross-entropy loss, outperforms regression with a mean squared error loss in its ability to learn high-entropy feature representations. Based on the analysis, we propose an ordinal entropy loss to encourage higher-entropy feature spaces while maintaining ordinal relationships to improve the performance of regression tasks. Experiments on synthetic and real-world regression tasks demonstrate the importance and benefits of increasing entropy for regression.

IMPROVING DEEP REGRESSION WITH ORDINAL EN- TROPY
Shihao Zhang', Linlin Yang', Michael Bi Mi,, Xiaoxu Zheng?, Angela Yao National University of Singapore, Singapore, 2 Huawei International Pte Ltd, Singapore
ABSTRACT
In computer vision, it is often observed that formulating regression problems as a classification task yields better performance. We investigate this curious phenomenon and provide a derivation to show that classification, with the cross- entropy loss, outperforms regression with a mean squared error loss in its ability to learn high-entropp feature representations. Based the analysis, we propose an ordinal entropy regularizer to encourage higher-entropy feature spaces while maintaining ordinal relationships to improve the performance of regression tasks. Experiments on synthetic and real-world regression tasks demonstrate the impor- tance and benefits of increasing entropy for regression. Code can be found here: https:/ / //gittttt gighhb.........g com htep//eeedyyleeeeee /artaaoao/oooaaaaanntrrroroo
INTRODUCTION
Classification and regression are two fundamental tasks of machine learning. The choice between the two usually depends on the categorical or continuous nature of the target output. Curiously, in computer vision, specifically with deep learning. it is often preferable solve regression-type problems as classification tasks. A simple and common way is to discretize the continuous labels; each bin then treated as class. After converting regression into classification, the ordinal information of the target space is lost. Discretization errors are also introduced to the targets. Yet for diverse set of regression problems, including depth estimation (Cao et al., 2017). age estimation (Rothe et al., 2015), crowd-counting (Liu et al., 2019a) and keypoint detection (Li et al., 2022), classification yields better performance.
The phenomenon of classification outperforming regression on inherently continuous estimation tasks naturally begs the question of why. Previous works have not investigated the cause, although they hint at task-specific reasons. For depth estimation, both Cao et al. (2017) and Fu et al. (2018) postulate that it is easier to estimate a quantized range of depth values rather than one precise depth value. For crowd counting, regression suffers from inaccurately generated target values (Xiong & Yao, 2022). Discretization helps alleviate some of the imprecision. For pose estimation, classification allows for the denser and more effective heatmap-based supervision (Zhang al., 2020; Gu et al., 2021; 2022).
Could the performance advantages of classification run deeper than task-specific nuances? In this work, we posit that regression lags in its ability to learn high-entropy feature representations. We arrive at this conclusion by analyzing the differences between classification and regression from mutual information perspective. According Shwartz-Ziv & Tishby (2017), deep neural networks during learning aim to maximize the mutual information between the learned representation Z and the target Y. The mutual information between the two can be defined as I(Z; Y) H(ZZ) 7((Z I(Z;Y Y) is large when the marginal entropy H(Z) is high, i.e., features Z are spread as possible, and the conditional entropy H(ZYY) low, i.e., features of common targets are as close as possible. Classification accomplishes both objectives (Boudiaf al., 2020). This work, shows through derivation that regression minimizes H H( but ignores H (). Accordingly, the learned representations Z from regression have lower marginal entropy (see Fig. (a)). A t-SNE visualization of the features (see Fig. (bb) (c)) confirms that features learned by classification have more spread than features leamndd by regression. More visualizations shown in Appendix B.
key contribution,
The difference in entropy between classification and regression stems from the different losses. We postulate that the lower entropy features learned by T.2 losses in regression explain the performance

---

260
240
220
dasssiic 116 disc
disefical
200
dlsss Fost
360
20
spach
(a) Entropy of feature space
(b) Regression
(c) Classification
Figure 1: Feature learning of regression versus classification for depth estimation. Regression keeps features close together and forms an ordinal relationship, while classification spreads the features (compare (b) vs. (c) ), leading to higher entropy feature space. Features are colored based on their predicted depth. Detailed experimental settings are given in Appendix B.
gap compared to classification. Despite its overall performance advantages, classification lags in the ability capture ordinal relationships. As such, simply spreading the features for regression to emulate classification will break the inherent ordinality of the regression target output.
To retain the benefits of both high entropy and ordinality for feature learning, we propose, as second contribution, an ordinal entropy regularizer for regression. Specifically, we capture ordinal relationships as a weighting based on the distances between samples in both the representation and target space. Our ordinal entropy regularizer increases the distances between representations, while weighting the distances to preserve the ordinal relationship.
The experiments various regression tasks demonstrate the effectiveness of our proposed method. Our main contributions are three-fold:
To our best knowledge, we are the first to analyze regression's reformulation as a classifi- cation problem, especially in the view of representation learning. We find that regression lags in its ability to learn high-entropy features, which in turn leads to the lower mutual information between the learned representation and the target output.
Based theoretical analysis, design ordinal entropy regularizer learn high- entropy feature representations that preserve ordinality.
Benefiting from our ordinal entropy loss, our methods achieve significant improvement on synthetic datasets for solving ODEs and stochastic PDEs well as real-world regression tasks including depth estimation, crowd outing and age estimation.
RELATED WORK
Classification for Continuous Targets. Several works formulate regression problems as classification tasks to improve performance. They focus on different design aspects such as label discretization and uncertainty modeling. To discretize the labels, Cao al. (2017); Fu al. (2018) and Liu et al. (2019a) convert the continuous values into discrete intervals with pre-defined interval width. To improve class flexibility, Bhat al. (2021) followed up with an adaptive bin-width estimator. Due to inaccurate or imprecise regression targets, several works have explored modeling the uncertainty of labels with classification. Liu et (2019a) proposed estimating targets that fall within a certain interval with high confidence. Tompson et al. (2014) and Newell et al. (2016) propose modeling the uncertainty by using a heatmap target in which each pixel represents the probability of that pixel being the target class. This work, instead of focusing task-specific designs, explores the difference between classification and regression from learning representation point of view. By analyzing mutual information, reveal previously underestimated impact of high-entropy feature spaces.
Ordinal Classification. Ordinal classification aims to predict ordinal target outputs. Many works exploit the distances between labels (Castagnos al., 2022; Polat et al., 2022; Gong et al., 2022) to
2

---

Ordinal
Entopy
Tightness (ZIV)
Diversity H()
MSE
Discretization
Entropy
Features to Cj
Zej
Features corresooning to Cy
(a) Illustration frameworks
(b) Tightness and diversity ordinal entropy
Illustration of (a) regression and classification for continuous targets, and the of our ordinal entropy for regression, (b) the pull and push objective of tightness and diversity on the feature space. The tightness part encourages features be close to their feature centers while the diversity part encourages feature centers to be far away from each other.
Figure 2:
preserve ordinality. Our ordinal entropy regularizer also preserves the ordinality by exploiting the label distances, while it mainly aims at encouraging a higher-entropy feature space.
Entropy. The entropy of random variable reflects its uncertainty and can be used to analyze and regularize a feature space. With entropy analysis, Boudiaf et al. (2020) has shown the benefits of the cross-entropy loss, i.e., encouraging features be dispersed while keeping intra-class features compact. Moreover, existing works (Pereyra et al., 2017; Dubey et al., 2017) have shown that many regularization terms, like confidence penalization (Pereyra et al., 2017) and label smoothing (Müller et al., 2019), are actually regularizing the entropy of the output distribution. Inspired by these works, explore the difference in entropy between classification and regression. Based entropy analysis, we design an entropy term (i.e., ordinal entropy) for regression and bypass explicit classification reformulation with label discretization.
A MUTUAL-INFORMATTON BASED COMPARISON ON FEATURE LEARNING
3.1 PRELIMINARIES
Suppose we have dataset {X, Y} with N input data X = [ and their corresponding labels Y {yr -- In a typical regression problem for computer vision, x, is an image or video, while U: E RD takes a continuous value in the label space y. The target of regression is to recover y; by encoding the image to feature z; (xx,) with encoder and then mapping to predicted target
; = fz,), with a regression function f(.) parameterized by 0. The encoder yp and are learned by
(v: minimizing a regression loss function such as the mean- mean-squared error Lmse = # E - -.)
To formulate regression a classification task with K classes, the continuous target Y can be converted to classes y = ( with some discretizing mapping function, where y E [0, K - categorical target. The feature z; is then mapped the categorical target yf classifier (-) parameterized by w. The encoder and w are learned by minimizing the cross- expuiz.
Iw (z,) with
entropy loss LoE symmetry of the two formulations.
Σin log (z:), where (gn(Z)))) Σ, expwi Fig. 2(a) visualizes the
The entropy of a random variable can be loosely defined the amount of "information" associated with that random variable. One approach for estimating entropy H(Z) for a random variable Z is the NeanNN entropy estimator (Faivishevsky & Goldberger, 2008). It can accommodate higher- dimensional Zs, and is commonly used in high-dimensional space (Faivishevsky & Goldherger, 2010). The meanNN estimator relies on the distance between samples to approximate p(Z). For D-dimensional Z, it is defined as
D
AA(Z) N(N -1)
log ||z, 2211 const.
3

---

3.2 FEATURE LEARNING WITH CROSS-ENTRON LOSS (CLASSIHICATION)
Given that mutual information between target Y and feature is defined as Z( Y)) H(Z) H(Z)YC), follows that [ZZ; can be maximized by minimizing the second term H H and maximizing the first term H()). Boudiaf et al. (2020) showed that minimizing the cross-entropy loss accomplishes both by approximating the standard cross-entropy loss LeE with pairwise cross-entropy loss CPCE. PPC serves as a lower bound for LCE. and can be defined as
N Σ
N
K
N
K
LPCE = 2A 1= JEly5-yi yfl
Σ
Zjz
+
log
exp
p5k
1
Σ 1=1
ck (2)
N
AN
2A
Tightees H(Z(YC)
Diversity & 'H(2)
where À E IR is make sure LeE is a convex function with respect to Intuitivelly, LrCE can be understood as being composed of both a pull and push objective more familiar in contrastive learning. We interpret the pulling force closely aligns the feature vectors within a given class, This results in features clustered according their class, i.e., lower conditional entropy H H()(yC). The pushing force from the second term encourages lower z 2 while forcing classes' centers c to be far from the origin. This results in diverse features that are spread apart, or higher marginal entropy H(Z). Note that the tightness term corresponds to the numerator of the Softmax function in LCF, while the diversity term corresponds to the denominator.
tightness term. encourages higher values for z z, and
3.3 FEATURE LEARNING WITH AN LASE LOss (REGRESSION)
In this work, we find that minimizing Lise. as done in regression, is proxy for minimizing Y without increasing H()). Minimizing Lmse does not increase the marginal entropy H(2) and therefore limits feature diversity. The link between classification and regression is first established below in Lemma 1. We assume that we are dealing with a linear regressor, as is commonly used in deep neural networks.
Lemma 1 We given dataset {x,, Vi N. where x; is the input and y, E y is the label, and linear regressor So(-) parameterized by 0. Let 24 denote the corresponding feature. Assume that the label space y is discretized into bins with maximum width n, and e₁ is the center of the hin to which y belongs. Then for any > 0, there exists >> O such that:
N
|mms
1
10 1 - cs 2n
20 2, - C. - 13/1 <E
(3)
N 1
The detailed proof of Lemma provided in Appendix A.
The result of Lemma 1 says that the discretization error from replacing regression target y: with c; can be made arbitrarily small the bin width 7j is sufficiently fine. As such, the Lmse can be directly approximated by the second term of Eq. iee, fisse R N ΣN (0T - c2)2. With this result, it can be proven that minimizing Luse is proxy for minimizing H(Z
Theorem Let 2e; denote the center of the features corresponding to bin center co and Φs be the angle between and z; Ze,. Assume that 0 is normalized, (Z* ~ N(ze, 1), where Z is the distribution of Ze, and that cos 0; is fixed. Then, minimizing Lmse can be seen proxy for minimizing H Y( without increasing H(Z).
Proof
Based on Lemma 1, we have
N
Lisee
Σ
O0 (0T(, - ==
1 N
Σ (11011|2, - -zc, cos e2
(4)
N
1 Nv
Σ 110171122 - 20,11 cos" coo8 0i x
N Σ || - Zc, 112.
N
4

---

Note, Ze; exist unless 9 0 and G + 0. Since it is assumed that Z ~ MM I), the term
I|: Ze, 21 can be interpreted as conditional cross entropy between Z and Z" it satisfies
H(2:291) 1
log(e - const
(5)
22 2e 112
where denotes equal to, up to multiplicative and an additive constant. The & denotes Monte Carlo sampling from the Z|Y distribution, allowing us to replace the expectation by the mean of the samples. Subsequently, we can show that
N
Cise O
Σ
2z Zo, 11.11 _ H(ZZZZ9) = H(Z))) + (Z)(211))))
(6)
N
The result in Eq. 6 shows that N Σ1 ||z Zz; 11 is an upper bound of the tightness term in mutual information. If (ZlY) N (Ze I). then Dx (2 is equal to and the bound is tight i... N Σi ||z, Zc, 12, 2 H(Z) Y). Hence, minimizing Lmse is a proxy for minimizing H(
Apart from H(2 YY), the relation in Eq. 6 also contains the KL divergence between the two conditional distributions P(Z and PYZ where Z are feature centers of Z. Minimizing this divergence will either force Z closer to the centers 2", move the centers Z around. By definition, however, the cluster centers D cannot expand beyond Z's coverage, so features Z must shrink to minimize the divergence. As such, the entropy H HZ) will not be increased by this term.
draw the conclusion that regression, with an MSE loss, overlooks the marginal entropy H(Z) and results in a less diverse feature space than classification with cross-entropy loss.
Based Eq. 2 and Theorem
It is worth mentioning that the Gaussian distribution assumption, i.e., Z AZze, I), is standard in the literature when analyzing features (Yang et al., 2021a; Salakhutdinov al., 2012) and entropy (Misra et al., 2005), and cos o; is constant value at each iteration.
4 ORDINAL ENTROPY
Our theoretical analysis in Sec. shows that learning with only the MSE loss does not increase the marginal entropy H (Z) and results in lower feature diversity. To remedy this situation, propose novel regularizer to encourage a higher entropy feature space.
Using the distance-based entropy estimate from Eq. 1, distances between feature centers Ze, to maximize the entropy of the feature space. Ze, are calculated by taking mean over all the features z which project the same yı- Note that feature spaces are unbounded, the features z must first be normalized, and below, we assume all the features z are already normalized with an L2 norm:
then minimize the the negative
- M(M i=1
1z0. 20yl12 x -H(Z),
where M is the number of feature centers in batch of samples sampled subset sampled from a batch. We consider each feature as a feature center when the continuous labels the dataset are precise enough.
While the regularizer C/ indeed spreads features to a larger extent, it also breaks ordinality in the feature space (see Fig. 3(b)). As such, opt weight the feature norms in L'2 with Wij. where wij are the distances in the label space )::
La M(M
Wayl |Ze, Zey l12. where wi = =1ly/ 33
(8)
1
5

---

(a) Regression
(b) Regression + (c) Regression +La (d) Regression + + Lt
Figure 3: t-SNE visualization of features from the depth estimation task. (b) Simply spreading the features (C') leads to a higher entropy feature space, while the ordinal relationship is lost, (c) By further exploiting the ordinal relationship in the label space (Ca), the features are spread and the ordinal relationship is also preserved. (d) Adding the tightness term (C₂) further encourages features close to centers.
As shown in Figure 3(c), La spreads the feature while also preserve preserving ordinality, Note that C is special case of La when Wij all equal.
To further minimize the conditional entropy H(()Y), we introduce an additional tightness term that directly considers the distance between each feature z; with its centers Ze, in the feature space:
N No
C,
z Z. 12,
where N₂ is the batch size. Adding this tightness term further encourages features close to its centers (compare Fig. 3(c) with Fig. 3(d). Compared with features from standard regression (Fig. 3()),, the features in Fig. 3(d) are more spread, i.e., the lines formed by the features are longer.
We define the ordinal entropy regularizer as Loe La + L., with diversity term Ca and tightness term Lt. Loe achieves similar effect as classification in that it spreads Ze, while tightening features 2, corresponding to Ze,- Note, if the continuous labels of the dataset are precise enough and each feature is its own center, then our ordinal entropy regularizer will only contain the diversity term, i.e. fae = La. We show our regression with ordinal entropy (red dotted arrow) Fig. 2(a).
The final loss function Liotat is defined as:
frotal = L. + Aaha + Xehi,
(10)
where L. is the task-specific regression loss and da and de are trade-off parameters.
5 EXPERIMENTS
5.1 DATASETS, METRICS & BASELINE ARCHITECTURES
We conduct experiments on four tasks: operator learning, synthetic dataset and three real-world regression settings of depth estimation, crowd counting, age estimation.
For operator learning, we follow the task from DeepONet (Luu al., 2021) and use a two-layer fully connected neural network with 100 hidden units. See Sec. 5.2 for details on data preparation.
For depth estimation, NYU-Depth-v2 (Silberman al., 2012) provides indoor images with the corresponding depth maps pixel resolution 640 480. We follow (Lee et al., 2019) and use ResNe150 (He et al., 2016) as our baseline architecture unless otherwise indicated. We use the train/test split given used by previous works (Bhat et al., 2021; Yuan et al., 2022), and evaluate with the standard metrics of threshold accuracy 81, average relative error (REL), root mean squared error (RMS) and average log10 error.
For crowd counting, we evaluate on SHA and SHB the ShanghaiTecch crowd counting dataset (SHTech) (Zhang et al., 2015). Like previous works, adopt density maps labels and evaluate with mean absolute error (MAE) and mean squared error (MSE). We follow CSRNet with ResNet50 the regression baseline architecture.
Li et al. (2018) and use

---

Table 1: Ablation studies linear and nonlinear operators learning with synthetic data and depth
estimation on NYU-Depth-v2. For operator learning, we report results as mean + standard variance over 10 runs. Bold numbers indicate the best performance.
Linear
Nonlinear
NYU-Depth-v-
Method
Cmmc(x1009)
(x10 4)
0,
REL L
RMS
logi0
Bascline Baseline + C/
3.0 0,93
2.5 2.0
0.793
0.148
0.302
0,064
2.2 + 0.42
0.8 + 0.3
0.795
0.147
0.505
0.063
Baseline- + LJ
1.6 = 0.34
0.5 = 0.3
0.808
0.144
0.483
0.061
Baseline + La | Le
0.811
0.143
0.478
0.060
w/ cosine distance
1.7 + 0.33
0.6 + 0.5
0.806
0.147
0.488
0,061
w/o normalization
5.3 + 1.10
15.5 + 0.4
0.790
0.153
0.510
0.064
Waj
11: Mj
2.0 + 0.60
0.4 + 0.2
0.800
0.149
0.498
0.063
W 1. Mr
1.8 + 0.50
0.5+ 0.2
0.799
0.147
0.496
0.062
For age estimation, we use AgeDB-DIR (Yang et al., 2021b) and also implement their regression baseline model, which uses ResNet-50 backbone. Following Liu et al. (2019b), we report results on three disjoint subsets (i.e., .Many, Med, and Few), and also overall performance (i.e. ALL). We evaluate with MAE and geometric mean (GM).
Other Implementation Details:
We follow the settings of previous works DeepONet (Lu et al., 2021) for operator learning, Adabins (Bhat et al., 2021) for depth estimation, CSRNet (Li et al., 2018) for crowd counting, and Yang et al. (2021b) for age estimation. See Appendix D for details.
Xa and 0₂ Ad, A the same value of 0.001, 1, 10, 1, for operator learning, depth estimation, crowd counting and age estimation, respectively.
empirically based on the scale of the task loss Am We use the trade-off parameters
5.2 LEARNING LINEAR AND NONLINEAR OPERATORS
We first verify our method on the synthetic task of operator learning. In this task, an (unknown) operator maps input functions into output functions and the objective is to regress the output value. We follow (Lu et al., 2021) and generate data for both linear and non-linear operator. For the linear operator, we aim to learn the integral operation G:
G ((() s(x) = 1." u(()dd, E [0, 1],
(11)
where u the input function, and is the target function. The data is generated with a mean- zero Gaussian random field function space: x2 / is the radial-basis function kernel with length-scale param- eter 1 0.2. The function is represented by the function values of
G(0, (z1, ø2)), where the covariance kernel
ku .22) ) = ear((-1lan
100 fixed locations {21. T22 ... ,Tm The data is generated as ([u, y), Gy)) where is sampled from the domain of G()). We randomly sample 1k data as the training set and test on the testing set with 100k samples.
For the nonlinear operator, we aim to learn the following stochastic partial differential equation, which maps b(I;w) of different correlation lengths E [1, 2] a solution u(z;w):
div(e Vu(a; w)) f(x),
(12)
where x E (0, 1), w from the random space with Dirichlet boundary conditions uu0) u(1) = 0. and f(x) 10. The randomness comes from the diffusion coefficient Mzxw) The function b(x:w) CP(ba cov(x11,;) is modelled as a Gaussian random process GP, with mean bo(x)= = 0 and cov(r1, x2) = o² -eeepplllaa
22112/2(²)) We randomly sample 1k training samples
and 10k test samples.
For operator learning, we set Lise as the task-specific baseline loss for both the linear and non-linear operator. Table shows that even without ordinal information, adding the diversity term i.e., L' to Luse already improves performance. The best gains, however, are achieved by incorporating the weighting with C,. which decreases Luse by 46.7% for the linear operator and up to 80% for the more challenging non-linear operator. The corresponding standard variances are also reduced significantly.
7

---

Table 2: Quantitative comparison of depth estimation results with NYU-Depth--2. Bold numbers indicate the best performance.
Method
ò1 t
REL +
RMS +
logio +
Laina al. (Laina et al., 2016)
0.811
0.127
0.573
0.055
DORN (Fu et al., 2018)
0.828
0.115
0.509
0.051
BTS (Lee al., 2019)
0.885
0.110
0.392
0.047
Adabins (Bhat et al., 2021)
0.903
0.103
0.364
0.044
NeW-CRFs (Yuan et al., 2022)
0.922
0.095
0.334
0.041
NeW-CRFs + Lu + fr
0.932
0.089
0.321
0.039
Baseline (ResNet-50)
0.793
0.148
0.502
0.064
Baseline (ResNet-50)- La
0,808
0,144
0.483
0.061
Baseline (ResNet-50)- La Lt
0.811
0.143
0.478
0.060
Table 3: Results on SHTech. Bold numbers indicate the best performance.
MAE 4
MSE 4
Method
SHA
SHB
SHA
SHB
Regression Baseline (Li et al, 2018)
68.2
10.6
115.0
16.0
+Ld
66.9
9.1
107.5
14.7
+La + Le
65.6
9.1
105.0
14.5
Note that we do not verify L. on operator learning due to the high data precision on synthetic datasets, it is difficult sample points belonging to the same Ze. Adding Lt, however, is beneficial for the three real-world tasks (see Sec. 5.3).
5.3 REAL-WORLD TASKS: DEPTH ESTIMATION, CROWD COUNTING & AGE ESTIMATION
Depth Estimation: Table 2 shows that adding the ordinal entropy terms boosts the performance of the regression baseline and the ttate-of-the-e regression method NeW-CRFs (Yuan et al., 2022). NeW-CRFs with ordinal entropy achieves the highest values for all metrics, decreasing and REL errors by 12.8% and 6.3%, respectively. Moreover, higher improvement can be observed when adding the ordinal entropy into simpler baseline, i.e., ResNet-50.
Crowd Counting: Table shows that adding La and L, each contribute to improving the baseline. Adding both terms has the largest impact and for SHB, the improvement is up to 14.2% MAE and 9.4% on MSE.
Age Estimation: Table 4 shows that with La achieve significant 0.13 and 0.29 overall improve- ment (i.e., ALL) on MAE and GM, respectively. Applying L, achieves further overall improvement over La only, including 04 MAE and 0.04 GM.
5.4 ABLATION STUDIES
Ablation results on both operator learning and depth estimation are shown in Table and Figure 4.
Ordinal Relationships: Table 1 shows that using the unweighted diversity term 'Baseline+ which ignores ordinal relationships, worse than the weighted version 'Baseline+L.'' for both operator learning and depth estimation.
Feature Normalization: As expected, normalization is important, as performance will decrease (compare 'w/o normalization' to "Baseline&..' in Table 1) for both operator learning and depth estimation. Most interestingly, normalization also helps to lower variance for operator learning.
Feature Distance [Z.e, Ze, |l: We replace the original feature distance L.2 with cosine distance (see 'w/ cosine distance') and the cosine distance is slightly worse than L2 for all the cases.
Weighting Function way The weight defined in Eq. 8 is based
L2 distance. Table 1 shows that L2 is best for linear operator learning and depth estimation but is slightly worse than yı for nonlinear operator learning.
Wij = l|
8

---

Table 4: Results AgeDBBDDIR Bold numbers indicate the best performance.
MAE
GM
Method
ALL
Many
Med.
Few
ALL
Many
Med.
Few
Baseline (Yang et al., 2021b)
7.77
6.62
9.55
13.67
5.05
4.23
7.01
10.75
+Ld
7.60
6.79
8.55
12.70
4.76
4.15
5.95
9.60
+La Lt
7.46
6.73
8.18
12.38
4.72
4.21
5.36
9.70
0.07
0.004
-10
Diersty
teatoon 1258 cless
1igttem
0907
-10
Samplix
(a) MSE with number of samples
(b) MSE with Aa
(c) Entropy of different methods
Figure 4: Based the linear operator learning and depth estimation, show (a) the effect of the number of samples on MSE, (b) the performance analysis with different X and (c) the entropy curves of different methods during testing. The results for the nonlinear operator learning are given in Appendix C.
Sample Size (M) In practice, we estimate the entropy from a limited number of regressed samples and this is determined by the batch size. For certain tasks, this may be sufficiently large, e.g., depth estimation (number of pixels per image x batch size) or very small, e.g., age estimation (batch size). We investigate the influence of M from Eq. 8 on linear operator learning (see Fig.4(b)). In the most extreme case, when M 2, the performance slightly better than the baseline model (2.7 x 10 vs. 3.0 10-3), suggesting that our ordinal regularizer terms are effective even with 2 samples. As M increases, the MSE and its variance steadily decrease as the estimated entropy likely becomes more accurate. However, at certain point, MSE and variance start to increase again. This behavior is not surprising; with too many samples under consideration, likely becomes too difficult to increase the distance between pair of points without decreasing the distance to other points, i.e., there not sufficient room to maneuver. The results for the nonlinear operator learning are given in Appendix C.
Hyperparameter Aa and A;: Fig. 4(b) plots the MSE for linear operator learning versus the trade-off hyper-parameter Xa applied to the diversity term La. Performance remains relatively stable up to 10-2 after which this term likely overtakes the original learning objective Luse and causes MSE to decrease. The results for the nonlinear operator and analysis on A are given in Appendix C and E.
Marginal Entropy H(2): We show the marginal entropy of the testing set from different methods during training (see Fig. 4(c)). We can see that the marginal entropy of classification is always larger than that of regression, which has downward trend. Regression with only diversity achieves the largest marginal entropy, which verifies the effectiveness of our diversity term. With both diversity and tightness terms, as training goes, its marginal entropy continues to increase and larger than that of regression after the 13th epoch. More experiment results can be found in Appendix F.
CONCLUSION
In this paper, we dive deeper into the trend of solving regression-type problems as classification tasks by comparing the difference between regression and classification from mutual information perspective. We conduct theoretical analysis and show that regression with an MSE loss lags in its ability to learn high-entropy feature representations. Based on the findings, we propose ordinal entropy regularizer for regression, which not only keeps an ordinal relationship in feature space like regression, but also learns high-entropy feature representation like classification. Experiments different regression tasks demonstrate that our entropy regularizer can serve as plug-in component for regression- -based methods to further improve the performance.

---

Acknowledgement. This research / project is supported by the Ministry of Education, Singapore, under its MOE Academic Research Fund Tier 2 (STEM RIE2025 MOE-T2EP20220-00155,
REFERENCES
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4009-4018, 2021.
Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European conference computer vision, pp. 548-564. Springer, 2020.
Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 28(1 28(11):314--3182 2017.
François Castagnos, Martin Mihelich, and Charles Dognin. A simple log-based loss function for ordinal text classification. In Proceedings of the 29th International Conference on Computa- tional Linguisties, Gyeongju. Republic of Korea, October 2022. International Committee on Computational Linguisties. URL https::t////tc /aaacaaantttt ogy org/ 2022. co iin--nnn-.. 1.007
Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, Iyad Rahwan, and Nikhil Naik. Regularizing prediction entropy enhances deep learning with limited data. In Proceedings of the Neural Information Processing Systems (NIPS), 2017.
Lev Faivishevsky and Jacob Goldberger. Ica based on smooth estimation of the differential entropy. Advances in neural information processing systems, 21, 2008.
Lev Faivishevsky and Jacob Goldberger. Nonparametric information theoretic clustering algorithm. In ICML., 2010.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002-2011, 2018.
Yu Gong, Greg Mori, and Frederick Tung. RankSim: Ranking similarity regularization for deep imbalanced regression. In International Conference on Machine Learning (ICML), 2022.
Kerui Gu, Linlin Yang, and Angela Yao. Removing the bias integral pose regression. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11067-11076, 2021.
Kerui Gu, Linlin Yang, and Angela Yao. Dive deeper into integral pose regression. In International Conference on Learning Representations, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international confer- on 3D vision (3DV), pp. 239 248. IEEE, 2016.
Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and II Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10326. 2019.
Yanjie Li. Sen Yang, Peidong Liu. Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang, and Shu-Tao Xia. Simee: simple coordinate classification perspective for human pose estimation. 2022.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1091-1100, 100, 2018.
10

---

Liang Liu, Hao Lu, Haipeng Xiong, Ke Xian, Zhiguo Cao, and Chunhua Shen. Counting objects by blockwise classification. IEEE Transactions on Circuits and Systems for Video Technology. 30(10): 3513-3527, 2019a.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long- long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019b.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-25) 2021.
Neeraj Misra, Harshinder Singh, and Eugene Demehuk. Estimation of the entropy of multivariate normal distribution. Journal of multivariate analysis, 92(2):324-342, 2005.
Rafael Müller, Simon Korbllith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pp. 483-499, Springer, 2016.
Gabriel Pereyra, George Tucker, Jan Chorowski, Fukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548. 2017.
Gorkem Polat, IIkay Ergenc, Haluk Tarik Kani. Yesim Ozen Alahdab, Ozlen Atug, and Alptekin Temizel. Class distance weighted cross-entropy loss for ulcerative colitis severity estimation. In Guang Yang, Angelica Aviles-Rivero, Michael Roberts, and Carola-Bibiane Schönlieb (eds.), Medical Image Understanding and Analysis, pp. 157-171, Cham, 2022. Springer International Publishing.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Dex: Deep expectation of apparent age from single image. In Proceedings of the IEEE international conference on computer vision workshops, pp. 10-15,2011
Ruslan Salakhutdinov, Joshua Tenenbaum, and Antonio Torralba. One-shot learning with a hierar- chical nonparametric bayesian model. In Proceedings of ICML Workshop Unsupervised and Transfer Learning, pp. 195-206. JMLR Workshop and Conference Proceedings. 2012.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 7000000000,,,. 2017.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. European conference on computer vision, pp. 746-760. Springer, 2012.
Jonathan Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training a convolutional network and a graphical model for human pose estimation. Advances in neural information processing systems, 27, 2014.
Haipeng Xiong and Angela Yao. Discrete-constrained regression for local counting models. 2022.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. JCLR, 2021a.
Yuzhe Yang. Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbal- anced regression. In International Conference Machine Learning, pp. 11851. PMLR, 2021b.
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. New erfs: Neural window fully-connen collycconnected crfs for monocular depth estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022.

---

Cong Zhang, Hongsheng Li, Xiaogang Wang. and Xiaokang Yang. Cross scene crowd counting via deep convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 83,-841, 2015.
Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-wware coordinate represen- tation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7093-7102, 2020.
12

---

Appendix
A
PROOF OF LEMMA
Proof
1
Luse
(0 y1)²
11
1 ((0T2: 2; (i) + (c: -vi))²
"1
1
Σ (0 24 - c;)2 + Σ (l:(20) - yi))
"
n
Since |4 - Vil S 2' we have:
1 500T -c))11
mmse
20 z, Cs Vil
71
For
2nz 21 22 2.
9v
n
C
10 2. -G) <E Σ
11
B VISUALIZATION
Experimental setting We train the regression and classification models on the NYU-Depth-v2 dataset for depth estimation. We modify the last layer of ResNet-50 model to convolution operation with kernel size 1 x 1, and train the modified model with Lime as our regression model. For the classification models, we modify the last layer of two ResNet-50 models to output N₂ channels, where N₂ is the number of classes, and train the modified models with cross-entropy. The classes are defined by uniformly discrete ground-trute depths into N₂ bins. The entropy feature space is estimated using Eq pixel-wise features over the training and test set of NYU-Depth-v2. After training, visualize the pixel-wise features of an image from the test set using t-distributed stochastic neighbor embedding (t-SNE), and features colored based on their predicted depth.
The visualization results are shown in Figure A. We exploit three entropy estimators to estimate the entropy of feature space H(Z). Entropy in the first of Figure A is estimated with the meanNN entropy estimator Eq. 1. Entropy in the second row is also estimated with the meanNN entropy estimator, where the input is the features normalized with the L2 norm. Entropy the third row is estimated with the diversity part of our ordinal entropy Eq. 8.
We make several interesting observations from the visualization results: (1) On both training and testing, compared with classification, regression always has a lower estimated entropy based on all three entropy estimators; (2) benefiting from the diversity part of our ordinal entropy La, regression produces higher entropy space than the classification model; (3) L, with the tightness term C, can result in feature space with similarly estimated entropy compared with classification.
C RESULTS ON THE NONLINEAR OPERATOR LEARNING PROBLEM
We analyze the effect of the number of samples MSE loss and the performance with X on the nonlinear operator learning task. The results are shown in Figure B. The results corroborate the conclusions derived from the linear task.
13

---

125.
repression tlarion
300
eltartaciin 255 caai flawery "pperie"
250
200
230
375
Brunity
150
150
(a) Training
(b) Testing
(c) Testing
010055
cassifaciinn (94 dasa)
Consiiicatio (18 deeo
regressaes
Nggrssicnn daset calton (16 cwa
dasal callon (34 s6a1
Coudiatte Joou Grandry
riscol raliss (NN0)
-20
eooch
ep3
mach
(d) (faatrn normalized)
(c) Testtng normalized)
(f) Testtng normalized)
cieiifaatinn 1230 diass
inression
oggessioon
1.
Clazaficction glandiiction (255 cima)
fraltor (258 deed
Eas:
(g) Training(ordinal entropy)
(h) Testing(ordinal entropy)
(i) Testing(ordinal entropy)
Figure A: Visualization results with different entropy estimators on training and testing set.
0.06
0.025
MEC
0.0100
0.075
00200
0.00
00225
10a
10a
10
10
10
10
10
10
10
Samples
Weight
(a) MSE with number of samples
(b) MSE with Xa
Figure B: Based on the nonlinear operator learning problem, we show (a) the effect the number of samples MSE loss, (b) the performance analysis with different Aa-
D EVALUATION METRICS
Here we introduce the definition of the evaluation metrics for depth estimation, crowd counting, and age estimation.
14

---

Depth Estimation. We denote the predicted depth at position p as yp and the corresponding ground truth depth as y/- the total number of pixels is n. The metrics are: 1) threshold accuracy dj _ %
of Up, s.t. max Ny V of yy
< t1, where 1 = 1.25; 2) average relative error (REL): Σp 10p 1,,%% 3)
By
root mean squared error (RMS): v Ep (By v/)2; 4) average (log10 error): 1 Ep logia
log1o (1p)1-
Crowd Counting. Given images for testing, Vi. 3/ are the estimated count and the ground truth for the i-th image, respectively. We exploit two widely used metrics measurements: 1) Mean Absolute
Error (MAE)= N/ Σ Me y/ll, and 2) Mean Squared Error (MSE)
Σ y -/1?.
Age Estimation. Given N images for testing, Yi and yf are the i-th prediction and ground-truth, respectively. The evaluation metrics include 1)MAE: N Ei 1 - -Vll. and 2)Geometric Mean (GM):: (TI 1y - v/l) +
E EFFECT OF X
We analyze the effect of de with an ablation study on the age estimation with Age-DB-DIR. Table shows that the final performance is not sensitive the change of λ and La is effective with small de Le. 0.1.
Table 5: Results on AgeDB-DIR. Bold numbers indicate the best performance.
MAE J
GM 1
X
ALL
Many
Med.
Few
ALL
Many
Med.
Few
0
7.60
6.79
8.55
12.70
4.76
4.15
5.95
9.60
0.1
7.50
6.53
8.64
13.50
4.71
4.04
5.92
10.62
1
7.46
6.73
8.18
12.38
4.72
4.21
5.36
9.70
10
7.49
6.70
8.27
12.86
4.79
4.23
5.61
10.21
F INFLUENCE OF THE SAMPLE SIZE (M)
Efficiency-wise, the computing complexity the regularizer quadratic with respect to M. The synthetic experiments operator learning (Table 6) use 2-layer MLP, the regularizer adds significant computing time when M gets large. However, the real-world experiments on depth estimation (Table 7) (27% and 0.3%, respectively). even with M = 3536. We exploit M = 3536 in our depth estimation experiments, where 3536 is 16x subsampling of the total number of pixels in an image. Note that these increases are only during training and do not add computing demands for inference, In addition, the added time and memory with L are also negligible (0.08% and 0%, respectively), even with M=3536.
ResNet-50 backbone, and the added time and memory negligible
Table 6: Quantitative comparison of the time consumption and memory consumption linear operator learning with synthetic data.
M
Training time (s)
Memory taken (MB)
0
155
2163
291
2163
10
296
2163
100
327
2163
1000
1033
2205
15

---

Table 7: Quantitative comparison of the time consumption and memory consumption depth estimation with NYU-v2, The training time one epoch training time.
M
Regularizer no regularizer
Training time (s) Memory taken (MB)
0
1836
12363
100
La + Lt
1877
12379
1000
Ld + Lt
1999
12395
3526
Ld
2342
12405
3526
Iad + T. 2344
12405
16


Reviews:
[{"review_id": "maqqYSh-aU", "reviewer": "Reviewer_1mqu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors analyze multiple deep learning prediction cases with continuous targets where discretizing the target into a classification problem results in better results compared to directly employing regression loss functions. The authors argue that the root cause of the problem is that the regression objective only optimizes the conditional entropy H(Z|Y) but not the marginal entropy H(Z), whereas the cross-entropy objective optimizes both. The authors then propose a remedy to the situation by incorporating a regularizer to also maximize H(Z). The author finally demonstrates that the regularized improve the performance of the regression model.", "summary_of_the_review": "The paper contains some weaknesses in the claim.\nTherefore, I recommend rejection.", "strength_and_weaknesses": "**Strengths**:\n- The authors aim to provide an explanation of the phenomenon that discretizing continuous targets into classification often time performs better than optimizing the original loss.\n- The proposed regularizer is motivated by the ordinal entropy.\n- The author shows the performance benefit of the regularizer compared to just optimizing the MSE loss.\n\n**Weakness**:\n\nEven though the authors aim to provide an explanation of why converting continuous tasks into a classification is better than running a regression, I don't think the provided analysis is sufficient enough to establish the claim. First, the analysis that the regression objective only optimizes H(Z|Y) is an interpretation/approximation under certain assumptions. The authors establish the relation, which says that the MSE is a proxy for maximizing H(Z|Y). However, the analysis cannot say for sure that the MSE also optimizes something else (like a proxy to H(Z)) as the relations are not exact. In addition, the analysis is only conducted on a linear regressor, not a very non-linear deep network. An analysis that works for linear regressor cases does not always translate to deep neural network settings. Therefore, a more thorough theoretical and empirical analysis needs to be conducted to establish the such claim.  \n\nIn the experiment, the authors did not compare the results with classification-based networks. This is important to back up the claim made by the authors that by modifying the continuous loss, the regression objective can outperform the classification baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "am5V-KVLh4W", "reviewer": "Reviewer_ejHX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an analysis of regression's reformulation as a classification problem and proposes an ordinal entropy loss to learn high-entropy feature representations which preserve ordinality based on the analysis.\nThe proposed method outperforms the previous regression loss, classification loss, and other comparison methods in various benchmark datasets. A thorough ablation study also properly supports the author's claim.\n\n", "summary_of_the_review": "Based on the strength, weaknesses, and novelty, I think this paper is novel and has high contributions. But I am not absolutely certain about my decision, I will adjust my decision after checking other reviewers' opinions.", "strength_and_weaknesses": "[Strenght]\n+ the analysis and finding of regression's reformulation as a classification problem.\n+ propose an ordinal entropy loss to learn high-entropy feature representations which preserve ordinality\n+ significantly improved performance on various datasets and tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BXVvgjM2nUY", "reviewer": "Reviewer_NgkU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed to loss term for deep regression models to increase the entropy of the learned representation and with that increase the performance of the predictive model. An additional term is also proposed to decrease the entropy of the class conditioned representation.", "summary_of_the_review": "see above", "strength_and_weaknesses": "Strengths\n-Well written.\n-Technically sound.\n-Analytical and intuitively well motivated.\n-Novel contributions to deep regression, of wide application.\n-Sound proposed methodology.\n\n\nWeaknesses\n-minor typos. For instance, \"The simplest mapping The feature encoding...\"\n-complexity/efficiency discussion. Since the proposed loss term seems to be quadratic in the number of observations, it would be interesting to discuss the efficiency or the impact on the batch size.\n-in ordinal regression (also known as ordinal classification), there are works adding a regression component to the classification loss. Although quite different from the current proposal, it may be interesting to comment and put into perspective in the related work.\nhttps://arxiv.org/abs/2202.05167\nhttps://aclanthology.org/2022.coling-1.407/\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7M-IOplJCn", "reviewer": "Reviewer_eTrs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors analyze the observation that reformulating regression as classification often yields better performance by better learning high-entropy feature representations. To retain the benefits of both high entropy and ordinality of feature learning, the authors propose an ordinal entropy loss to encourage higher-entropy feature spaces while maintaining ordinal relationships for regression tasks. Extensive experiments demonstrate the empirical improvement of the method. ", "summary_of_the_review": "The authors propose an ordinal entropy loss with the original regression task loss to improve the regression performance, based on the theoretical analysis that reformulating regression as a classification problem can often improve the performance. The method is also experimentally demonstrated on multiple different regression tasks. Therefore, I recommend acceptance. ", "strength_and_weaknesses": "Strength:\n\n(1) I like the analysis of regression and its reformulating classification through the lens of mutual information and entropy. It does theoretically explain the difference between regression penalized by MSE and regression penalized by cross-entropy. Besides the empirical improvement, I do value this theoretical finding. \n\n(2) The idea is well-presented and easy to follow. The experimental demonstration is also convincing and extensive. \n\nWeakness:\n\n(1) The authors should consider conducting an ablation study on $\\lambda_d$ and $\\lambda_t$. Is the final performance sensitive to these two hyperparameters? \n\n(2) Are Eq. (8) and (9) defined on the mini-batch or the entire dataset? How does adding tightness and diversity terms increase the overall training cost? \n\n(3) Missing related work [1]. The idea of preserving ordinality by using the label distances is interesting and technically sound. A similar idea in the case of preserving ordinality in regression is also used in [1]. The authors should discuss their difference.\n\n(4) Minor point: I encourage the authors to further proofread the manuscript. I can find some of the typos as follows:\n- Section 3.1 P2 L3 ‘The simplest mapping The feature encoding’\n- Sec. 3.2 P2 L2 ‘being a being composed’\n- Introduction Paragraph 4 ‘and regress’ to ‘and regression’\n- Section 4 P3 L2 ‘is precision enough’\n\n\n\n[1] Gong et al., RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, ICML 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster