Improving Deep Regression with Tightness
Authors: Shihao Zhang
Yuguang Yan
Angela Yao
Venue: ICLR 2025 Poster (2025)
Keywords: regression representation
ordinality
tightness
depth estimation
age estimation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: regression suffers from the weak ability to minimizing H(Z|Y), and preserving the ordinality of targets can reduce H(Z|Y).

Abstract: For deep regression, preserving the ordinality of the targets with respect to the feature representation improves performance across various tasks. However, a theoretical explanation for the benefits of ordinality is still lacking. This work reveals that preserving ordinality reduces the conditional entropy $H(Z|Y)$ of representation $Z$ conditional on the target $Y$. However, our findings reveal that typical regression losses fail to sufficiently reduce $H(Z|Y)$, despite its crucial role in generalization performance.  With this motivation, we introduce an optimal transport-based regularizer to preserve the similarity relationships of targets in the feature space to reduce $H(Z|Y)$. Additionally, we introduce a simple yet efficient strategy of duplicating the regressor targets, also with the aim of reducing $H(Z|Y)$.  Experiments on three real-world regression tasks verify the effectiveness of our strategies to improve deep regression.  Code: https://github.com/needylove/Regression_tightness

Published as a conference paper at ICLR 2025
IMPROVING DEEP REGRESSION WITH TIGHTNESS
Shihao Zhang', Yuguang Yan', Angela Yao'
National University of Singapore zhang. shihao@u. nus. edu
GGuangdong University of Technology
ygyaněgdut edu.. on ayaoнcomp..... nus edu. sg
ABSTRACT
For deep regression, preserving the ordinality of the targets with respect the feature representation improves performance across various tasks. However, a theoretical explanation for the benefits of ordinality is still lacking. This work reveals that preserving ordinality reduces the conditional entropy Y) of representation Z conditional on the target Y. However, our findings reveal that typical regression losses fail to sufficiently reduce H(Z despite its crucial role in generalization performance. With this motivation, we introduce an optimal cansport-based regularizer to preserve the similarity relationships of targets in the feature space to reduce Y Additionally, we introduce a simple yet efficient strategy of duplicating the regressor targets, also with the aim of reducing ((Z)Y), Experiments on three real-world regression tasks verify the effectiveness of our strategies improve deep regression. Code: https: //g1 thub. com/
needylove Sellyy/loe/Regressttee httee//eeeeeeeeeeiiiiieeeeeeeeee
INTRODUCTION
Classification and regression are two fundamental tasks machine learning. Classification maps input data to categorical targets, while regression maps the data to continuous target space. Representation learning in deep neural networks is well-studied for classification (Boudiaf et al., 2020; Achille Soatto, 2018 8), but is less explored for regression. One emerging observation in deep regression is the importance of feature ordinality (Zhang al., 2023). Preserving the ordinality of targets within the feature space leads to better performance, and various regularizers to enhance ordinality have been proposed (Gong et al., 2022; Keramati et al., 2023). But what is the underlying link between ordinality and regression performance?
The information bottleneck principle (Shwartz-Ziv & Tishby, 2017) suggests that neural network learns representations Z that retain sufficient information about the target Y while compressing irrelevant information. The two aims can be regarded as minimizing the conditional entropies ((Y(Z) and Y) (Zhang et al., 2024). Compression reduces representation complexity, prevents overfitting, and bounds the generalization error (Tishby & Zaslavsky, 2015; Kawaguchi et al., 2023; Zhang et al., 2024). We find that preserving ordinality enhances compression by minimizing H(ZZYY,, i.e., the conditional entropy of the learned representation Z with respect to the target Y. Following (Boudiaf al., 2020; Zhang et al., 2024), we refer this conditional entropy as tightness, and its compression as tightening the representation.
But are ordinal feature spaces not learned naturally by the regressor? We explore this question through gradient analysis and comparing the differences between regression and classification. We find that typical regressors are weak in tightening the learned representations. Specifically, given fixed linear regressor with weight vector 0, the update direction of for a given sample tends to follow the direction of 0. The movement of z; can be regarded as a probability density shift (Sonoda & Murata, 2019). Deep regressors update and tighten the representation in limited directions perpendicular to O. In contrast, we find that deep classifiers update z; more flexibly and in diverse directions, leading to begher-tightened representations. Such finding sheds insight into why reformulating regression as a classification task may be more effective (Farebrother et al., 2024; Liu et al., 2019) and why classification losses benefit representation learning for regression (Zhang et al., 2023).
So how can regression representations be further tightened? We take inspiration from classification, where one-hot encodings allow separate set of classification weights 0g for each class k. Similarly,

---

Published as a conference paper at ICLR 2025
augment the target space of the regressor into multiple targets. The multiple-target strategy adds extra dimensions to the regression output and incorporates additional regressors, making it more flexible to tighten the feature representations. Additionally, we introduce Regression Optimal Transport (ROT) Regularizer, or ROT-Reg. ROT-RReg captures local similarity relationships through optimal transport plans. By encouraging similar transport plans between the target and feature space, the regularizer can tighten representations locally. It also helps to preserve the target space topology, which is also desirable for regression representations (Zhang al., 2024).
Our main contributions are three-fold:
We the first to analyze the need for preserving target ordinality with respect to the representation space for deep regression and link it to feature tightness.
We reveal the weakness of standard regression in tightening learned feature representations, as the representation updating direction is constrained to follow a single line.
We introduce a multi-target learning strategy and an optimal transport-based regularize, which tighten regression representations globally and locally, respectively.
2 RELATED WORK
Regression representation learning. Existing works mainly focus on the properties of continuity and ordinality. For continuity, DIR (Yang et al., 2021) tackles missing data by smoothing, based on the continuity of both targets and representations. VIR (Wang & Wang, 2024) computes representations with additional information from data with similar targets. Preserving representation continuity also encourages the feature manifold to be homeomorphic with respect to the target space and is highly desirable (Zhang et al., 2024).
For ordinality, RankSim (Gong et al., 2022) explicitly preserves ordinality for better performance. (Keramati et al., 2023) further incorporated a contrastive regularizer to preserve ordinality. It is worth mentioning that continuity overlaps with ordinality conceptually; ensuring continuity for neighboring samples inherently involves ordinality. Although ordinality plays a key role in regression representation learning, its importance and characteristics are underexplored. This work tackles these questions by establishing connections between target ordinality and representation tightness.
Recasting regression as classification. For diverse set of regression tasks, formulating them into a classification problem yields better performance (Li et al., 2022; Bhat et al., 2021; Farebrother al., 2024). Previous works have hinted at task-specific reasons. For pose estimation, classification provides denser and more effective supervision (Gu et al., 2022). For crowd counting, classification more robust to noise (Xiong & Yao, 2022). Later, Pintea al. (2023) empirically found that classification helps when the data is imbalanced, and Zhang et al. (2023) suggests that regression lags in its ability to learn a high entropy feature space. A high entropy feature space implies that representations preserve necessary information about the target. In this work, we provide derivation and further suggest that regression struggles to compress the representations.
3 ON THE TIGHTNESS OF REGRESSION REPRESENTATIONS
3.1 NOTATIONS & DEFINITIONS
Consider a dataset (x,, Zi, Di L. sampled from distribution P. where x; is the input, y E R is the corresponding label, and z, E Z C Rd is the feature corresponding by a neural network. A regressor So parameterized by 0 maps z; to a predicted target ú; = fo fz,z)) Specifically, when So is a linear regressor, which is typically the case in deep neural networks, have 9. = TTz,. z, The encoder and fe are trained by minimizing a task-specific regression loss Creg. Typically, the mean- equared error is used, i.e. Lreg + M (v: 0.)
the input x, extracted
To formulate regression as a classification problem, the continuous target y is quantized to K classes uf E (1,... K.), and the cross-entropy loss is used train the encoder and classifiers
2

---

Published as a conference paper at ICLR 2025
0 LCE = - Eie log E oxp 00
where 0g is the classifier corresponding to the class k. The function d(*, measures some distance between two points, egg., Euclidean distance.
3.2
ORDINALITY AND TIGHTNESS
This section shows that preserving ordinality tightens the learned representation, and conversely, tightening the representation will help preserve ordinality. A lower YZ) represents higher compression (Zhang et al., 2024). The compression is maximized when H(Z ()) in its minimal (H(Z) -c for differential entropy and H(|/Y) = 0 for discrete entropy).
First, we define the ordinality following (Gong et al., 2022):
Definition (Ordinality). The ordinality perfectly preserved if Vi, k, the following holds: d(y,, MU) Σ d(rs, 9k) = d((2,2z) ) Σ d(z,, 2z)-
Theorem Let (zz,c) c) = {z' Zid(z,z), z') S c} be the closed ball center at with radius e. Assume that v(x, z, E and Ve > 0, 3 .z'./)) P such that z' Blz, c) and y' # y. Then if the ordinality is perfectly preserved, v(x,, 2, Vi), (x,, Z.. Us) E P. the following hold: y/ = yJ + d(z,, 2j) -..
The detailed proof of Theorem is given in Appendix A.1. Theorem states that the ordinality is perfectly preserved, then the tightness (i.e. H(Z is minimized. This suggests that preserving ordinality will tighten the representations. The assumption in Theorem aligns with the learning target that learning continuously changes representations from continuous targets.
Conversely, the representations can be correctly mapped to the target and are perfectly tightened, then the representations collapse into manifold homeomorphic to the target space (e.g., collapse into a single line when the target space is a line) [(Zhang et al., 2024), Proposition 2]. Thus, ordinality will be perfectly preserved locally. Note that reserving ordinality globally constrains the line to be straight, which is not necessary.
3.3 REGRESSION TIGHTNESS
Why are additional efforts to emphasize ordinality necessary? In this work, we find that standard deep regressors are weak in their ability tighten the representations due to the gradient update direction with respect to the representations. Consider a fixed linear regression with a typical regression loss (e.g., MSE, L1), which has the following gradient with respect to z,;
0C Oz:
Cleg (0 -1)=-
8x, is determined solely by the direction of 0. As such, during learning, all the z, are moved in the direction of 0 (or away). This movement can be regarded
Here, the direction of 8Cms
probability density shift (Sonoda Murata, 2019). so regression suffers from a weak ability to change probability density in directions perpendicular O, which indicates limited ability tighten representations in those directions. In other words, regressors can only move z; to Sy. but cannot tighten Sy., where Sy, {z) (z) } is the solution space of fø(z) yi. More generally, for a differentiable regressor, we have the following:
Theorem Assume fe is differentiable and Sy, is a convex set, then Vzl, 2/3 E Sy:
dLwg (z) - -z,)= = 0. Oz.
(2)
where u the predicted target of zi.
The detailed proof of Theorem is given in Appendix A.2. The regressor so generally differentiable for gradient backpropagation, and Sy, is commonly convex set with widely used regressors, such as the linear regressor. Theorem shows that the gradient with respect to the representation will be perpendicular to its solution space and has effect within the solution space. In other words, with
In this work, classifier represents single 0, rather than the whole set (0 K}.
3

---

Published as a conference paper at ICLR 2025
fixed regressor, the gradient only moves the representations to the corresponding solution space and lags in its ability to tighten the feature space.
In reality, the regressor is not fixed (i.c.. updating with training), and the solution space is also changing during training. In the case of a linear regressor, the gradient with respect to 0 over a batch of b samples be given as:
dreg 00
1 b
Cog
( v1)z
w.Z.
Here, the direction of aL will tend to be the weighted mean of the direction of z;. As discussed, the 30
direction of z; approaches the direction of O. Thus, z; will distribute around 0 and offset each other, resulting in limited impact on the direction of 0.
worth mentioning that the tightness here is specific to H(Z Y indirectly related to the predicted results and performance. By contrast, the tightness outside Sy. directly affects the predicted results and potentially plays more important role.
yi) within S which is
3.4 COMPARISON IN TIGHTNESS FOR CLASSIFICATION
When comparing classification with regression, we find that classification has greater flexibility to tighten representations diverse directions 0x, suggesting an ability to better tighten the representa- tion. For the gradient with respect to over a batch of b samples:
OCCE θz.
0( 5 Σ OT 24) a E log Σ3
K
Pij 0
0
(4)
Oz.
Oz:
0,2, where Pij Ekmi exp0 is the probability of sample i belonging to class j. Here, the direction of dces is affected by all og. and z, will approach 8y, with training. In contrast, the direction dz.
of 8Caes is purely determined by 0. Classification moves z, to its corresponding classifier 0y. o2.
even if the sample is correctly classified. At the same time, regression has effect z, if it is aEw; 0). This suggests that classification has a higher ability to tighten representations in the solution space Sy. Here, Sy, for classification defined as the set of z, that are classified as class y/:
correctly predicted (i.e., dz,
In reality, the classifiers O2 are not fixed and are updated with training. The gradient with respect to htth classifier O₂ over a batch of b samples is given as:
OCCE 00,
b
Σ kg:==
2 2
(Dak ky.
(5)
b
==1
where Pit:
Σ 3 is the probability of sample i belongs to the class k, and 5 is the
Kronecker delta function. For classification, the direction of 0₂ will biased toward z; with respect to the class k. while z, will also bias towards its corresponding classifier. In contrast, for regression, the direction of 0 will tend to be the weighted mean of the direction of z; Thus, the effect of many z; on the direction of will offset each other and have limited impact. As result, changes in the directions of 0 are generally greater than the change of the A direction in regression, and therefore classification can move z more flexible and thus potentially better tighten the representation.
METHOD
Our analysis in Sec. 3 inspires to tighten the regression representations. To this aim, we introduce the Multiple Target (MT) strategy and the Regression Optimal Transport Regularizer (ROT-Reg) to tighten the representations globally (i.e., minz H Y)) and locally (i.e., minz H y:), e Biz,, <), e control the degree of locality). Inspired by the effect of multiple classifiers classification, the MT strategy introduces additional targets constraints to compress the representations. For ROT-Reg, we exploit it to encourage representations to have local structures similar to the targets, which implicitly tightens the representations.
4

---

Published as a conference paper at ICLR 2025
0g
0
yoo
M
S yo
M
Vo
2
02
Yo
0
Yo
01
9 = [[00,, 0)0)
y = [,0,, 0, (0, 02, 0)z]
Figure Illustration of the MT strategy. Changing the target from to [y+ yl will introduce an additional regressor to predict the additional y. The original solution space Syo is line in the feature manifold. The additional y introduces a new constraint, tightening Syo from a line to a point.
4.1 TARGET SPACE WITH EXTRA DIMENSIONS BETTER TIGHTEN THE FEATURE SPACE
Our analysis in Sec. 3.4 suggests that classification outperforms regression in its ability compress representations in multiple directions, resulting from multiple classifiers. Inspired by this, we introduce simple yet efficient strategy, which adds extra dimensions for the target space bring in extra regressors as constraints. Here, the additional regressors have similar effect as individual classifiers. As shown in Figure I, the additional constraints will result in lower-dimensional Sy. which indicates higher compression. The number of additional targets depends on the intrinsic dimension of the feature manifold, In our Multiple Targets strategy, the final predicted target the average over the multiple predicted targets:
M
(6)
M
where M is the number of the total target dimension and of is the th predicted target.
4.2 REGRESSION OPTIMAL TRANSPORT REGULARIZER (ROT-REG)
The MT strategy tightens the representations globally through additional regressors. We propose to further tighten the representations locally. Specifically, we preserve the local similarity relations between the target and representation space. The local similarities optimal transport model (Yan et al., 2024; Landa et al., 2021). The model determines the optimal plan is move set of samples to the set itself with minimal transport costs, while each sample cannot be moved to itself.
characterized by self entropic
Formally, Given set S (S1, (si,..., S,), the corresponding weight vector p R" reflects how many masses the samples have, where the weights simplify the simplex constraint Σx P3 1. Usually, one can easily implement p as a uniform distribution, i.e., Ps = 1. V E [n]. C is the transport cost between s; and sj, which is generally adopted the Euclidean distance between the samples, and T35 indicates how many masses transported from the locations of to sj. The self entropic optimal transport is defined as follows:
T(S) arg min (CS, T) +T)(TT T
s.t. T1, =p. 1, = P. Th = Vi [n],
(8)
where y a trade-off parameter, and Q(T) Σn Σソ= T, log 2 is the negative entropic regularization, which used to smoothen the solution and speed up the optimization (Cuturi, 2013).
Given the solution TS minimizing the above objective, the element 7 measures the similarity relation between samples s; and 8j. since two samples with large distance CS, will induce a small transport mass T between them. As a result, the optimal total transport cost (C*, TS) reflects the tightness of the samples.
5

---

Published as a conference paper at ICLR 2025
Motivated by this, we employ the self optimal transport model to capture local similarity relations of target and representation spaces, respectively, and encourage a relation consistency between two spaces. In specific,
first construct self optimal transport model on the target space to obtain
T =YYY), which describes the local similarity relations between the regression targets. After that, we learn regression representations Z such that the corresponding optimal transport matrix T TZ) is consistent with T which is achieved by the following loss function
at = |C " - T
(9)
ROT-Reg is updating C through gradient backpropagation minimize Cot- In contrast, simply minimizing the gap of T and T can introduce optimization challenges, as the two matrices are obtained iteratively through the Sinkhorn algorithm rather than simply through gradient backpropa- gation. In addition, directly minimizing ||C* CY || imposes an overly strict constraint on the feature manifold, forcing it to become identical to the target space, which is unnecessary.
is worth mentioning that y controls the 'smooth' of the transport plan T and determines the degree of locality. When 0, T will approach the minimal spanning tree (i.e., only transports mass to its nearest neighbor), and for will encourage the representations to have the same minimal spanning tree to the targets, which is shown to be strategy to preserve the topology of the target space (Moor et al., 2020). In fact, the topological auto-encoder (Moor et al., 2020) preserves topological information in this way. Compared to topology autoencoder, Lot captures more local structures of targets when y > O. The final loss L, sums the specific loss C, and the regularizer with a trade-off hyper-parameter A:
C₁= Le ACots
(10)
EXPERIMENTS
We experiment on three deep regression tasks: age estimation, depth estimation, and coordinate prediction and compare with RankSim (Gong et al., 2022), Ordinal Entropy (OE) (Zhang et al., 2023), and PH-Reg (Zhang et al., 2024). RankSim explicitly preserves ordinality serve as the ordinality baseline. OE leverages classification for better regression representations and serves as regression bascline. PH-Reg preserves the topologiaal structure of the target space by the Topological autoencoder (Moor et al., 2020) and tightens the representation by Birdal's s regularizer (Birdal et al., 2021), serving as a topology baseline. More details are given in Appendix B.I.
5.1 REAL-WORLD DATASETS: AGE ESTIMATION AND DEPTH ESTIMATION
For age estimation, we use AgeDB-DIR (Yang et al., 2021) and evaluate using Mean Absolute Error (MAE) as the evaluation metric. and are set to 0.1 and 100, respectively. For depth estimation, we use NYUD2-DIR (Yang et al., 2021) and evaluate using the root mean squared error (RMSE) and the threshold accuracy 81 as the evaluation metrics. 7 and X are set to 0.05 and 10, respectively. We set the total target dimension M to be for both tasks. Both AgeDB-DIR and NYUD2-DIR contain three disjoint subsets (i.e., Many, Med, and Few) divided from the whole set. We exploit the regression baseline models of (Yang et al., 2021), which use 5e (He et al., 2016) as the backbone, and follow their setting for both tasks.
Tables and 2 show results on age estimation and depth estimation respectively. Both the Multiple Targets strategy (MT) and Lut improve regression performance, and combining both further boosts performance. Specifically, combining both achieves 0.52 overall improvements (i.e. ALL) on age estimation, and 0.156 reduction of RMSE on depth estimation.
5.2
Lot PRESERVES THE LOCAL SIMILARITY RELATIONSHIPS
The effectiveness of Cor is verified with the coordinate prediction task from Zhang et al. (2024). This task predicts data coordinates sampled from manifolds such as Mammoth, Torus, and Circle, which have different topologies. The inputs are noisy data samples and the goal is to recover the true data coordinates. Figure shows that Lat successfully preserves the similarity relationships of the targets, resulting in feature manifold similar to the targets. Quantitative comparisons in Table 3 indicate

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison (MAE) AgeDB-DIR. We report results + standard deviation over 10 runs. Bold numbers indicate the best performance.
Method
ALL
Many
Med.
Few
Baseline
7.80
6.80 0.06
9.11 + 0.31
13.63 + 0.43
+ RankSim
7.62
6.70 0.10
8.90 + 0.33
12.74 + 0.48
OE
7.15
4.72:0009
8.77 + 0.49
13.28 = 0.73
+ PH--Rg
7.32
6.50
8.38 + 0.11
12.18 + 0.38
MT
7.67 0.06
6.72
8.87 0.13
13.36 0.16
+Loc
7.36 + 0.08
6.55
0.07
8.40 0.14
12.14 + 0.33
+ MT Loe
7.28 0.05
6.52 0.10
8.26 + 0.19
11.86 + 0.24
Table 2: Quantitative comparison on NYUD2-DIR.
RMSE
81
Method
ALL
Many
Med.
Few
ALL
Many
Med.
Few
Baseline
1.477
0.591
0.952
2.123
0.677
0.777
0.693
0.570
+RankSim
1.522
0.565
0.889
2.213
0.666
0.791
0.735
0.513
+OE
1.419
0.671
0.925
2.005
0.668
0.727
0.702
0.596
+PH-Reg
1.450
0.789
0.911
2.002
0.620
0.621
0.680
0.596
+ MT
1,367
0.605
0.854
1.952
0.715
0.776
0.759
0.636
+Lot
1.353
0.654
0.934
1.899
0.689
0.736
0.697
0.638
+ MT + Lot
1.321
0.685
0.951
1.829
0.701
0.731
0.689
0.675
Table 3: Results (Lmse) on the coordinate prediction dataset. We report results as mean standard deviation over 10 runs. Bold numbers indicate the best performance.
Method
Mammoth
Torus
Circle
Baseline
211 55
3.01 + 0.11
0.154 + 0.006
+ InfDrop
367 + 50
2.05 + 0.04
0.093 + 0.003
+ OE
187 88
2.83 + 0.07
0.114 0.007
+ Topological Autoencoder
80 + 61
0.95 + 0.05
0.036 0.004
PH-Reg
49 +
0.61 + 0.05
0.013
+ MT
174 76
2.99 0.11
0.152 0.005
+ Fot
87 + 26
0.77 + 0.02
0.010 0.001
MT+ Lot
76 + 53
0.75 4 0.03
0.010 + 0.001
02
477
13
10947550004 NN 6/S
48
4
(a) Target Space
(b) Regression
(c) PH-Reg
(d) Cot
Figure 2: Visualization of the feature manifolds, which shows that Cot preserves the local similarity relationships of the target space.
that Loe performs similarly to PH-Reg, specifically designed preserve similarity relationships. However, the Multiple Targets (MT) strategy has a limited impact in this context, likely because the target space three-dimensional, providing sufficient constraints for the feature manifold.
7

---

Published as a conference paper at ICLR 2025
Table 4: Correlation between feature and label similarities. Results mean + std dev over 10 runs.
RMSE
Cosine Distancc
Euclidean Distance
Method
(ALL)
Spearman'ss '
Kendall's
volume
Spearman's s 1
Kendall's t
volume
Baseline
1,477
0.39 + 0.15
0.27 0.11
0.573 + 0.071
0.35 + 0.14
0.24 + 0.10
7.72 + 0.92
RRnnSiim
1.522
0.09 0.04
0.06 0.03
0.000 1 0.000
0.60 1 0.16
0.44 1: 0.13
4.26 1.29
MT
1.367
0.49
= 0.14
0.34 0.10
0.492 + 0.047
0.47 0.11
0.33 = 0.08
6,56 1.18
Cot
1.353
0.48
0.16
0.34
0.12
0.010 0.003
0.42 0.15
0.29 0.11
5.57
0.77
MT Cust
1.321
0.64 0.09
0.46
0.08
0.006
1 0.002
0.61 + 0.11
0.44 + 0.09
0.11
86
01
11
12.
12
-15
23
20
(a) Baseline
(b) + RankSim
(c) MT
Figure 3: Visualizations of the feature manifold on NYUD2-DIR for depth estimation. Preserving the
ordinality (+ RankSim) has an effect similar to MT, which explicitly tightens the representations.
5.3 TIGHTNESS AND ORDINALITY AFFECT EACH OTHER
Compression for better ordinality. We examine the impact of tightness on the ordinality. Table presents the Spearman's 's rank correlation coefficient (Spearman, 1961) and Kendall rank correlation coefficient (Kendall, 1938) between the feature similarities (based on Cosine distance and Euclidean distance) and the label similarities. The two correlation coefficients measure how well ordinality is preserved. Since tightness compresses the feature manifold, reducing its volume, we use volume as a proxy for tightness, and the volume is approximated by the mean of the similarities between samples. The experiments are conducted on NYUD2-DIR, we randomly sample 1000 pixels from a batch of 8 test images. The label similarities pixels, while the corresponding feature similarities are the distances between their corresponding representations. The results in Table 4 show standard regression fails preserve the ordinality, while MT and for both improve the ordinality, although they are designed to tighten the representations. Combining both has similar effect on preserving ordinality as RankSim, which specifically designed for this purpose. The lower volumes of our method compared to the baseline indicate that the feature manifold more compressed. We provide the visualization of the feature similarities in Appendix C.
calculated the Euclidean distances between the 1000
Ordinality for a better compression. further verify that preserving ordinality leads to better compression, we visualize the feature manifold of the depth estimation task in 3D space. This is done by changing the last hidden layer's feature space three dimensions. As shown in Figure explicitly preserving the ordinality (i.e., +RankSim) compresses the feature manifold into a thin line, which shows a similar effect to explicitly tightening the representations (i.e., +MT).
5.4 TIGHTNESS OF REGRESSION
Our theoretical analysis in Sec. 3 focuses on the gradient direction of representations. However, in reality, the neural network updates parameters to update the representations indirectly. Here, we verify our analysis by visualizing the update of and in the depth estimation task.
The update of z. We change the last hidden layer's 's feature space to 2 dimensional for visualization. We randomly sample 1000 pixels from a batch of 8 images in the NYUD2-DIR test set to visualize the feature manifold. Figure 4(a) displays the feature manifolds at epoch 1 (blue dots) and epoch 10 (the final epoch, red dots), the corresponding pixel representations Aligned with our theoretical analysis, the directions of the representation updates follow the direction of 0. To verify this quantitatively, calculate the principal component of the update directions using
connected by black arrows.
8

---

Published as a conference paper at ICLR 2025
regressor epoch 1
epoch 10
3.28
daccher 5
dandher 6
Carre
3.06
danaher
-6
334
deai'r's
desdhe
332
6
Speth
10
(a) Update of z
(b) Update of 0
(c) Update directions of 0
Figure 4: (a) Visualization of the z update, which aligns with 0, (b) f update, which is steady through the training process, (c) the updating directions of Os, which distributed along line, with the original as the center.
PCA. We find that the cosine distance between this principal component and @ is very small (0.03), indicating that the updating directions of representations from the beginning to the end of training follow the direction of 0, The visualization also shows that the feature manifold tightened limited in the direction perpendicular to 0 throughout the training. The visualizations of feature manifolds at each epoch are provided in Appendix C.2, which reveals that the tightening effect in the direction perpendicular to f between adjacent epochs is even smaller.
The update of 0. As discussed in Sec. 3. the effect of z, on the direction of i tends offset each other and results a limited impact, while changes the directions of 0₂ in classification are generally greater. Here we quantitatively verify this by calculating the cosine distances between and 0 at each epoch from to 10 (final epoch), where represents 0 at epoch 10. We also convert this regression task into a classification task by uniformly discretizing the target range into 10 classes, and monitoring the change of Og in the same way. As shown in Figure 4(b), the changes in O₂ are all larger than the changes in O. The maximum cosine distance between O at different epochs is very small (i.e., 0.0004), which also verified the limited change of O.
Multiple Os. Adding additional Os (our MT strategy) with random initialization, does not change the update speed of 0 (see Figure 7 in the Appendix C.3). The update directions of all Os are even aligned. Let v/ = 0 O be the updating vector of in iteration Figure 4(c) plots the set of points (ugli - 500k, & E 2,0 < < 100} for three Os. This visualizes the change of Os throughout the training process. The three Os are distributed along a line, with the original as the center. When we calculate the principle components of {fa} for the three Os using PCA. the maximum cosine distances between the three principle components are less than 1 - 4, which quantitatively shows the updating directions of all As are the same direction. As shown in Eq. 3, ng is the weighted mean of batch of z. Different 0 leads to different magnitudes of the weight mean while the directions remain steady. It is worth mentioning that the multiple Os do not collapse to a single in reality, although their updating directions are the same. This is because Os are randomly initialized, and their directions remain nearly identical during training (see Figure 4(b)), due to the three reasons: 1)
The magnitude of (ha is 'scaled' by w,, since w, often follows a Gaussian distribution centered at the origin, as assumed in models like Bayesian linear regression. When w and z are independent &Lag be 'scaled' to 0. 2) According to the
"00
or weak dependent, E[w,z) z; will approach 0 and causing ag central limit theorem, the updates of 0 follow Gaussian distribution. This causes partial offsets between updates and results in a reduced accumulated effect. In addition, we empirically observe that the mean of this Gaussian distribution approaches 0 (see Figure 4(c)), indicating that w and are independent or weakly dependent. 3) The effect of z; on the direction of diiz over a batch of samples offsets each other, resulting in the stability of the direction of 0 throughout training. This occurs because z; tends to be distributed around More details can be found in Appendix C.3.
20
5.5 ABLATION STUDY
We conduct the ablation study AgeDB-DIR for age estimations. The results given in Table 5.
9

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study AgeDB-IDIR for age estimation. We report MAR mean + standard
deviation over 10 runs, and the default X, 7 and M are set to 100, 0.1 and 8, respectively.
Method
ALL
Many
Med.
Few
Baseline
7.80 + 0.12
6.80
0.06
9.11
0.31
13.63 + 0.43
+ Lee
À =
7.68 + 0.08
6.75 + 0.11
8.81 + 0.19
13.38 + 0.37
À 10
7.55 0.05
6.64 0.07
8.71 + 0.12
12.88 + 0.35
À 100
7.36
6.55
8.40
12.14 0.33
À = 1000
8.80
0.10
+.32
17.26 + 0.53
Y = 0.1
0.08
6.55 +.07
8.40
12.14 0.33
Y
7.47
6.61 0.08
0.37
12.55 + 0.49
Y 10
7.51 0.07
6.63 + 0.08
8.60 + 0.25
12.75 0.31
+ MT
M=2
1.72
6.77 + 0.07
8,92 0.20
13.37 + 0.50
M=4
7.74
6.77 0.10
8.96 0.13
13.52 + 0,41
M=8
7.67 + 0.06
6.72 0.08
8.87
13.36 + 0.16
M=16
7.70
6.13
8.93 0.25
13.44 0.36
M=32
7.71
6.74
8.96
13.40 0.32
noise
8.00 + 0.23
6.91 0.20
9.43 0.33
14.36 + 0.67
Table 6: Time and memory consumption.
Method
Time (mins)
Memory(MB)
Baseline
65
14433
+ MT
70
14457
Lot
74
14587
+ MT + Cos
82
14689
Hyperparameter X, 7. We keep X, at their default value 100, 0.1, and vary each individually to examine their impact. As shown in Table The MAE (ALL) decreases consistently as increases. However, it overtakes the task-specific learning target when set too high (e.g.. 1000) and decreases performance. For the y, the MAE (ALL) decreases consistently as 7 decreases. However, we empirically find that setting 7 (e.g., 0.01) too low will easily result in NaN values when calculating the transport matrixes using the Sinkhorn algorithm (Cuturi, 2013). We thus set 7 = 0.1.
Number of the total targets M. As shown in Table 5, the performance generally improves with the increase M, when < 8, and stays steady when M increases further. The primary factor affecting the selection of M is the intrinsic dimension of the feature manifold, which determines how 1) are required to compress the manifold. The range of Y has limited impact the selection of M, since M equals works well in NYUD2-DIR (y E [0.7, 10) and AgeDB-DIR ( y E [0, 101]).
many additional constraints (i.e.,
Mean D vs. single ÿ. We verify the strategy of the mean operation in MT (see Eq. 6), which potentially brings in an ensemble effect. We find that of are very similar for all f. For a model with M, the MAE(ALL) results calculated by of for E T with mean equal 7.579 and standard deviation 0.0003. Thus, the improvement of MT is not due to the ensemble effect, and the mean operation optional.
Additional vs. noise. Adding additional targets noise, shown in Table 5, does not work.
Time and memory consumption. We monitor the time and memory consumption for training model from the beginning to the end with a batch size equal to 128. Table 6 shows the added memory negligible(1.799 and the added time is limited (17 min).
CONCLUSION
In this paper, for the regression task, we provide a theoretical analysis that suggests preserving ordinality enhances the representation tightness, and regression suffers from weak ability tighten the representations. Motivated by classification and the self entropic optimal transport, we introduce simple yet effective method tighten regression representations.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement. This research / project is supported by the Ministry of Education, Singa- pore, under the Academic Research Fund Tier 1 (FY2022), National Natural Science Foundation of China (62206061, U24A20233), Guangdong Basic and Applied Basic Research Foundation (2024A151501 1901), Guangzhou Basic and Applied Basic Research Foundation (2023A0411700)
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-295 2905, 2018
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4009-4018, 2021.
Tolga Birdal, Aaron Lou, Leonidas Guibas, and Umut Simsekli. Intrinsic dimension, persistent homology and generalization in neural networks. Advances in Neural Information Processing Systems, 34:6776-67777 2021.
Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European conference on computer vision, pp. 548-564 Springer, 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https:: neurips cc/paper. fww.fflleepppe /etee//pppee//11// ELLe:/AF2200s7722222//2222222333 Papeedd--aap.....ee Paper.ppe.. pdf.
Jesse Farebrother, Jordi Orbay. Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In Forty-first International Conference on Machine Learning, 2024. URL https: openreview. net/ forum? Berummid-dVpFKfEe 3R.
Yu Gong, Greg Mori, and Frederick Tung. RankSim: Ranking similarity regularization for deep imbalanced regression. In International Conference on Machine Learning (ICML), 2022.
Kerui Gu, Linlin Yang, and Angela Yao. Dive deeper into integral pose regression. In International Conference on Learning Representations, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference on Machine Learning, pp. 16049-16096. PMLR, 2023.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Mahsa Keramati, Lili Meng, and R David Evans. Conr: Contrastive regularizer for deep imbalanced regression. In ICLR, 2023.
Boris Landa, Ronald R Coifman, and Yuval Kluger. Doubly stochastic normalization of the gaussian kernel is robust to heteroskedastic noise. SIAM journal mathematics of data science, 3(1): 388 413, 2021.
Yanjie Li. Sen Yang, Peidong Liu. Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang, and Shu- Tao Xia. Simce: simple coordinate classification perspective for human pose estimation. In European Conference Computer Vision, pp. 89-106. Springer, 2022.

---

Published as a conference paper at ICLR 2025
Liang Liu, Hao Lu, Haipeng Xiong, Ke Xian, Zhiguo Cao, and Chunhua Shen. Counting objects by blockwise classification. IEEE Transactions on Circuits and Systems for Video Technology. 30(10): 3513-3527, 2019.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pp. 7045-7054, PMLR, 2020.
Silvia L Pintea, Yancong Lin, Jouke Dijkstra, and Jan C van Gemert. A step towards understanding why classification helps regression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19972-19981, 2023.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: / 703.00810, 2017.
Sho Sonoda and Noboru Murata. Transport analysis of infinitely deep neural network. Journal of Machine Learning Research, 20(2):1-52, 2019.
Charles Spearman. The proof and measurement of association between two things. 1961.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (ihw), pp. 1-5. IEEE, 2015.
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. Advances in Neural Information Processing Systems, 36, 2024.
Haipeng Xiong and Angela Yao. Discrete-constrained regression for local counting models. In European Conference on Computer Vision, pp. 621-636. Springer, 2022.
Yuguang Yan, Zhihao Xu, Canlin Yang, Jie Zhang, Ruichu Cai, and Michael Kwok-Po Ng. optimal transport view for subspace clustering and spectral clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16281-16289, 2024.
Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbal- anced regression. In International Conference Machine Learning, pp. 11842-11851. PMLR, 2021.
Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. ICLR, 2023.
Shihao Zhang, Kenji Kawaguchi, and Angela Yao. Deep regression representation learning with topology. In International Conference Machine Learning (ICML), 2024.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOF OF THEOREM 1
Theorem 1 Let B(z,e) = =(z E Zid(z, x²) S E} be the closed ball center at z with radius e. Assume that v(x, 2. P and Vc > 0, 0x3,'''z', xxx,z',//) P such that z' B(z,c) c) and y + Then if the ordinality perfectly preserved, V(x,, Z, M1), (x,, y1 yj - d(zi, 2z) -.
23. Uz) E P. the following hold:
Proof
d(z;, 23) d(z; 2k + 2k 2j)
(11)
d(z; 2%} d(zı: 2z);
(12)
where 2y e(2,e). Since d(yk, Vj) S d(yk: vi). and the ordinality is perfectly preserved, have:
d(zx 2j) S d(z; 2z)-
(13)
Thus:
5 2i) S 2d(zz =xx)= S 2c.
(14)
Let 0, the result follows.
A.2 PROOF OF THEOREM 2
We first give a lemma:
Lemma Let Sy = {zig(z) = y} be convex set, where z E R is the representation, is the target and g is the regressor. Assume g is differentiable, then Vzk Z,, Z5 E Sy, we have:
Vg(zz)(z; 2z))
(15)
Proof Let z (1 e)zk + ez,, where E [0, 1]. Since g differentiable, using Taylor expansion, we have:
g(2;)) g((1 c)zx + ezi)
(16)
g(z + czz z;))
(17)
g(z()) + eV gl (Bzz (z; 2z. + o(E).
(18)
Since Sy is convex set, we have 2 E Sy. Thus:
g(z = g({)) +
2x) + +((c)
(19)
y = y + <V (zz) (zi 2x olE)
(20)
o(c) Vg((=)(zz z1).
(21)
Let - 0:
Vg(zx)(Zz zi) = lim o(c) 0.
(22)
c-2
Similarly, have:
(g(z))(2z - 22)) O.
(23)
Combining the two equations above, have:
(z(z))(zz 2,) Vg(2x) [2; 2% 2s. -2,) Vg((z) - 2z) - Vg(zz) - 2;)
(24)
(25)
0.
(26)
13

---

Published as a conference paper at ICLR 2025
Theorem 2 Assume fe is differentiable and Sy is a convex then V2,, Syy
ÖLreg (z) - =zj = O.
(27)
dz.
where y is the predicted target of z
Proof
1c
rce (9(zi) - Vii)
(28)
Oz.
Oni
Ö Goog - v.)) (Zg) y/)
(29)
0(g(zi) - 4/1) Oz.
(g(z)((z) - -y6) -/s) Vg(z))
(30)
Based on Lemma 1, we have:
Vg(2,)(2) - -2,) = O.
xz)=
(31)
Thus,
Ô dReg (2) x//) flcg (g(() U4) Vz): -2,)
(32)
Oz.
Cleg(11 (2,) U1) 0
(33)
O.
(34)
B DETAILS
B.1 DETAILS ABOUT THE REAL-WORDL TASKS
For the age estimation on AgeDB-DIR, we adopt the suggested hyper-parameters train the RankSim, where X, y are set to 2, 1000, and the results of OE and PH-Reg are adopted from their published papers. The evaluation metric MAE: 1 N 9: - -%/l where is the total number of samples, N
yi, 1/f are the label and the predicted result.
For depth estimation on NYUD2-DIR, we adopt the suggested hypar-parameters of OE and PH- Reg to train the models. For RankSim, train the model with the 7 range from 1 to 1000. We report the best results for all three baselines. The evaluation metrie: threshold accuracy 5n & % of
yps s.t. max( De V/2 V Vp
1.25, and root mean squared error (RMS): v 1. E, 8/)²
C VISUALIZATIONS
C. VISUALIZATION OF THE UPDATING OF
We provide the visualization of the feature similarities in Figure. 5.
C.2 VISUALIZATION OF THE UPDATING OF z
The visualizations feature manifolds at each epoch are provided Figure 6. For the neural collapse of regression, the feature manifold will collapse into a single line when the target space is line and the compression is maximized (Zhang et al., 2024). This trend can be observed in Figure 6, where the feature manifold looks like thick line and evolves to a thinner line regression's 's limited ability to tighten representations results in a slower collapse. In contrast, our proposed method and RinkSim both accelerate this collapse, as shown in Figure 3.
training. However, standard
14

---

Published as a conference paper at ICLR 2025
(a) Baseline
(b) RankSim
(c) MT
(d) + Cot
(c) MT + Lot
Figure 5: Feature similarity matrices (Euclidean Distance). Tightening the representations results in better ordinality.
C.3 UPDATING OF MULTIPLE 0
The experiments are conducted on NYUD2-DIR, we change the last hidden layer's feature space to three dimensions for visualization, and the M in our MT strategy set to 3. The change multiple Os throughout the training is shown in Figure 7. We further plot the change of {ug L 500, & e 2, 0 S k S 500} for three Os. The visualizations are given in Figure The visualization shows the update directions of Os align with each other, even for neural network without training.
15

---

Published as a conference paper at ICLR 2025
regressor
regressor
regressor epoch epoch
opach
epoch 2
epoch 2
-2
-2
epoch 3
4
6
8-
0
4
8
(a) epoch to
(h) epoch 2
3
(c) epoch 3 to
0
0
regressor
regressor
regressor
-2
epoch
epoch 5
-2
epoch
epoch
epach 6
epoch
-4
-4
-6
-6
-8
-8
2
4
6
8
0
2
2
(d) epoch 4 to
(e) epoch 5 to 6
(f) epoch 6 to 7
0
regressor epoch 7
regressor epoch
regressor epoch 9
-2
-2
-2
epoch 8
epoch
epoch
44
4-
-6
-6
-8
0
2
0
6
(g) epoch 7 to 8
(h) epoch to 9
(i) epoch 9 to 10
Figure 6: Change of z between adjoin epochs.
16

---

Published as a conference paper at ICLR 2025
regressorD regressor1
0.0004
regressor2
0.0003
Cosine distance
0.0002
0.0001
0.0000
10
Epoch
Figure Change of the multiple As.
regressor 1
regressor 2
regressor 3
original point
0.000075
0.000050
0.000025
0.000050
0.000075
-0.00005
0.000100
0.00000
0.000125
0.00005
0.00010
0 00005
0 0.00000
0.00010
-0.00005
-0.0000000100
-0,00010
-0.00015
Figure Change of Os within the iteration [0, 500].
17


Reviews:
[{"review_id": "lFGMtBpY5V", "reviewer": "Reviewer_uwbB", "summary": "This work addresses the fundamental yet underexplored problem of regression, arguing that limited research attention has been devoted to it compared to classification tasks. The authors posit that conditional entropy H(Z|Y) , which is extensively studied in information bottleneck, is important for regression problem. They term this entropy measure “tightness” and argue that minimizing it is essential for regression tasks.  Towards tighter representations, the authors propose two strategies, which are multi-target and an optimal transport-based regularize. Experimental results show that both of the proposed strategies can improve the regression performance.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method achieves superior performance compared to prior deep regression techniques on two benchmark datasets, demonstrating its effectiveness.\n\n+ The authors provide an interesting analysis on why ordinal feature spaces are not naturally learned under typical regression loss functions, highlighting a crucial aspect often overlooked in regression tasks.\n\n+ The authors offer a comparison between classification and regression, explaining why classification losses tend to better constrain (or “tighten”) representations, which may help readers understand the distinct requirements of these tasks.", "weaknesses": "Although the authors discuss why minimizing Mean Squared Error (MSE) may fail to learn ordinal feature spaces, they do not provide empirical results or visualizations, such as t-SNE plots, to support this claim. Comparative visualizations between the proposed method and RankSim would strengthen the discussion\n\nInconsistency between Eq 3 and Eq 5 – Should both of them be from the batch level? For example, change N to b?\n\nThe rationale for employing multiple regressors is unclear. Why is a multi-regressor approach potentially better than a single one? Is this analogous to an ensemble method? Additionally, will the different ranges of Y affect the selection of M?\n\nIn [1], the authors discuss the neural collapse phenomenon can be similarly achieved by deep networks when minimizing MSE, which seems somewhat contradictory to statements made in the current paper. Is this contrast due to the nature of the target label (discrete versus continuous)? Expanding on this point with further discussion and comparison would add clarity.\n\n[1] NEURAL COLLAPSE UNDER MSE LOSS: PROXIMITY TO AND DYNAMICS ON THE CENTRAL PATH, ICLR21", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "foHKCwK4Zw", "reviewer": "Reviewer_QZxF", "summary": "The paper proposes enhancing deep regression by preserving the ordinality of target values within the feature representation space, which is linked to reducing the conditional entropy between the learned representation and the taget. Given typical regressors struggle to naturally reduce the conditional entropy, they propose two solutions for the problem. The first is to use augmenting for the target to mimic training in classifition, so that the model parameters can be with more flexible and diverce updating directions. The second is to use ptimal transport to encourage the similarity between target and feature spaces. Results in some real-world tasks demonstrate the effectiveness of their solutions.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is overall well-written. The proposal of linking conditional entropy with ordinality preserving  in regression also seems new and interesting.\n\n2. The authors propose the ROT Regularizer and a multi-target learning strategy, both of which are innovative methods for improving regression. These techniques address the limitations of standard regression by refining the feature space structure and better preserving relationships among targets, enabling more robust and accurate predictions.", "weaknesses": "I have limited knowledge in this specific area and currently lack the expertise to identify any potential weaknesses in the paper. The overall manuscript appears satisfactory to me. The authors may gain additional insights for refinement through feedback from reviewers more experienced in this topic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kb6adrGaIR", "reviewer": "Reviewer_CpDH", "summary": "This paper proposes a deep regression method focusing on the tightness of features.\nThe authors found that the tightness of features is important but not sufficiently optimized in normal regression training \nbecause the updated directions of features are limited in normal regression training.\nBased on this finding, the multiple target (MT) strategy and regression optimal transport regularizer (ROT-Reg) are proposed to promote tightness during training.\nMT increases the number of regression heads so that features are updated toward various directions.\nROT-Reg closes the topology gap between features and targets via self-entropic optimal transport.\nExperimental results show that the proposed methods improved the tightness of features and regression performance.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- S1: The finding that the updated directions of the features are limited in normal regression training is intriguing.\n- S2: The finding is theoretically supported.\n- S3: The paper is well-organized and easy to follow.", "weaknesses": "- W1: The difference between the global and local tightness is ambiguous. While $\\mathcal{H}(\\mathbf{Z}|\\mathbf{Y})$ is called tightness, formulating the global and local tightness would strengthen the theoretical analysis.\n- W2: The justification of design choices of the proposed methods needs to be clarified.     \n  - For MT, is there a possibility that the multiple regressors will collapse into a single solution? Are the solution spaces $S_y$ orthogonal?   \n  - For ROT-Reg, is the self-entropic optimal transport the best choice? For example, one may simply minimize the gap of affinity matrices in Eq. (9).\n- W3: The performance improvement by the proposed method is marginal. For instance, in Tab. 2, why did incorporating MT and ROT-Reg underperform MT?\n- W4: It would be beneficial to visualize the improvement of tightness not only in toy datasets but also in real-world datasets with e.g., PCA.\n- W5: Displaying the regression performance in Tab. 4 would be helpful to strengthen the efficacy of tightness on the performance.\n- W6: There are many typos. For instance,    \n  - L185: $\\partial$ is missing in partial derivative   \n  - L113: differentiable entropy -> differential entropy?   \n  - L252: \"$=$\" -> \"$\\in$\" ?   \n  - L317: \"performance the performance\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)