Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed to loss term for deep regression models to increase the entropy of the learned representation and with that increase the performance of the predictive model. An additional term is also proposed to decrease the entropy of the class conditioned representation.
strength_and_weaknesses: Strengths
-Well written.
-Technically sound.
-Analytical and intuitively well motivated.
-Novel contributions to deep regression, of wide application.
-Sound proposed methodology.


Weaknesses
-minor typos. For instance, "The simplest mapping The feature encoding..."
-complexity/efficiency discussion. Since the proposed loss term seems to be quadratic in the number of observations, it would be interesting to discuss the efficiency or the impact on the batch size.
-in ordinal regression (also known as ordinal classification), there are works adding a regression component to the classification loss. Although quite different from the current proposal, it may be interesting to comment and put into perspective in the related work.
https://arxiv.org/abs/2202.05167
https://aclanthology.org/2022.coling-1.407/



clarity,_quality,_novelty_and_reproducibility: Very good.
Clear, of high quality and mostly reproducible.
summary_of_the_review: see above
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors analyze multiple deep learning prediction cases with continuous targets where discretizing the target into a classification problem results in better results compared to directly employing regression loss functions. The authors argue that the root cause of the problem is that the regression objective only optimizes the conditional entropy H(Z|Y) but not the marginal entropy H(Z), whereas the cross-entropy objective optimizes both. The authors then propose a remedy to the situation by incorporating a regularizer to also maximize H(Z). The author finally demonstrates that the regularized improve the performance of the regression model.
strength_and_weaknesses: **Strengths**:
- The authors aim to provide an explanation of the phenomenon that discretizing continuous targets into classification often time performs better than optimizing the original loss.
- The proposed regularizer is motivated by the ordinal entropy.
- The author shows the performance benefit of the regularizer compared to just optimizing the MSE loss.

**Weakness**:

Even though the authors aim to provide an explanation of why converting continuous tasks into a classification is better than running a regression, I don't think the provided analysis is sufficient enough to establish the claim. First, the analysis that the regression objective only optimizes H(Z|Y) is an interpretation/approximation under certain assumptions. The authors establish the relation, which says that the MSE is a proxy for maximizing H(Z|Y). However, the analysis cannot say for sure that the MSE also optimizes something else (like a proxy to H(Z)) as the relations are not exact. In addition, the analysis is only conducted on a linear regressor, not a very non-linear deep network. An analysis that works for linear regressor cases does not always translate to deep neural network settings. Therefore, a more thorough theoretical and empirical analysis needs to be conducted to establish the such claim.  

In the experiment, the authors did not compare the results with classification-based networks. This is important to back up the claim made by the authors that by modifying the continuous loss, the regression objective can outperform the classification baselines.
clarity,_quality,_novelty_and_reproducibility: There are some clarity issues, like an unfinished sentence in the second paragraph of section 3.1. In Lemma 1, even though the statement mentioned \eta as consideration, it does not appear in the stated inequality. The authors are recommended to present it such that it is clear to the reader.
summary_of_the_review: The paper contains some weaknesses in the claim.
Therefore, I recommend rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an analysis of regression's reformulation as a classification problem and proposes an ordinal entropy loss to learn high-entropy feature representations which preserve ordinality based on the analysis.
The proposed method outperforms the previous regression loss, classification loss, and other comparison methods in various benchmark datasets. A thorough ablation study also properly supports the author's claim.


strength_and_weaknesses: [Strenght]
+ the analysis and finding of regression's reformulation as a classification problem.
+ propose an ordinal entropy loss to learn high-entropy feature representations which preserve ordinality
+ significantly improved performance on various datasets and tasks.

clarity,_quality,_novelty_and_reproducibility: [Clarity & Quality & Novelty]
+ The paper is well-written and easy to follow.
+ The analysis and proposed loss have high contribution and novelty.
- The source code is not currently available.

summary_of_the_review: Based on the strength, weaknesses, and novelty, I think this paper is novel and has high contributions. But I am not absolutely certain about my decision, I will adjust my decision after checking other reviewers' opinions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors analyze the observation that reformulating regression as classification often yields better performance by better learning high-entropy feature representations. To retain the benefits of both high entropy and ordinality of feature learning, the authors propose an ordinal entropy loss to encourage higher-entropy feature spaces while maintaining ordinal relationships for regression tasks. Extensive experiments demonstrate the empirical improvement of the method. 
strength_and_weaknesses: Strength:

(1) I like the analysis of regression and its reformulating classification through the lens of mutual information and entropy. It does theoretically explain the difference between regression penalized by MSE and regression penalized by cross-entropy. Besides the empirical improvement, I do value this theoretical finding. 

(2) The idea is well-presented and easy to follow. The experimental demonstration is also convincing and extensive. 

Weakness:

(1) The authors should consider conducting an ablation study on $\lambda_d$ and $\lambda_t$. Is the final performance sensitive to these two hyperparameters? 

(2) Are Eq. (8) and (9) defined on the mini-batch or the entire dataset? How does adding tightness and diversity terms increase the overall training cost? 

(3) Missing related work [1]. The idea of preserving ordinality by using the label distances is interesting and technically sound. A similar idea in the case of preserving ordinality in regression is also used in [1]. The authors should discuss their difference.

(4) Minor point: I encourage the authors to further proofread the manuscript. I can find some of the typos as follows:
- Section 3.1 P2 L3 ‘The simplest mapping The feature encoding’
- Sec. 3.2 P2 L2 ‘being a being composed’
- Introduction Paragraph 4 ‘and regress’ to ‘and regression’
- Section 4 P3 L2 ‘is precision enough’



[1] Gong et al., RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, ICML 2022
clarity,_quality,_novelty_and_reproducibility: Clarity: 

[+] The presentation of the idea is clear and easy to follow. The experimental demonstrations are also convincing and solid. However, the authors should proofread the manuscript and pay more attention to the typos. 

Quality:

[+] The theoretical and empirical demonstrations are both clear and technically sound. 

Originality: 

[+] To the best of my knowledge, I think this work is novel and can be a contribution to the ML community. 

Reproducibility:

It would greatly improve the contribution if the authors can release the code. 
summary_of_the_review: The authors propose an ordinal entropy loss with the original regression task loss to improve the regression performance, based on the theoretical analysis that reformulating regression as a classification problem can often improve the performance. The method is also experimentally demonstrated on multiple different regression tasks. Therefore, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper gives an explanation on why regression model underperform with respect to classification based regression method from an information theory point of view and propose a regularizer based on ordinal entropy  that improves the regression with MSE loss. 

The paper was discussed between authors reviewers and AC.

The main strength of the paper is in analyzing under certain assumptions the performance of MSE loss. The main weakness of the work is that the analysis is limited to linear models and to restricting assumptions and does not explain what happens in the deep setting and beyond the assumptions listed in the paper. 

Overall the reviewers had positive views on the work with the exception of reviewrer 1mqu. We encourage the authors to be more precise in listing the contributions of the work and to take into account the constructive  feedback of Reviewer 1mqu on better scoping the work and to not extrapolate the results to other settings without any proof. 
justification_for_why_not_higher_score: The paper claims are interesting but limited to linear network settings 
justification_for_why_not_lower_score: Analyzing why classification method outperforms regression method is an interesting topic.
Decision: Accept: poster