Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the deep regression problem and proposes a new automated label encoding learning framework along with two regularizers. In specific, the authors relax the assumption of binarized label encodings of the BEL method and propose to search for continuous label embeddings. Moreover, two regularizers are introduced. The first restricts the learned label encoding to preserve the distance of the bits. The second is to enrich the bit transitions. Experiments show the proposed method obtains competitive results to manually designed label encodings.
strength_and_weaknesses: Pros:

1. The handcrafted label encoding, though effective, increases the difficulty of the practical deployment of deep regression models. This work proposes an automated label encoding learning method that is much more practical. 

2. The experimental results are promising as the proposed method achieves competitive or better performance than manually designed label encodings. 

3. The paper is well-organized and mostly written clearly. 

Cons:

1. While the presentation is generally good, the methodology is hard to follow. Some notations are not clearly defined. For example, the authors used B, Z, E to denote the label encodings. What are their differences? Why E is the average of E? What is \mathcal{E}(Q_i) on page 5?

2. The 2.1 section (and Figure 2), while important and informative, isn't really clear to me. Why is the bit transition from 1->0 and 0->1 important for label encoding? Also, how does the bit position measures the binary classifier's decision boundary? Personally, I recommend the authors polish this section for a clear explanation of the important properties of label encodings.

3. I noticed that the softmax operator and the cross-entropy loss are used. It suggests the regression network learns a multi-class classifier. While this paper frequently mentioned the binary classifier, I would like to see the difference between these two designs. In particular, it seems a multi-class classifier may reduce the complexity of the encoding space.

4. The authors propose to regularize the matrix D instead of the original label encodings in Eq. (4). While empirically effective, there are no theoretical insights on why it works. 

clarity,_quality,_novelty_and_reproducibility: This paper is overall well-written and the proposed method is original. But some notations and analyses are confusing. Source code submitted.
summary_of_the_review: See above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an end-to-end automated approach to find label encodings for deep regression problem. In my understanding, these tasks are usually using images as input and are outputting a continuous value. Due to the nature of deep regression, a general approach to regress directly does not show good results compared to task-specific approaches. Their work is largely based on a prior work [Shah 2022], in which 1) binary-encoded labels were introduced to encode quantized regression values 2) three desirable properties of suitable encoding/decoding functions were identified. This paper extends [Shah 2022]'s work by not using manually identified binary codes, but learn the continuous label encodings from the data directly. Their empirical experiments show that their learned-encodings and the new model result in lower or comparable errors to [Shah 2022] work.

[Shah 2022] Deval Shah, Zi Yu Xue & Tor M. Aamodt. LABEL ENCODING For REGRESSION NETWORKS. ICLR 2022.


strength_and_weaknesses: Strength: clarity and quality of writing. Extensive experiments.

The paper is well written with a pretty clear introduction of background and related work. I also like how the author motivates the underlying work by showing the challenges in the design space. By giving a literature review on the binary encoding's usage in regressor, I was able to understand why the properties of suitable encodings (identified by prior works) are important in such a work. By following this guidance to the properties, the author started to introduce the regularization functions to encourage such properties when using real-valued label encodings. The loss function is then introduced based on the regularizer design. The experiments use almost the same datasets (from Shah 2022) so that a fair comparison with its predecessor can be easily conducted. The author also compares with a few other label encoding design approaches and regression approach to show that their work is a generic regression approach and generalizes better with even a small dataset.

Weakness: not strong originality.

Most of the originality in this paper is to propose a learned continuous label encoding (compared to existing work on using manually designed binary encoding). From the regularizer design, the author identifies effective regularization functions to encourage the expected property within a continuous encoder. The proposed method seems effective but incremental in nature. 

Questions to the author:
1. How would you decide the number of quantization buckets ($M$ in the continuous predicted encoding $Z_i$)? Is there an optimal dimension size for various dataset?
2. Computationally (during training or inference), is there any significant difference between RLEL and BEL?
3. In equation (3), can you give more explanation that why you choose the magic number 2?
4. It seems to me (by looking at Table2) BEL is reasonably performing well though it is manually designed. I kindly expect that adaptively learned encoder shall perform even better if more data is given during training. I am not familiar with the scale of the experimental dataset you have used. How many examples are there in each of them? Were you able to test your method in a significantly large-scaled problem (e.g., with many more training examples) to show the superiority of your model)?
5. Does RLEL use uniform quantization (similar to BEL)? Is there something that can be done to adaptively select quantization level and non-uniform quantization?  
clarity,_quality,_novelty_and_reproducibility: The quality of writing is excellent. The author provides extensive experiment setting and code in the appendix and supplemental materials. I believe it will be sufficient to reproduce the work. 

Minor comment:
In figure 1(a), the legend (the blue dot vs red dot is too small) is almost unrecognizable.

summary_of_the_review: Weak accept. The paper proposes an effective way to automatically learn continuous encoding to help improve deep regression model performance. If the author could give more theoretical analysis on why the proposed model encourages the expected encoder properties, I would be happy to increase my scores.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on deep regression by using label encoding. Specifically, this paper proposes Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encodings. RLEL contains two regularization terms, designed to encourage encodings with certain properties. The first one encourages the rule that the L1 distance between label encodings should increase as the difference between two labels increases. The second one encourages the rule that the L1 distance between label encodings for adjacent target label values should be minimized. The effectiveness of the proposed method is demonstrated by a lot of experiments.
strength_and_weaknesses: Strengths:
- This paper proposes a novel label encoding method to solve regression and seems can alleviate the disadvantage of the previous label encoding method, the space of label encodings for regression is large.
- This paper introduces two simple yet meaningful rules for label encoding and designs corresponding regularization terms to encourage label encodings to satisfy the rules.
- Comprehensive experimental results on various datasets clearly demonstrate the effectiveness of the proposed method.

Weaknesses:
- According to Eq. (5), the generation of soft target labels is affected by the batch size K. It seems impossible. So, I guess it should be the number of target labels N rather than K in the generation of soft target labels. 
- The authors should use unified representation. For example, the correlation vector is denoted by $\hat{C}_i$ in most cases. However, $C_i$ is used in Eq. (5) and Fig. 3.
- The authors assume $y_i \in [1, N]$ for simplicity as the real-valued targets with any arbitrary numeric range can be scaled and shifted to this range. However, they do not provide details and discussions of N. Obviously, different N would cause different performance.


clarity,_quality,_novelty_and_reproducibility: This paper is written well and organized well. Although some details and discussions of the number of soft target labels N are missing, this paper has done well in other aspects.
summary_of_the_review: Overall, this paper proposes a novel label encoding method that is reasonable to solve the deep regression problem. There are also some shortcomings in this paper as mentioned above. The authors should carefully check the manuscript repeatedly and provide necessary details and discussions to further polish this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encoding, by combining continuous label encodings space with regularizers. The proposed method is extensively demonstrated on 11 benchmarks. 
strength_and_weaknesses: strength:

(1) The authors tackle the problem of deep regression through the lens of binary classification. It is also interesting and technically sound to extend the conventional hand-crafted encoding method to be differentiable. Overall, I like this simple yet useful idea. 

(2) The experiments are extensive, and the proposed end-to-end encoding method has competitive performance compared to hand-crafted. The ablation study also demonstrates the effectiveness of two proposed regularizers.

Weakness: 

(1) In the experiments, manually designed BEL outperforms RLEL in many cases. I'm curious why an end-to-end training method seemingly achieves a suboptimal solution, compared to a hand-crafted method. Can the authors provide some intuition on it? Also, how different is it between the learned encodings and (better performed) hand-crafted encodings? 

(2) I encourage the authors to discuss the efficiency of the method. What is the time cost to incorporate the label encoding learning with regularizers, compared to other hand-crafted encodings? 
clarity,_quality,_novelty_and_reproducibility: Clarity:

The manuscript is well-presented and the idea is simple and easy to follow.

Quality:

The method is extensively demonstrated on 11 benchmarks with significant improvement upon baselines. The ablation study also demonstrates the effectiveness of the proposed regularizers.

Reproducibility:

The authors have their code attached, and I think the implementation details provided in the appendix are sufficient.  
summary_of_the_review: The authors propose a simple yet effective end-to-end training to simultaneously learn the label encoding and features. The experiments are solid and extensively demonstrate the effectiveness of the method. Therefore, I recommend borderline acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studied indirect regression based on deep learning that is to perform regression tasks by an encoding-classification-decoding pipeline. This indirect approach is recently shown to be better than direct regression but its label encoding needs to be manually crafted. This paper made it end-to-end with two intuitive and effective regularizations and empirically demonstrated that the learned label encoding can be superior to the crafted one. All reviewers agreed that the novelty and significance are above the bar and thus we should accept it for publication.
justification_for_why_not_higher_score: I am not sure how many oral presentations we will have, but perhaps this submission is not that competitive to make an oral presentation.
justification_for_why_not_lower_score: The paper is quite good and studied a very fundamental problem (i.e., regression), so I think many researchers/practitioners may be interested in it.
Decision: Accept: notable-top-25%