Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a general framework to solve a regression problem as a binary classification.
The labels (targets) are binary encoded and hence the regression problem is solved using a set of binary classifiers.
Since the process is dependent by the type of Encoding and Decoding used, the authors compare different type of configurations showing, empirically, the effect of different design parameters on the final accuracy.
main_review: Strengths:
- The paper is detailed with a lot of information useful for the reproduction of the experiments.
- The theory of the binary-encoded label is well described also with clarifying examples.
- The analysis of Encoding/Decoding functions has been conducted for a simple Unary coding (BEL-U) and the Johnson coding (BEL-U) (in the Appendix).
- The authors introduced an estimation of the error probability for BEL-J and BEL-U and an interesting relation between the two methods (in terms of the percentage difference between errors).
- Even though the usage of classification for regression problem is not original, the work try to shed lights on some design choices.

Weaknesses:
- The paper is generally well written but I suggest to revise the organization because some parts are introduced after they are effectively used and this can make confusion. 
For example: 
  - The Unary code, Johnson Code and other Encodings are described on 3.2 but Unary and Johnson are used already in 3.1. I suggest to anticipate the description of the encoding functions or at least for Unary and Johnson ones.
  - The name of BEL-J decoding is not introduced neither in 3.1 nor in 3.3 but in the appendix and it is used in the caption of Figure 5.  
- For the two detailed codes (Unary and Johnson), I suggest to describe clearly the Encoding and Decoding functions and then the computation of the error and other aspects present in sections 3.1, 3.2 and 3.3. For this reason I think that the Sections 3.1, 3.2, 3.2 should be revised in order to improve readability. 
- According to Figure 4, the authors seems to have used only 1 Fully connected layer to connect feature vector to the output. In the Figure 13, there is a "feature extraction extension" but, generally speaking, it seems that only a shallow network is present between the feature extractor and the output layer. In this situation the using of binary-encoded classifier seems to present some advantages. 
Unfortunately, no tests have been performed to understand if increasing the number of Fully connected layers impact on the results.
- The BEL approaches outperform the other tested approaches but the best design parameters depend on task and dataset. As authors said this is left for the future works but the doubt is that with more layers the situation could change.
 

Other comments:
- Figure 1(a) the caption should be "(a) Training (top) and inference (bottom) flow"
- Instead of use "MAE/NME" I suggest to use "Error (MAE or NME)" as done in Table 2 since the currently used name could be misleading.
- Why in Table 2 some approaches (all specialized approach and some direct regression) don't have the indication of standard deviation? 
- Caption of Figure 5 should be improved.

Some typos:
resepct --> respect
deocding --> decoding
summary_of_the_review: The authors propose a general framework to approach regression problems using binary encoding and a set of classifiers. The paper is detailed, some interesting theoretical aspects are discussed and experiments seem to demonstrate the high accuracy of proposed approach even though only using shallow network after the feature extractor.

=====POST-REBUTTAL COMMENTS========

The added experimental results and manuscript modifications address all of my concerns. 
I have raised my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles regression problems using an error-correcting code (ECOC) approach, i.e., reducing a *regression* problem into multiple binary *classification* problems.
ECOC have been so far mainly used for classification tasks, and their application to regression problems in this paper is elegant and seems novel.
The paper examines several combinations of (encoding method $\times$ decoding method $\times$ loss function) on several (deep) regression tasks and reports significant improvements over existing methods.
Some brief analytic discussion is made.
main_review: # A. Strengths

1. **Novelty.**  
  Error-correcting codes are used in ML for classification tasks since at least 1995. To the best of *my* knowledge, the authors are the first to propose using such codes in regression tasks. This is interesting.

1. **Reported empirical performance seems much better than other methods.**  
The results speak for themselves (especially in Table 2). 
I should say that I am not an expert in deep regression tasks, and cannot say whether the methods the authors compare to (e.g. Ruiz et al. 2018 or Behera et al. 2021) are sufficient.
Moreover, I have some reservations regarding the comparison to the simplest baseline (the direct regression baseline) which I address below.

# B. Weaknesses

1. **Empirical advantage over other methods should be more carefully demonstrated.**  
Since the reported results are *remarkable* and are claimed to be SOTA, I feel like "extraordinary claims require extraordinary evidence".
Importantly, no code is provided, thus I feel even more obligated to verify that the comparisons are fair.
Especially, I am bothered by the comparison to the "direct regression" baseline. 
It seems like the feature extractors' parameters were *frozen* when comparing direct regression to the proposed BEL methods.
In this case, I suspect the added linear layer (reduction to $\theta$ features) significantly helps the model, and that its absence from the direction regression models makes the comparison *unfair*.  
I would even say that I would be slightly surprised if when the pretrained feature extractors will be also learned (rather than frozen), the direction regression approach will not be competitive with the proposed BEL method.  
I therefore decrease my general score, until more evidence and details are provided.

2. **Readability issues.**  
Throughout my reading, several points were not very clear, and it made the reading slower and more difficult than actually needed.
For instance,
* The experimental part is especially hard to understand. It was unclear what method exactly is the "Multiclass classification" one. Appendices D-G are also hard to interpret. The reader is expected to cross between many tables that use different names for the same methods. For instance, see Table 11. How can one distinguish between results for HPE2 and results for HPE4?
* Usage of the "decision boundaries" term is untraditional and can be confusing.
* Description of the HEXJ encoding is hard to understand.
* Paper says "We evaluate the Hadamard code to show that..." but the results are not actually shown anywhere.
* Figure 1b is hard to understand, especially on B&W hard copies.


3. **Analytic results do not contribute much.**   
Section 3.1 offers some basic analysis of two encoding-decoding combinations (using the Unary and Johnson codebooks).
I am not sure what is the contribution of this analysis to the rest of the paper or to the readers' understanding.

# C. Other comments / thoughts

1. I allow myself to assume the Hadamard codebook fails because each column has many "bit flips" (i.e. decision boundaries in the authors' terminology). This implies that traditional classification codebooks (e.g. Hadamard and random ones) are not suitable for regression tasks. 

1. Using a fine quantization, codebooks like the unary or Johnson codebooks will largely "inflate" the output space. The authors' HEXJ codebook hints that the output space does not have to grow *linearly* with the number of quantization "buckets". This is related to the use of ECOC for extreme classification (see *Learning compact class codes for fast inference in large multi class classification* [Cisse et al. 2012] and *Efficient Loss-Based Decoding on Graphs for Extreme Classification* [Evron et al. 2018]). A brief discussion in that direction can be interesting.

1. In the second paragraph of Section 3.2, the statement "we note that a Hamming distance does not..." should be justified.

1. Just before equation (1), should it be "for $k>Q_i$ instead of $k<Q_i$"?

1. The decoding function in equation (1) is not really intuitive and it feels unmotivated given the structure of the unary code.

1. In Table 2, the best FDL2 method is the specialized one, not BEL.
summary_of_the_review: Idea is novel and interesting.
Results and findings are mainly empirical and look remarkable, achieving SOTA performance.
I believe these findings require more careful description of the exact empirical process, and that as a reviewer I should make sure the comparison to other baselines is fair. A provided code would also be helpful and more assuring.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This study presents a label encoding method (binary encoded labels) to transform a regression problem into multiple binary classifiation problems. The authors designed encoding and decoding functions to encode a real-value into multiple classes and decode the classes into the real value. The proposed demonstrated improved performance on benchmark datasets.

main_review: - as far as i understood the regression network and the decoding function are trained end-to-end. It is unclear how to perform end-to-end training. what is the total loss?
- is the proposed method applicable to the prediction models other than regression networks? (although end-to-end training may not be available)
- it would be interesting to see the effect of skewness, multimodality, and outliers of the real-valued label on the effectiveness of the proposed method. they needs to be investigated to demonstrate the robustness of the method.
- also, how does the scale of the original label and any normalization of the label on the proposed method? 
- for each dataset how did you choose the hyperparameters for label encoding? e.g. which encoding/decoding functions to choos, quantization level and theta. How they can be determined for a new dataset? Although it seems they were empirically determined in the experiments, it would be great if a guideline is suggested.
- for readers' convinence, please describe what each notation is in the main body of the manuscript. (e.g. theta in Table 1 is only described in appendix)
summary_of_the_review: The idea is interesting. The method is simple yet effective. I think the some issues needs to be clarified.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper analyzes regression problems in which continuous targets are predicted using a set of binary classifiers. There is a large design space including quantization, encoding, decoding, and loss functions. An expected value-based function (D^{GEN-EX}) is proposed as a new decoding function.

The authors apply various combinations of these functions and compare them theoretically and experimentally. It is suggested that no single combination outperforms the others, but the proposed BEL approach has a potential to improve the prediction accuracy of continuous regression problems.

main_review: ## Pros
- It is interesting to consider such encoding/decoding approaches for regression problems in an application-agnostic setting.
- The end-to-end training using L1/L2 loss with the proposed D^{GEN-EX} seems reasonable, though experimentally "CE + GEN-EX" works better in some cases (D2 vs D4 in Fig.5).
- The error probability of a binary classifier is modeled well with the mixture of Gaussians, and Fig.3(c) gives a nice explanation for the comparison between BEL-U and BEL-J.
- The experiments are conducted with a wide range of regression applications.

## Cons
- I wonder how the best BEL results are selected in Table 2. If they are chosen to minimize the test error, it is not fair to compare them with the results of other approaches. A validation set should be used for model selection.
- For example, HEXJ/GEN-EX/BCE is chosen for PN and its MAE is 3.11 in Table 2, however, the same combination (D5+HEXJ) results in about 5.2 MAE in Fig.5. Why do such discrepancies occur?

## Minor comments
- I would like to see more detailed analysis about the dicision boundaries (red lines in Fig.2) and its relationship to the continuous prediction. It is denoted in p.6 that "Although the hamming distance between 1 and 7 is small, since the differing bits (C1 and C4) are far from decision boundaries and less likely to be mispredicted, a 1 is less likely to be mispredicted as a 7". I can understand the statement, however, it is not clear how such findings affect the prediction.
- In Fig.2, what do NRB mean?
- In Fig.2, I could not understand how B1JDJ and B2JDJ differ in predicting continuous targets.
- In p.7, facial landmark detection seems better abbrebiated by FLD (instead of FDL).
- In p.8, it is denoted that "For HPE4, FDL1, and AE1, Johnson does better than unary." However, I can see no such trend in Fig.5.
- In my opinion, scientific manuscripts should not refer to wikipedia.

summary_of_the_review: Overall, I vote for weak rejection. I like the motivation of the work and the authors provide a nice analysis of the problem from both theoretical and empirical perspectives. My major concern is about the clarity of the experiments in which the proposed BEL approach is compared with the existing regression models, as mentioned in the main review above.

=====POST-REBUTTAL COMMENTS========

Thanks for the authors' response. The newly added experimental results and manuscript modifications address all of my concerns. I have raised my score to recommend this paper to be accepted.
review_title: null

Meta-review: The paper focused on deep regression problems and proposed a label encoding technique which can be thought as a sibling of the famous error-correcting output codes but designed for regression problems. The main idea is well illustrated in Figure 1 at the top of page 3, where the encoder and decoder are the main objects of the proposal (and a quantizer is also needed for using the encoder/decoder which is a uniform quantizer in the paper). The idea/proposal is supported by solid theoretical arguments and convincing empirical evidences (not only the paper but also the rebuttal). While there were some concerns in the beginning, the authors have successfully clarified all the concerns and then the average score has been increased from 5.5 to 7.5. As a result, the paper is clearly above the bar of acceptance. What is more, an advantage is that the proposed method is task-agnostic and can be combined with different task-specific feature extractors borrowed from very complex regression problems (e.g., head pose estimation, facial landmark detection, age estimation, and end-to-end autonomous driving), making its significance and potential impact high. Given these facts, I think the paper might be selected as a spotlight presentation at the conference.
Decision: Accept (Spotlight)