Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper researched on the recently proposed long-tailed regression problem with self-supervised learning method. Two questions are investigated by this paper: 1) how to measure the similarity and dissimilarity under the regression sense; 2) it is not guaranteed that the sampled with perturbations are similar to the original samples without any noise. The former problem is addressed by providing a formal definition of similarity and the later question is addressed by limiting the volume of noise on the output. Authors report the results on three datasets to demonstrate the superiority of our proposed methods, including IMDN-WIKI-DIR, AgeDB-DIR and NYUD2-DIR.
main_review: Pros:

[1] The imbalanced regression problem studied by this paper is of great significance not only to the academic research but also to the real-world applications. 

[2] Authors did comprehensive experiments on several popular benchmarks to show the effectiveness of the paper.

[3] How to do noise generation, which ensures a high degree of similarity between the disturbed sample and the original sample, is very intuitive and easy to follow.

Cons:

[1] The performance improvements to current state-of-the-art method is quite marginal. For NYUD2-DIR dataset, the improvements to RMSE and delta1 are 0.05 and 0.0, respectively. For AgeDB-DIR dataset, the improvements to MAE and EM are 0.12 and 0.16, respectively. For IMDB-WIKI-DIR dataset, the improvements to MAE and GM are 0.15 and 0.11, respectively. 

[2] The performance of the proposed method on few-shot classes is much worse than the previous SOTA on IMDB-WIKI-DIR dataset, as shown in Table 1. Is there any reason to explain it? 

[3] In Figure 1(a) and Figure 2(b), the author drew several conclusions about noise-related issues, including the sensitivity to noise and the impact of noise, based only on one selected sample. Such a special example is difficult to be convincing enough, especially when we are dealing with regression tasks that contain many outlier/noisy samples. My suggestion is that the author can try to provide statistical analysis on the overall datasets, and quantitatively compare the benefits of noise generation method against vanilla random noise method. 

[4] The idea of integrating self-supervised learning with imbalanced recognition has been investigated by previous works (1). Although this paper is working on a different task, i.e. regression, the technical contribution of how to integrate self-supervised learning with imbalanced learning is relatively limited.

[5] There are several typos in the paper, for example, "Update ri by by taking" in the algorithm workflow "Algorithm 1: Self-Supervised Imbalanced Regression (SSIR)" should be "Update ri by taking". Authors may want to double check the draft.

(1) Rethinking the Value of Labels for Improving Class-Imbalanced Learning. NuerIPs 2020.
summary_of_the_review: Although the imbalanced regression problem is very interesting and import, I have concerns on the technical contribution and effectiveness of the method. Therefore, my current rating is: "5: marginally below the acceptance threshold"
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel algorithm SSIR to address the imbalanced problem in regression tasks, which seamlessly combines self-supervised learning and imbalanced regression by giving the formal definition of similarity in the regression task. Besides, the authors specifically propose to limit the volume of noise on the output, and in doing so to find meaningful noise on the input by back propagation. Experimental results show that our approach achieves the state-of-the-art performance.
main_review: Strengths: 
This paper seamlessly combines self-supervised learning and imbalanced regression by giving the formal definition of similarity in the regression task, and proves that self-supervised learning really relieve the long-tailed regression problem. The Experiments are substantial.

Weaknesses: 
1)  In the penultimate paragraph of Section 4: “On the other hand, considering that a single back-propagation optimization may not be an accurate approximation, we therefore performed a two-step optimization of r.” Why a two-step optimization can be helpful for obtaining an accurate approximation? Please give an explanation.
2)  In Section 4, why the authors apply noise to the output instead of the input? How to solve the imbalanced regression problem when applying noise to the input?
3)  In Section 3, Data generation, why the test set generated by the authors is balanced? How would the proposed method perform if the test data is unbalanced?
4)  In Section 3, the fourth line of second paragraph: y=w^Tx, the vector transpose symbol ‘T’ should not be bolded.
5)  The sentence: “the left image suggest a greater change with augmented data.” “suggest” should be “suggests”.


summary_of_the_review: I think this paper in its present form cannot be accepted for publication in ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to leverage self-supervision to improve imbalanced regression. An augmentation method is designed to keep the regression target after augmentation close to the original target.
main_review: Strength:

1. Although the self-supervision signal has been discussed under the context of classification, leveraging it in the imbalance regression task is novel.
2.  The paper identifies and addresses an important problem: how to define positive pairs for the regression task and what is a valid augmentation to produce a positive pair. Solving the problem is a necessary step to introduce more self-supervised methods into the imbalanced regression domain.
3. The solution proposed in the paper is novel. Augmenting on the label space and reversely finding the mapping in the image space is interesting. The method also shows promising empirical results in Tables 1,2, 3.

Weakness:

1. It is unclear why only noise-based augmentation is discussed in the paper. 

   The paper only discusses noise-based augmentation. However, random noise is only one of the many augmentation techniques. There are a number of other augmentations popular in self-supervised contrastive learning, for example, random cropping, random blur, and random color distortions. Can they transfer to the regression task? Can the proposed method apply to these augmentation techniques as well?
2. The argument about "infeasible migration" in the Introduction is weakly supported and lacks empirical evidence.

   The paper argues that "In the case of regression, however, it's not at all feasible when a similar migration was made (from classification to regression) ". The authors support the argument using an illustration in Figure 1, where regression models make drastically varying predictions to different noises and hence prove that the random noises ambiguate regression targets. 

   However, the support is a little weak considering that a classifier will also make wrong predictions when noises are added [1]. For example, in Figure 1 (a), the cat could be classified as a dog/bird when strong enough noise is added. In another word, the illustration does not fully convince me why the use of noise should be different on classification and regression and why the "migration" is not feasible. Why would simple thresholding on the noise level like in classification [2] fail on the regression task?

   Moreover, the empirical evidence for the argument is missing in the paper. There lacks a contrastive learning baseline that uses random noise to produce positive pairs like in the classification tasks. Compared with the sophisticated data generation procedure proposed in the paper, it would be interesting to know how a naive random noise augmentation performs. 
3. The settings in Motivation (Section 3) and Method (Section 4) are inconsistent, which undermines the paper's motivation.

   In the motivation section, self-supervision is used as a pre-train and steady improvement is observed in Figure 2. However, in the method section, self-supervision is used as an additional training objective (Equation 11). Why not keep the pre-training setting in the method? 

4. There are some minor issues with the parameter study.

   The paper concluded that the proposed method achieves the best performance with $\epsilon = 1e-3, \lambda=1.0$. However, no results are provided when $\lambda>1$.



[1]: Zheng et al., Improving the Robustness of Deep Neural Networks via Stability Training, CVPR 2016

[2]: Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020.



summary_of_the_review: I recommend a borderline acceptance. Although the motivation of the paper is not fully justified and some discussions are too limited to a specific domain (random noise augmentation), the paper explores a new topic and provides a technically sound solution. My overall attitude towards the paper is positive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies long-tailed regression and explores self-supervised learning to alleviate class imbalance. More specifically, the work broadens the concept of similarity and dissimilarity between noise samples and original samples from classification to regression, and develops a new self-supervised learning based method for long-tailed regression.  
main_review: Positive points:
1. This paper studies an important and practical task, namely long-tailed regression. The importance of this task can also be supported by a recent survey of deep long-tailed learning [1], which highlights long-tailed regression as an important future direction. 


2. This work explores contrastive self-supervised learning (CSL) for imbalanced regression, and finds the difference between CSL used for classification and CSL used for regression. That is, the regression is more sensitive to data augmentation (i.e., adding noise).  This may inspire future research when using CSL for imbalanced regression. 


3. This paper broadens the concept of similarity and dissimilarity between noise samples and original samples from classification to regression, and develops a new self-supervised learning based method for long-tailed regression. Extensive results verify the effectiveness of the proposed method.


Negative points: 
1. In the definition of similarity and dissimilarity, it is unclear whether the model should be well-trained when evaluating the similarity or dissimilarity of two samples. Moreover, what is the distance function D? Do different distances influence a lot? The definition of the similarity and dissimilarity in the regression would be more solid if the authors can make these concepts more clear.


2. It is unclear why the proposed self-supervised regression helps to handle imbalance. Based on the current method, it seems that self-supervised regression helps to train a more noise-invariant model. Although such an operation may empirically improve the performance of long-tailed regression (on some baselines in experiments), it is unclear how it helps to handle class imbalance. Does it learn a more balanced feature space? I am okay if the authors want to leave this as future work, but I still expect the authors to discuss it. 


3. Since this study explored a self-supervised learning based method for long-tailed regression and discussed contrastive self-supervised learning, I really suggest the authors to review more recent contrastive learning based long-tailed studies, in addition to the mentioned KCL (Kang 2021). According to the survey [1], contrastive learning based long-tailed methods also include Hybrid [2], PaCo[3] and DRO-LT [4].



Minor suggestions:
1. In the abstract, the noise is not defined and may confuse readers. The writing can be further improved.


2. The caption of figure 1 is a little confused, and I suggest the authors further improve it.


3. In the first paragraph of Section 5, it would be better to add citations for baselines.


References:

[1] Deep long-tailed learning: A survey. ArXiv, 2021.

[2] Contrastive learning based hybrid networks for long-tailed image classification. In CVPR, 2021.

[3] Parametric contrastive learning. In ICCV, 2021.

[4] Distributional robustness loss for long-tail learning. In ICCV, 2021.


summary_of_the_review: Overall, I like this paper because of its task importance and the new formulation of self-supervised learning for long-tailed regression. If the authors can fix the negative points mentioned above, this paper would be more solid and thorough.
review_title: null

Meta-review: This paper proposes to use self-supervised learning in the context of "imbalanced regression", where some values of the outcome variables are rare, such as in long-tailed regression. The author's proposal can be interpreted as a Monte Carlo approximation of a density smoothing technique, akin to Yang et al. 2021. They test their approach on three datasets. Overall, it provides marginal improvements, whose statistical significance are not assessed. All reviewers agreed that the paper has merits but that it should be further improved to demonstrate that the proposed method is indeed a step forward  in solving the problem of imbalanced regression. The authors should also provide stronger motivation for their pipeline details and experimental setup choices. I therefore recommend rejection, with encouragement for improvement in two directions: strengthening the experimental section, in particular by assessing statistical significance, and by improving the writing of the paper by developing a more rigorous exposition.
Decision: Reject