Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a simple method for regression tasks. It improves the mixup data augmentation by selectively interpolating examples with similar labels, or re-weighting the probability of mixup pairs. Extensive experiments on various data modalities have been conducted to validate its effectiveness.
strengths_and_weaknesses: ### Strengths
* clear motivation and good presetation;
* Extensitive expermients;

### Weaknesses
I have not found the critical drawbacks of this paper. However, I have not carefully checked the correctness of sections 3.2, 3.3, and 3.4. In my opinion, section 3.1 has thoroughly presented the proposed method, and the rest provides little information but harmed the readability.

questions: 1. lines 50-56 are hard to be understood. Does Figure 1(a) only present the ShapeNet1D Pose Prediction task? At the same time, why do pair 1 and pair 3 have close input similarities? Besides, why is the regression task more sensitive to noise than the classification task? Is there any literature to support this claim, and what's noise in the whole paper (Can an image have noise?)?

2. The choice of bandwidth. I first appreciate the ablation study of bandwidth sigma in section 4.5, but it has not convinced me of the claims that mixReg reduces the efforts to tune the bandwidth. For example, the label of ShapeNet1D has a range of [0, 360], and the other datasets may have a much different value range. How to deal with this problem to choose the correct sigma? Why is the output normalized to [0, 10] in line 270? The xlabels of Figure 3 are incorrect.

3. An essential property of mixup is the robustness of label noise. As mixReg uses the distances between labels to select the pairs, will the label noise harm the generalization of mixReg?

4. Can mixReg be applied to more complex and real-world natural tasks (eg, counting or pose estimation)? Mixup has shown its effectiveness on various classification tasks such as ImageNet. This paper only presents the experiments on some simple datasets, such as Shape1D and etc.

5. Will the code be publicly available?
limitations: See above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a mixup algorithm (mixReg) to improve generalization on regression tasks. Different from mixup methods for classification tasks, the proposed method adjusts the sampling probability based on the similarity of labels. Theoretical analysis shows that the proposed mixReg can achieve a lower mean square error and improve in-distribution generalization, task generalization, and out-of-distribution robustness. Experiments on eleven datasets prove the improvements of the proposed mixReg.
strengths_and_weaknesses: ### Strengths

(1) This paper studies an interesting problem that applies mixup augmentations to general regression tasks. Since data augmentations in regression tasks are rarely studied, the proposed mixReg is the first work that introduces the general regression augmentation strategy.

(2) The proposed method in three sceneries is well supported by theoretical analysis and proofs.

(3) Extensive experiments verify the effectiveness of the proposed mixReg from three aspects.

### Weakness

(1) Details of selecting mixup samples according to Eq. (6) are vague. There might be some thresholds to determine the intra-cluster samples based on Eq. (6). Meanwhile, the way to determine the bandwidth hyper-parameter is not clear. It is better to provide an empirical range of bandwidths for easy implementation in various scenarios.

(2) Although the author performs extensive experiments on many datasets, more mixup methods with various mixing policies should be compared. For example, CutMix [1] can be performed on 2D images and 1D time series (replacing the segment). See questions for more details.
questions: Why mixReg work? The proposed mixReg only takes the intra-cluster (semantically similar) samples to perform vanilla mixup. Although the theoretical analysis proves the superior of mixReg upon vanilla mixup (and its variants) and mixup with input-similarity, the compared methods usually yield worse performances than ERM. The cause of bad performances of vanilla mixup might be the unreliable inter-cluster samples, and mixReg improves the mixup performance by removing these samples (similar to AdaMixup [2]). However, mixReg might degenerate to ERM with the large sample size N and the small bandwidth. What if we have a better inter-class mixup policy like PuzzleMix [3] and AutoMix [4]? We might transform regression tasks into classification tasks by splitting the regression targets into C bins and training the classifier to generate more reliable inter-class mixup samples [3, 4].

[1] CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In ICCV, 2019.

[2] MixUp as Locally Linear Out-Of-Manifold Regularization. In AAAI, 2019.

[3] Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. In ICML, 2020.

[4] AutoMix: Unveiling the Power of Mixup for Stronger Classifiers. In ECCV, 2022.


================ Post-rebuttal ================

Since the author has addressed my concerns and updated the revision according to my comments during the rebuttal period, I change my score from 5 to 6.
limitations: The author discusses the limitations of this paper in the appendix, which might be solved by further experiments. I do not find any negative social impact of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors propose to use mixup to improve generalization in regression tasks. They argue that when mixup is directly using in regression, it may generate arbitrary labels which are incorrect as the linear assumption may not hold. The authors aim to overcome this drawback by adjusting the sampling probability based on the similarity of the labels. They further show that this label similarity obtains smaller mean square error and also improves out-of-distribution robustness. Interpolating examples with similar labels mitigates the effect of domain-specific information and pushes invariant representations. The authors evaluate their work on several benchmarks and show its effectiveness on in-distribution generalization, task generalization and out-of-distribution robustness.
strengths_and_weaknesses: #### **Strengths**
The following are some strengths of the work
- The authors argument that when mixup is directly applied to regression task generates incorrect labels due to the failure of the linear assumption rule is quite interesting. Its also quite interesting that mixup in regression tasks has been understudied and the authors aim to bridge this gap.
- The experimental analysis is quite exhaustive and the authors show the performance of MixReg on many different benchmarks and in-distribution generalization, task generalization and out-of-distribution robustness.
- The paper is also quite clear to read and is well presented.
questions: #### **Weakness**/ **Questions**

I appreciate the authors efforts in developing an interesting idea. However, I have the following concerns/ queries which I would like the authors to answer:
- I quite did not understand the difference between sampling probability and the interpolation factor $\lambda$ typically used in mixup papers.
- In L133, the authors say that its meaningful to use $d(i,j) = ||y_i - y_j||^2$. However, what is the dimension of $y$? is it a one-hot vector. This is quite unclear again. Also instead of using the $L_2$ distance, how about using cosine similarity in lower-dimensional embeddings instead of $L_2$ distance in the label space?
- In the original mixup paper [62], the authors show that interpolating examples from semantically similar manifolds leads to sub-optimal performance and the best performance is achieved by randomly mixing samples from the data manifold. However, in MixReg, the authors claim that mixup samples from semantically similar data manifold results in the best performance. Could the authors please explain its significance?
- Continuing from the above point, interpolating between semantically similar points in the data manifold could lead to a problem of manifold intrusion [1] - Given two examples with different class labels, the interpolated example may actually lie in a region associated with a third class in the feature space. Im curious to know if the authors face this problem in MixReg, if not, what is being done to avoid manifold intrusion.  
[1] Guo et al., Mixup as locally linear out-of-manifold regularization. In AAAI, 2019.   

- Could the authors evaluate their work on large scale tasks such as semantic segmentation which is also a pixel-wise regression task on Pascal-VOC or MS-COCO? How does one apply mixup to such kind of tasks in general?
- The authors have missed citing several state-of-the-art mixup works such as PuzzleMix [2], Co-Mixup [3], SaliencyMix [4], AlignMixup [5], StyleMix [6], StyleCutMix [6],  AutoMix [7] etc. It would be nice to discuss these papers in the related work section.    

[2] Kim et al., Puzzle mix: Exploiting saliency and local statistics for optimal mixup. ICML 2020.   
[3] Kim et al., Co-mixup: Saliency guided joint mixup with supermodular diversity. ICLR 2021.   
[4] Uddin et al. Saliencymix: A saliency guided data augmentation strategy for better regularization. ICML 2021.   
[5] Venkataramanan et al., AlignMixup: Improving Representations By Interpolating Aligned Features. CVPR 2022.   
[6] Hong et al., Stylemix: Separating content and style for enhanced data augmentation. CVPR, 2021.    
[7]  Zhu et al., Automix: Mixup networks for sample interpolation via cooperative barycenter learning. ECCV, 2020

limitations: Yes
review_title: null

Meta-review: metareview: This is an interesting and technically solid paper. The reviews are very consistent as well. 
confidence: Certain
Decision: Accept