Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an interesting idea on data augmentation via mixture of samples. Unlike typical ways of mix-based data augmentation methods, this work performs iterative mixtures, which is able to reuse the augmented mixed samples and thereby generates more diverse samples. The idea is simple yet effective, which is validated by extensive experiments.
strengths_and_weaknesses: Strengths:
1. The idea of recursive mixtures of samples is novel.
2. Extensive experiments demonstrates the effectiveness of the method.
3. The paper is written well and easy to follow.
questions: 1. Is such recursive mixture operation performed the same times of iterations of the whole optimization? Or is there an upper-bound on the recursive times? Intuitively, more recursive times leads to richer and more diverse semantics in the augmented samples, however, incorporating too many  classes of semantics may degenerate the performance. Maybe it is better to conduct experiments to investigate the effect of recursive times of mixture operations.

2. It is claimed that the introduced consistency loss can help the model to learn the spatial-correlative semantic representation, which potentially benefit the downstream tasks like object detection or semantic segmentation. Thus, it is better to conduct such ablation study to verify the effect of the consistency loss on these two tasks. 
limitations: Refer to the questions posed above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a recursive data mixing augmentation called RecursiveMix. It pastes a historical image patch onto the current training sample to promote the data diversity for image mixing. Furthermore, a consistency loss is introduced to force the local pasted patch prediction is invariant to the background. Extensive evaluation on image classification, object detection, and semantic segmentation shows that RM largely improves performance.
strengths_and_weaknesses: # Strength
1. The method is well-motivated and finely designed. The two key components are tightly coupled: recursive data mixing promotes image diversity, while local patch consistency calls for representation invariance regardless of background. 
2. The experiments are truly comprehensive. The author not only testifies the performance on a wide range of mainstream visual tasks but also provides an extensive ablation study on each component of the method.
3. The proposed method shows compelling improvements compared with existing methods.
4. The paper is clearly written and easy to follow.

# Weakness
1. The current method seems not to provide striking new insights. The current RM design is mainly a improved version of exiting data mixing pipeline, by considering the historical data information. I do not mean the derived approach is "not novel". I just want to see more fundamental insights or rigorous discussion on "why the current design is reasonable?" and "How does it improve the performance?". Good performance itself just indicates the results, but tell nothing about the reason.
2. No ablation on $\lambda$. In lines 127-131, the author claims that "$\lambda$ denotes the proportion of the historical images, which is not suggested to be quite large". However, no ablation is provided on resizing ratio $\lambda$. I strongly suggest providing the results. 
questions: the proposed method needs an additional task head, which inevitably adds on the training computation. However,  the authors only compare the cost of the classification task, with a linear head. In contrast, the task head in detection and segmentation is extremely large compared to the backbone. Could the author please compare the computation cost overhead on detection and segmentation?
limitations: The limitation is discussed in Line 313-319.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel data augmentation strategy (RecursiveMix). The proposed augmentation recursively resizes the historical input and then fills it into the current batch. Experiments demonstrate that the proposed RecursiveMix consistently outperforms the popular augmentations such as Mixup and CutMix. 
strengths_and_weaknesses: Strengths:
1. The proposed augmentation is novel and interesting. 
2. Abundant experiment results are provided. According to the experiment results, RecursiveMix can consistently improve the performance.

Weaknesses:
1. The discussion subsection 3.2 is not informative. Most information has been included in the introduction.

2. The paper also lacks an experiment in which several basic augmentations, e.g. cropping, scaling and flipping, have been applied and then the authors further add and compare RecursiveMix to advanced augmentations, e.g. mixup and cutmix. In other words, the authors should consider the joint usage of multiple augmentations.

3. It will be better to give the standard deviation of multiple experiments. 

4. The proposed method involves two hyper-parameter \alpha and \omega. The ablation study shows that the performance is sensitive to the hyper-parameters considering that the improvements over Mixup and CutMix are usually less than 1%. 

Small problems:
1. The font in figures can be larger for reading easily.
2. The interval between bars in Figure 1 can be uniform. 
3. Syntax error in Line 129 “Because …”

questions: Please address the above weaknesses.
limitations: The authors have discussed the limitations. No potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors of this paper propose to increase the model generalizability via historical mix-up. Specifically, the compositional training batch from previous time-step is stored and ultilized in the training process at current time-step.
strengths_and_weaknesses: Weaknesses:

1. Is there any experimental comparison with hierarchical mixed-up in large-size batch? Specifically, the comparison with hierarchical mixed-up within single batch, rather than using the last time-step information. Since during the training process of large datasets, the number of batch in each epoch could be large, which may cause two problems: 
a) Over-memorizing the information(images) appears in the early batches; 
b) Unexpected semantic dependencies caused by certain information co-occurance.

2) Line38. The proposed recursive-mix improves the inner-data diversity within a batch, while decrease the intra-data diversity within an epoch by information paste.

3) Although the increase of RM's model complexity compared with others is small, is there any possible that the performance gain comes from the increased model complexity? Is there any experiment to verify this assumption?

4) In Eq(5). a) Since H and H' are with different weights, which part of the model is actually optimized for distribution alignment? The encoder ot the projector H?
b) The network is updated after processed each batch, will the KL-term affacts/prevent the optimization process? Another possible is that it will result in updating momentum appears in contrative learning methods, is there any possible that the performance gain actually comes from the momentum rather than the mix-up process?
c) The information loss is ignored during the mix-up process. Even an instance can still be observed at current stage, it is hard to ensure that the information loss is aligned with the scale loss. For example, without the tiny boarders, a billboard will be same with the wall, thus its key information may disappear within few time-steps.
questions: Please refer to Sec.2.
limitations: Please refer to Sec.2.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents RecursiveMix that applies CutMix iteratively. The proposed method is efficient and effective and shows strong generalization ability for different tasks including image classification, object detection, instance and semantic segmentation.
strengths_and_weaknesses: ### Strengths 

1. The idea is simple and effective, which makes it easy to follow.

2. The performance it achieves is impressive. It can lead to consistent gain for four different tasks and all of them were evaluated on challenging benchmarks. Meanwhile, it only requires marginal computational cost during training and is completely cost-free during inference.

3. The experiments and ablation studies are comprehensive and extensive. The authors evaluate their methods on a wide range of tasks and datasets, which can help to attract more audiences with different backgrounds.


### Weaknesses

1. RecursiveMix changes the previous crop-based CutMix to resize-based CutMix and keeps the original label assigning strategy in CutMix, which seems counter-intuitive. In this way, the foreground area of the crop in the mixture will be always kept since it will resize the whole image and keep all content there. But the crop can cover the foreground area of the other image, which means the valid object can sometimes disappear. Intuitively, the label assignment under such circumstances should always keep the label of the cropped image as 1 since the foreground object in the crop always exists, and the label confidence for the other image in the mixture should be lower. 
questions: See weaknesses
limitations: See weaknesses
review_title: null

Meta-review: metareview: The manuscript has been reviewed by five reviewers with ratings of 4,5,6,7,7. The reviewers are in general happy with the contributions, novelty, experimental validation, and mostly recommended acceptance. The AC agrees with the majority vote and would like to recommend acceptance. Congratulations!


confidence: Certain
Decision: Accept