Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new batch mixup method, co-mixup, to improve the networks’ generalization performance and robustness. It formulates the construction of a batch of mixup data by maximizing the data saliency measure of each individual mixup data and the supermodular diversity among the constructed mixup data. An iterative submodular minimization algorithm is used to solve the proposed problem through approximation. Promising empirical performance is reported on several tasks. 

All previous mixup methods are limited to mixup example generation between a pair of input examples. This paper can be viewed as an extension of the Puzzle mixup, and it generates a mixup example over multiple input examples. The optimization problem formulated seems to be very reasonable by maximizing the saliency and diversity of the mixup data. The proposed method also demonstrates slightly better performance than other mixup methods.  Overall, this is an interesting paper.

There are however several issues that are not clear. The authors can clarify the following questions: 

1.	How are the labels of the mixup examples determined?  Will each generated example become a multi-label example? How to perform training with the generated data?

2.	As the proposed method requires additional process to produce information such as the saliency information and compatibility information (the matrix A), won’t this induce additional computational cost?
3.	As stated in the paper, the A_c matrix measures the distances between locations of salient objects in the input examples. Does this require object localization? Or it simply computes the distances between the feature locations? 
4.	How much does the \omega value (to compute A) affect the performance of the proposed approach?

5.	In Section 4.2, two criteria are described and then an approximation is proposed based on the two criteria.  Although there is nothing wrong about the criteria, it is unclear how good can an approximation be by simply satisfying these two criteria? That is, what is the quality of the approximation to the original problem?  Can the approximation guarantee an optimal solution to the original problem?

6.	In the experiments, although the proposed approach demonstrates promising results, the differences between the proposed method and the comparison method, Puzzle Mix, are very small. The paper claims Co-Mixup significantly outperforms all other baselines. How significant are the differences in Table 1?

7.	The paper limits the comparison to mixup methods. How is the performance level of the co-mixup by comparing with other regularization based methods? For example, VAT regularizaiton based methods.

review_title: interesting paper but requires some clarification

final_score: rating: 7: Good paper, accept
review_content: 1) Summary
- Mixup is one of the representative data augmentation techniques to improve the generalization of the network.
- The authors proposed the co-mixup technique which is novel.
- Especially, the technique can mix several images with z, and the z is found by optimizing their objective function.
- It is novel, and the experimental results are convincing. 

2) Strong points
- co-mixup approach was formulated well
- clearly outperforms the other mixup-like technique such as CutMix and Puzzle Mix

3) Weak points
- Training is slower than others, even if computing z is fast.
- Can we mix more than three images? How about 1000 images?


This paper is a good one, and I look forward to acceptance.
review_title: Review for Co-MIXUP

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new mixup method that encourages diversity among the samples mixed from a minibatch of data in addition to saliency of each mixed sample. The authors formulate two objectives: 1. a BP set function (submodular + supermodular), and 2. a submodular relaxation obtained by modularizing the supermodular component. Then they solve this problem approximately with coordinate descent by modularizing wrt the update coordinate at each step. This approach outperforms mixup baselines on image classification and several other tasks (calibration, object localization, and robustness).


The paper presents interesting ideas with impressive accuracy results. My biggest concerns with this work are clarity, thoroughness of experiments, and whether it is too computationally expensive to be used in practice.


Significance:
- The proposed algorithm's running time may be prohibitive for some applications. In the appendix the authors mention partitioning each minibatch and running the algorithm on each partition to make running time feasible, which suggests that accuracy improves at the expense of running time. Section 4.2 presents the algorithm as having linear running time, the exponential dependence on the number of labels $|\mathcal{L}|$ should be mentioned here

Experiments:
- The results sections compare against good baselines across several tasks, but this would be stronger if it compared to non-mixup baselines. 
- A more thorough ablation study would analyze each term in the proposed objective function, compare to the mixup baselines applied to m>2 inputs, and compare to the original set modularization method.
- The paper proposes an alternative heuristic to the set modularization method of Narshiman and Bilmes 2005. The claim that Algorithm 1 is faster and produces better solutions would be stronger with a thorough empirical comparison to set modularization.

Clarity:
- Saliency is described at a high level several times, but it's unclear how these values are actually computed in experiments
- Notation for $z$ is somewhat difficult to parse in the statements/proofs of Propositions 1 and 2. $f(z)$ is pairwise supermodular when considering the same column index across 2 different matrices of output coefficients $(z_{j_1,k},z_{j_2,k})$ (Proposition 1), but $f(z)$ is modular when considering the full output label matrix $z_j$ (Proposition 2). What does this mean when the optimization is over all matrices $z$? Do the ground set and constraints change in each case?
- typo: In section 4.2, "proposition 1" should be "Proposition 2"

Organization:
- Sections 5.4 (5.5) on robustness (sensitivity) as written in the main text are not informative. They should either contain more concrete statements about results or be moved entirely to the supplementary material. 
- Also, discussion of the ground set would be much clearer if it appeared before the bottom of page 6

Pros
- Improves over previous strong mixup baselines
- The proposed method outperforms baselines even when m=2 (it's worth mentioning this in the main text even if there is not enough room for a full ablation)

Cons
- Running time of the algorithm may be prohibitive in certain applications, this should be analyzed an discussed
- Several issues with clarity and organization (see above)


Questions:
- Does larger m always improve accuracy, or does this eventually decrease?
- Exactly how is saliency computed in the experiments? Is it the same pretrained map or does it vary with architecture?
- How much time does co-mixup take compared to training the network?
- Are Algorithm 1 and applying mixup ideas to m>2 samples novel contribution?


EDIT: The author response addressed all my concerns and answered all my questions, in particular that the exponential running time is a worst-case bound that is indeed loose in practice. I believe that the revised version will be much clearer and am therefore increasing my score

review_title: Interesting, but questions about saliency and running time

Meta-review: This paper proposes a type of Mixup-style data augmentation that works at the batch level rather than simply between pairs of examples. Each generated example accumulates salient images from potentially many other examples while ensuring diversity across the generated examples. This is achieved through a 4-part objective with submodular and supermodular components. The paper demonstrates the method using extensive experiments, including generalization performance on CIFAR-100, Tiny ImageNet, ImageNet and GoogleCommands. It also explores weakly supervised object localization, expected calibration error, and robustness to random replacement and Gaussian noise.

Reviewer 1 thought the approach was interesting but raised some concerns with clarity, thoroughness of experiments and whether the approach was computationally prohibitive to be used in practice. I was surprised myself that a discussion on the trade-off between computational expense and accuracy gain was not discussed in the submission. The authors responded to the review, adding a comparison to the BP algorithm (Narshiman and Bilmes 2005). The empirical result seems to back up the claim that the proposed algorithm finds a better solution and with less variance. It also appears to run much faster. The authors also responded to minor issues raised with respect to clarity and organization. In their response, the authors provided considerable detail with respect to running time and time complexity, and show that models trained with co-mixup are practical, though they do come with a significant added cost. The authors added the requested comparisons to non-mixup baselines and enhanced the ablation study. In my opinion, this is a comprehensive and satisfying response, and the paper has improved in many respects since submission.

The review from R2 was largely positive, though limited in its scope. They also expressed concerns with training time (addressed in the response to R1). Clearly the approach extends to an arbitrary (m) number of images; this was explicit in the paper/formulation and clarified by the authors. I have some concern that R2 may have skimmed the paper if they missed this point.

Reviewer 4 thought the paper was interesting and asked several clarifying questions. They expressed concern with the significance of the reported gains. Similar to R1, they asked about non-mixup baselines (VAT specifically). This was addressed in the response to R1. The authors responded to the clarifying questions and addressed the issue of significance.

Like the reviewers, I think that this is an intriguing, fresh, and elegant way to perform data augmentation. I appreciate that it has been evaluated just not from the pure generalization setting, but from other angles like robustness and calibration. There are still some outstanding concerns regarding the computational effort required to use Co-Mixup, so this would be nice to see in follow-up work.
Decision: Accept (Oral)