Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies empirically the effect of artefacts and biases in model analysis and training methods relying on data manipulation. They argue that common metrics used for analysing shape bias and occlusion robustness also (undesirably) include the effect of artefacts (such as edge sharpness) introduced by the corresponding image manipulations, and propose a new metric, iOcclusion, which is more robust to these effects.  Further, it is argued that these artefacts also manifest themselves in training schemes based on mixed sample data augmentation (MSDA), and can be beneficial by introducing a systematic bias towards complexity.
main_review: Overall, this paper makes a number of interesting and novel insights regarding current practices in data modification for image classification. These are well supported for the most part by (clear and well-designed) experimental evaluations, though the lack of formal definitions and analysis means that some claims remain somewhat speculative. There are some issues with the clarity of the narrative, though I believe this is non-critical and can be fixed.

The paper firstly introduces the DI metric and show that models can have similar shape biases (according to GST dataset), but perform differently under DI w/ shuffling. It is then argued that this shows measures based on patch-shuffling are unreliable. However, since DI was introduced by the authors, I think a better representative for “effect of patch-shuffling” would be accuracy after shuffling, and how this compares to GST. At least, it could be discussed how the class-focused DI compares to accuracy. **AR**: I am satisfied with the author response and it is good to see that there is still a discrepancy between accuracy after shuffling and GST.

The authors then make a similar argument for occlusion robustness. However, it is not clear to me what "occlusion robustness" actually means, since there is no formal definition, nor clear empirical gold standard like GST dataset for shape bias. In particular, since the model is treated as a black-box for all occlusion robustness metrics (including iOcclusion), how can one distinguish whether the model’s representation is “not occlusion robust” or “using edge bias/artefacts”, if this leads to the same input-output behaviour, i.e. misclassification? **AR**: Thank you for the explanation. If I understand correctly, the distinction is between whether the performance only drops for some types of occlusion such as a box (indicating sensitivity to artefacts), or all types of occlusion (indicating lack of robustness/inability to classify without the occluded information). Firstly, I would suggest the authors define this formally as the target/proxy to the intuitive notion of "how much information can be hidden from the network". Secondly, I accept the authors' point that iOcclusion has been shown to be less sensitive to extraneous factors such as edge artefacts. However, invariance is only a necessary, and not sufficient condition to show that it captures occlusion robustness. For instance, a function which randomly assigns a value to each network would also be invariant. While Figure 3 does indeed show that FMix is given a greater iOcclusion robustness than the basic network, more analysis is needed to definitively prove this link/correlation between occlusion robustness and iOcclusion. For instance, one could assess, for a fixed occlusion type, how correlated *accuracy drop* is with *iOcclusion*, with respect to different networks. 

The definition of iOcclusion using the train-test generalization gap to improve consistency (w.r.t. colour/shape pattern of occlusion patch) of the metric is a very interesting and, to the best of my knowledge, novel idea. I find the argument regarding consistency with weaker models (Lines 207-216, Figure 4) comparatively weaker as I expect CutOcclusion to also have this property if it was instead defined as the drop in accuracy, as is commonly done.

Finally, the authors argue against existing model-based measures of data augmentation quality such as diversity and affinity, and suggest to instead focus on introduced biases. This is well supported by the inter-dataset experiments.

In terms of clarity, I found the paper very readable and well-written on the small scales, but I felt the narrative of the paper is sometimes muddled and unclear, and the "big picture" can get lost in the minutiae. For instance, a clear statement of the desiderata of iOcclusion at the start of Section 3 would be helpful. Also, I found the Introduction to be quite difficult to follow without having read the rest of the paper beforehand. For instance, I found the statement in Lines 43-44 “In summary, it is currently assumed that the artefacts introduced by changes in the data are negligible when evaluating models, while those introduced when training are important and undesirable.” to be very difficult to interpret without having read the whole paper to understand what is meant by “negligible” and “important and undesirable”. I think the paper would be improved if the Introduction focused more on examples and the intuitive argument. **AR**: The authors have agreed to address clarity issues (also raised by other reviewers); I would once again strongly encourage the authors to carefully reassess each section to make sure a) the idea is clear without reading the whole section and b) all quantities and concepts  are fully defined and the logical argument is made as clear as possible. Clarity is always important but especially paramount for papers such as this which follow an unconventional structure and whose main contribution is a critical analysis of previous work.

**AR**: In light of the authors' response, I have increased my score and vote to accept the paper. As the other reviewers have pointed out and the authors have noted, the study mostly focuses on negative results showing counterexamples to prior work rather than new solutions. However, in my opinion this is not a key limitation as such results are also important, and the paper makes a number of interesting observations, such as the bias of artefacts in robustness evaluation metric, backed up with empirical evidence. If the logical arguments are laid out more clearly, along with the adjustments that the authors have agreed to make, I believe that the paper consititutes a good contribution to the area which will be of utility to other researchers. 

Other questions:

• In Figure 3, for fraction removed 0.1 and the FMix model, iOcclusion is much greater than one, meaning the generalization gap is larger for occluded images than for the original images. Do the authors have an explanation for why this is? 

• In the definition of DI, is increase in misclassification for class $c$ calculated for images with true label $c$, or predicted as $c$ by the classifier?


limitations_and_societal_impact: Yes, limitations and societal impact are adequately discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper delves into the assumption that data modification is detrimental to training, while negligible when analysing models. It aims to encourage better practice when dealing with data distortions rather than elimination. With experiments and analyses, the authors show that current shape biased identification methods and occlusion robustness measures are biased. The authors then propose a fairer alternative to measure occlusion robustness. A series of experiments are conducted to disprove common assumptions and put forward the argument that not preserving the data distribution can lead to learning better representations. 
main_review: 1. The authors question the common assumption that data modality is negligible when evaluating models and important when training, and construct counter-examples against the common belief. The idea is novel and challenges the common assumption. 

2. The paper does not follow the common structure of ‘Introduction - Related Work - Methodology - Experiments - Conclusion’. It is organized by posing and answering questions, which would strongly disprove common beliefs and put forward the authors’ arguments. 

3. The clarity of this paper is good in general. However, the paper lacks an easy-to-read summary for each experiment. Maybe a brief and clear answer to the question should be stated ahead of the experiment and analysis part. 

4. The paper aims to encourage better practice when dealing with data distortions rather than elimination. It proposes a fairer alternative named ‘interplay occlusion’ to measure occlusion robustness. It points out the limitation of previous studies and argues that correctly interpreting the effect of data distortion is of critical significance not only to make the models more trustable but also to inject more informed prior knowledge in future applications. 

limitations_and_societal_impact: 1. As the authors mentioned in the paper, there still remains a question: 	How can we better capture the bias that is introduced and its quality. 

2. It is meaningful that this work corrects and strengthen the community’s perception of how distorting data affects learning. It argues that the impacts should be understood and exploited, however, the future applications of this work are not fully investigated.

3. Some conclusion is drawn with observation rather than strict proof, which is not convincing enough, e.g. shape-biased model and texture-biased model. 

4. The paper lacks an easy-to-read summary for each experiment. It is organized by posing and answering questions creatively, but it lacks a brief summary of each answer. When reading the paper, I could only understand the answers to the questions after I have finished read the whole texts. Maybe a brief and clear answer to the question should be stated ahead of the experiment and analysis part. 

5. There are some minor typos in the paper, e.g. ‘asses’ in line 156 should be ‘assess’
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper studies impacts of data augmentation. Authors propose new measures for evaluating data interference and occlusion robustness. Through analyzing changes in accuracy induced by data distortion, authors argue that current practice about using and treating data augmentation has led to biased model analysis and interpretation methods. 
main_review: The authors deal with an important area of research that discovers hidden biases in data augmentation. However, unfortunately, this paper is hard to follow. Vague words uncommon in the community are given without context or explanation. For instance, in L 43-44, what does mean by “artefacts introduced by changes in the data are negligible when evaluating models”? Similarly, what are “the artefacts introduced when training”?

Also, key measures proposed in the paper require more explanations and motivations. For instance, for data interference (Eq. 1), why is the percentage represented by the dominant class important? Similarly, why is this percentage required to be weighted by the maximum increase? In addition, it is unclear why the authors consider only increases in the error rates when we are evaluating the impacts of “data interference.” Similar issues exist for the interplay occlusion (Eq. 2).

limitations_and_societal_impact: The authors adequately addressed limitations and societal impact in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors examine the effects of distorting data and its effects on  biased results. They emphasize on image modifications, and they propose an alternative measure of occlusion robustness (called the interplay occlusion based on the interplay between seen and unseen data.) The authors also focus on questions about mixed sample data augmentation. They mention the need for arguing about bias from distortion and its role in trusting models and injecting informed knowledge as well.
main_review: It would have been helpful to discuss more applications in real world that would suffer from such distortion bias. Also, in the end the authors ask for better practice when dealing with all forms of data, so it could help discuss the side-effects of  this kind of bias when we have different kinds of datasets compared to the image datasets the authors explore. Some of the main experiments are mentioned (and referred to a lot) in the Appendix. So although the idea is new and the goal is clear, the writing could still improve. The significance would be more pronounced if the applications/real-world ruinous after-effects are mentioned in a separate paragraph or section.

-It is mentioned that occlusion in real-life scenarios could be caused by non-uniformly colored objects, so a suitable measure should generalize to all color patterns. It would be helpful to elaborate how this was actually done.

-It is mentioned that mostly simple features (similar to the common features) are impacted. But if the information happens to interfere with a feature that is important for separating, the performance could degrade. It could be useful to add the reasons as to why the experiment is not sufficient for general conclusions about this point and how one might try examine this issue.

-Is there a way to see what proportion of the features are the ones in the simplicity bias? That is, what proportion of them are similar to the common features.

limitations_and_societal_impact: The authors mention the limitations of their proposed approach and new measure (interplay occlusion). But a separate paragraph on this topic could improve readability.
However, the societal impact still needs more details. As the real world applications and how the side effects of such bias would be  harmful need more detailed explanations.
review_title: null

Meta-review: This paper studies the interesting topic of the artefacts and biases that can be introduced by approaches that rely on data manipulation.

After the discussion period, the paper received final scores of 7663, but where the 3 was a very low confidence review with little detail and for which the reviewer did not participate in discussions or requests to add more detail to their review.  I have predominantly thus discounted this review from my deliberations, but not entirely as I do think the failure of this reviewer to understand the paper is actually partly representative of clarity issues of the work raised by other reviewers.  It is also worth noting that this was a paper where it proved extremely difficult to recruit appropriate reviewers, with very few bids received in the bidding process and numerous assigned reviewers pulling out because they did not feel they could sufficiently well understand the work to assess it; something which I think is again reflective of clarity issues as discussed below.

Overall, I see this paper as very much on the borderline, with reviewers praising some of its underlying content and insights, but repeated issues raised on clarity and reviewers sometimes unsure about the paper's contributions.  Indeed, I myself struggled substantially when I tried to read the paper and believe it currently requires substantial effort to understand.  In particular, I think the introduction (and to a lesser extent the abstract) are both very poor at outlining what the paper is trying to achieve and giving motivation and context for the work.  I think one reviewer puts it very well when they say "I found the Introduction to be quite difficult to follow without having read the rest of the paper beforehand" which is exactly the opposite of how a paper should read: the introduction is usually the most critical part of the paper and lays the foundation for the reader to understand the work.  Similarly, issues then percolate down through the paper, which, amongst other things, is regularly too vague and imprecise; often lacks a central narrative and fails to properly link the lower level ideas back into the higher level aims; and generally lacks the hand-holding and high-level glue that make a paper easy to follow.

In light of this, the keys questions become whether the strengths of the paper are sufficient for these shortfalls on clarity to be overlooked and whether the required adjustments can reasonably be expected to be made in time for the camera-ready.  Though I think this is a very tight decision (which I mulled over for some time), unfortunately, my final conclusion is that too much needs to be done for the paper to be accepted at this time (e.g. I think the introduction needs a complete rewrite), bearing in mind the highly competitive nature of this review process.  I hope though that the reviews and discussions have been helpful to the authors and that they will make the required updates and successfully resubmit to another venue.
Decision: Reject