Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: Similar to previous studies, this paper studies the feature representation of a DNN by empirically investigating the distribution of interactions of input images. On the one hand, the authors investigate interactions of three types of mis-classified input images (including clean, adversarial, and corrupted images). On the other hand, the authors investigate interactions on two types of DNNs, including CNNs and version Transformers. Finally, the authors obtain some conclusions by analyzing tendencies in encoding interactions in these settings.  
strength_and_weaknesses: [Weakness]
 
1. This paper has very limited novelty. In fact, the main content of this paper is very similar to the line of previous studies (Cheng et al., 2021; Deng et al., 2022; Ren et al., 2021; Wang et al., 2021; Zhang et al., 2021), which proposed to use the distribution of interactions encoded in input images to study the feature/conceptual representation of a DNN. This paper directly adopts theories, algorithms, and experimental designs in previous studies, and the core difference is that this paper uses different types of input images and different types of DNNs in experiments. Specifically, it can be found that in Sections 3 and 4, the main contents of introducing (multi-order) interactions are similar to previous work(Cheng et al., 2021; Deng et al., 2022; Ren et al., 2021; Wang et al., 2021; Zhang et al., 2021). 

2. Some conclusions also overlap with those of previous studies. For example, Figure 2 shows that adversarial attacks make DNNs use more negative high-order interactions, which has been found in previous work(Ren et al., 2021). Besides, this is the only significant conclusion in Figure 2, while the difference in the first and last columns of Figure 2 is marginal.

3. Experiments in the paper are not sufficient. In the main paper, the authors only conduct experiments on two models on a dataset, and only one setting is used for adversarial attack and corruption, respectively. Therefore, it is questionable whether the obtained conclusions also appear in other DNNs/datasets/settings. I suggest the authors conduct more experiments on more architectures with more settings to demonstrate the validity of their conclusions. 

4. The comparison between ResNet-18 and Swin-T in Figure 3 is not convincing. First, the comparison may be unfair. The magnitudes of outputs of the two models may be different, so the value range of $\Delta f(i,j,S)$ may be different in the two models. Therefore, interactions in the two models cannot be directly compared. Second, the 0-order interaction in Figure 3 is significantly affected by baseline values in the computation of interactions, and the authors do not discuss this problem. Third, the difference in the interaction median between the two models is not significant, and it is hard to judge whether the difference is caused by other factors like initialization, dataset, or the training method. The phenomenon on only one pair of models cannot support the conclusion. 

5. The authors do not provide any theoretical explanations for conclusions in the paper.

6. The experiment and conclusion about adversarial transferability in Section 6 are not convincing. When comparing the transferability of inputs with different interactions of a specific order, the interactions of other orders are not controlled to be the same. Therefore, the high transferability of inputs with high 0.8n-order interactions may be due to interactions of other orders. This experiment cannot provide a fair comparison between different orders.

In general, in my opinion, this paper neither presents theoretical (algorithmic) contributions, nor introduces new experimental designs so as to provide new insights. More crucially, the conclusions also have large overlap with conclusions of previous studies.
clarity,_quality,_novelty_and_reproducibility: Clarity: good;
Reproducibility: good; 
Quality: poor; 
Novelty: limited;  
summary_of_the_review: This paper has very limited novelty. In fact, the main content of this paper is very similar to previous studies (Cheng et al., 2021; Deng et al., 2022; Ren et al., 2021; Wang et al., 2021; Zhang et al., 2021), which proposed to use the distribution of interactions encoded in input images to study the feature/conceptual representation of a DNN. This paper directly adopts theories, algorithms, and experimental designs in previous studies, and the core difference is that this paper uses different types of input images and different types of DNNs in experiments. Specifically, it can be found that in Sections 3 and 4, the main contents of introducing (multi-order) interactions are similar to previous work (Cheng et al., 2021; Deng et al., 2022; Ren et al., 2021; Wang et al., 2021; Zhang et al., 2021). 
Moreover, some conclusions also overlap with those of previous studies. For example, Figure 2 shows that adversarial attacks make DNNs use more negative high-order interactions, which has been found in previous work (Ren et al., 2021). Moreover, this is the only significant conclusion in Figure 2, while the difference in the first and last columns of Figure 2 is marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work empirically investigates various types of image misclassification from a game-theoretic view, which includes clean images, adversarially perturbed images and corrupted images. It characterizes the misclassification with the distribution, order, and sign of the interactions, and numerical results show three types of misclassifications have different tendencies in interactions. And it also explores interactions of misclassifications in different model architectures, CNNs and Vision Transformers, which implies Vision Transformers may exploit the features that CNNs do not use for the prediction.
strength_and_weaknesses: Originality:

1. Focusing misclassifications by using interactions among pixels is interesting, which manifests three types of misclassifications have a distinct tendency in interactions and each of them arises from different causes. 

2. It also analyzes Vision Transformers by using interactions to show their feature extraction that is different from CNNs.

Question:

Adversarial training is a well-known standard method to improve models' robustness. However, misclassification seems to be inevitable even if model is trained by adversarial training. So, dose there exist a significant difference between the interactions of misclassifications original models and those trained by adversarial training?
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and understanding misclassification via interactions among pixels is interesting.
summary_of_the_review: This work studies misclassification of various types of images from a game-theoretic perspective. It provides a novel understanding of misclassification for different types of images and model architectures.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates various types of misclassifications from a game-theoretic perspective. The authors study the misclassification of clean, adversarial, and corrupted images, and found that the dominant order of interactions is different between the three types of misclassification. The authors also found that the interactions of ViT are also different from CNNs.
strength_and_weaknesses: Strength:
This work conducted extensive experiments to support the idea. The Shapley value interaction seems to be an interesting idea.
Weakness:
1. The difference between distributions of interactions for different types of images is quite small in Figure 2. Especially between clean and corrupted images and low order of interactions. I was wondering if the authors can provide some explanations for this.
2. More images like the visualization in Figure 1 may need to be provided in the appendix to make the intuition more convicing.
clarity,_quality,_novelty_and_reproducibility: In general, I think this paper is well-written and easy to follow. 
summary_of_the_review: I think the general idea of this paper is interesting. The authors provided some interesting perspectives for adversarial examples through Shapley value interactions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
The paper studies various types of misclassification of images by deep neural networks. The study is done using a quantity called  "interaction" between pixels. This quantity is motivated from game theoretic concept of quantifying interaction between players in a co-operative game. This quantity has been used in other studies on deep learning. ( Cheng et al., 2021; Deng et al., 2022; Ren et al., 2021 and others).  The paper characterizes the misclassification of clean, adversarial, and corrupted images with the distribution, order, and sign of the interactions. They find that each type of misclassification has different tendencies in interactions, which indicates that each type of misclassification is triggered by different causes. They also provide an analysis of Vision Transformers by using interactions and report that the difference in distributions of interactions between misclassified and correctly classified images are clearer and also different from the case with CNNs. Further they also found that the images that are more adversarially transferable have the opposite tendency in the interactions between Vision Transformers and CNNs.

strength_and_weaknesses: Studying when and why deep models misclassify is an important problem. The paper gives some insights on this problem through the lens of interaction between pixels. They study misclassification of clean images and the ones with adversarial and non-adversarial corruptions. For the clean images they observe that the successfully classified images have local cooperation between pixels that increase the confidence score, while the misclassified images do not have such cooperation, leading to the deterioration in the confidence score. For the adversarially corrupted images they observed that misclassification by adversarial perturbations is caused by the destruction of the model from the meaningful cooperation between pixels to the spurious one, which is useless or harmful to make a prediction, while these harmful effects are milder for non-adversarially corrupted images. Further, they conducted same study with Vision transformers and found that Vision Transformer more clearly exhibit the prediction characteristics than CNNs. 

Weaknesses/Questions,
Could you provide more intuitive understanding of interaction ( something like Figure 1). Why are the left and middle figures roughly the same in Figure 1 but have opposite signs of interaction?
 There are a ton of methods to generate adversarial images, will the paper's observations remain applicable to these diverse set of adversarial methods? Soon one can design a method to generate adversarial images that show no difference in terms of interaction as well. Or can this be shown that it is not possible? 
Since the conclusions are drawn from experiments alone, it would be more helpful to see if the similar observations hold for various datasets and models as well? 
clarity,_quality,_novelty_and_reproducibility: Mostly clearly written, seems reproducible and the work is novel.
summary_of_the_review: The paper studies and important problem of understanding misclassification by neural nets and studying it through the lens of interaction is novel. Their empirical observations give insights into various types of misclassifications. I have a few questions/concerns listed above that could improve my assessment of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper diagnosis the behavior of CNN and Transformer models on three classes of images-- clean, adversarial, and real-world corruptions-- for the ImageNET dataset. For this purpose, they use `interactions'-- shapely values that determine how much a pair of pixels contribute to the class prediction, given a set of pixels in the image (that determine the order of interaction). Additionally, the authors consider the sign of interaction which helps them determining if the pixels together help increase/decrease the confidence of prediction.
strength_and_weaknesses: ### Things I liked

The paper considers an interesting angle to investigate misclassification of a classifier and is able to highlight interesting pattens from the study (eg. corrupted and adversarial images impact different kind of interactions and order).

### Things that need clarification / improvement

- The authors highlight that Cheng et. al. 2021 argue that lower-order reflects local properties while higher order reflects global properties (Sec 4.2). Unfortunately, I am not entirely convinced. I would rather only consider the 1.0n case which measures the interaction of a pair of pixels in the context of the entire input image.

- The paper is incomplete in many ways.
  - While the authors do an analysis of the multiple input image classes on which a classifier misclassifies, I am not sure how one can use it to improve a classifier robustness. For example, does the interesting observation that corrupt and adversarial images impact different types/orders of interaction have any conclusion like existing defense mechanisms cannot target both ends and fixing on will not help fix the other?
  - How does data augmentation or other defense strategies affect these interactions?
  - Why is interaction-based diagnosis better/more useful compared to other debugging approaches (eg. GradCAM, analyzing the impact of individual pixels on loss that is used to craft adversarial examples).

- The authors simply use a shapely value type measure to characterize the contribution of pixel pairs to prediction via masking. This is not strong enough a motivation to say they have done a game-theoretic understanding of misclassification, as the title suggests.

- There are several clarity issues with the paper.
  - In Figure 1 (a), one can form many pairs of black pixels. Do all combination of these pairs have $\delta f(i,j,S)$ as strongly positive?

- In Figure 2, why is the curve for correctly classified clean examples (in green) different in the middle figure for ResNet-18 (i.e. top row) compared to the left and right one? I was expecting the clean curves to be similar in all the 3 graphs for a row (which also seems to be the case for Swin-T).

- In Figure 3 and Figure 4, should the distribution of clean images not be the same in the frequency charts. For example, I am a bit confused that Swin-T has only green bars in the positive side for the clean images in Fig 3 (left) but has a normal looking distribution for clean images in Fig 4 (b). What is the n for the distribution in Fig 4, I see it is an expected value over I(i,j) but I wasn't sure where the difference is coming form?

Other corrections:

- [Sec 5.3] "The strength of the interactions for adversarial images is lower" -> "The strength of the interactions for corrupted images is lower"

clarity,_quality,_novelty_and_reproducibility: The paper has several clarity issues and although the investigation of misclassification with interactions is novel, comparison to other diagnostic approaches is necessary.
summary_of_the_review: The paper, although interesting in how it tries to diagnose misclassification in CNN and transformer models for image domain via interactions, is incomplete in several regards. It does not compare it self to any other diagnostic took, it does not highlight the usefulness of the diagnostic tool.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper investigates the misclassification of images by deep neural networks using a game-theoretic concept of interactions between pixels. The paper claims to provide insights into the causes and characteristics of different types of misclassification, such as clean, adversarial, and corrupted images, and to compare the interactions of CNNs and Vision Transformers. The paper also explores the relationship between interactions and adversarial transferability.

The reviewers appreciate the interesting angle of studying misclassification through interactions and the extensive experiments conducted by the authors. However, the reviewers also raise several major concerns that prevent the paper from being accepted.

First, the paper lacks novelty and originality, as it largely follows the line of previous work that used interactions to study the feature representation of deep neural networks. The paper does not introduce any new theoretical or algorithmic contributions, nor does it provide any new experimental designs. The paper also does not compare or contrast its findings with those of previous work, and some of the conclusions overlap with existing ones.

Second, the paper lacks clarity and rigor in presenting and analyzing the results. The paper does not provide enough intuitive explanations or visualizations of the interactions and their meanings. The paper also does not address some potential confounding factors or sources of variation in the experiments, such as the magnitude of outputs, the baseline values, the initialization, the dataset, and the training method. The paper also does not provide any error bars or statistical tests to show the significance of the differences in the distributions of interactions. The paper also does not explain some inconsistencies or anomalies in the figures, such as the different curves for clean images in Figure 2 and Figure 3.

Third, the paper lacks practical relevance and usefulness of the proposed diagnosis method. The paper does not show how the interactions can be used to improve the robustness or performance of the models, or to design better defense mechanisms or attack strategies. The paper also does not compare its method with other existing diagnostic tools, such as gradient-based methods or saliency maps, and does not justify why interactions are a better or more informative measure. The paper also does not generalize its results to other datasets, models, or settings, and does not discuss the limitations or challenges of applying the method in practice.

In summary, the paper has some interesting aspects, but suffers from several major flaws that undermine its quality, novelty, and significance. Therefore, the paper is rejected.
justification_for_why_not_higher_score: The paper can not be accepted because it does not meet the standards of the conference in terms of originality, clarity, rigor, and relevance. The paper does not make any significant or novel contributions to the field, does not provide clear and convincing evidence for its claims, does not demonstrate the practical value or usefulness of its method, and does not compare or contrast its method with other existing methods. The paper also does not address the feedback or concerns of the reviewers, and does not improve the quality or presentation of the paper. The paper does not have enough merits to warrant acceptance.
Decision: Reject