Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an attention mechanism that is more robust to misleading visual context. The motivation follows from the fact that current recognition models tend to mistake correlations (e.g. between background and objects) for causal factors, leading to mistaken predictions when e.g. observing the background without the foreground object.

The proposed attention mechanism comes in two variants: one that relies on two successive attention layers, and a more heavyweight version which involves a transformer-based encoder inserted between the aforementioned layer. These extra processing layers are combined with a form of sampling-based intervention, which should allow the model to ignore spurious context.

The method is evaluated on two multi-label classification tasks (MS-COCO and Pascal VOC), and compares favourably to competing methods. An ablation study shows how different components of the approach improve over the baseline.
strength_and_weaknesses: I struggled to understand the approach and in particular to connect the stated motivation with the method being proposed as well as the results. The paper starts by describing an often observed problem, namely that common recognition methods struggle when the combination of foreground and context violates expectations. Based on the presentation, it is neither clear how the proposed method would address this, nor why the (small) improvements over the baseline indicate that the problem is indeed being successfully addressed. 

The presentation of "causalities in contextual bias" in section 3 is confusing. The presented structural causal models (see Fig 2) only model the problem at a very abstract level. Its also for example not clear what is being proposed in (2a), if the directions of the edges make sense, why a misleading relationship between X & Y might be reinforced specifically by attention, and what it means to break the connection between the context and the image (2c) both conceptually and in practice. The connection between what is presented here and the method is very tenuous. How is the "multiple-sampling operation with Dot-Product Attention (DPA) re-weighting" "essentially [a] causal intervention"? What happens in the case when the context is necessary to correctly identify e.g. a small/low-resolution object? 

The paper argues that an attention map that tracks more closely with the ground truth mask of the object is a sign of the method's success. Such results are presented e.g. in Fig 1b and in the Appendix (A.4, see final paragraph in section 5). These results show modest improvements and are also based on the faulty assumption that the context is to be ignored lest it be faulty, when in fact one would rather prefer a model that can take context into account when needed.
clarity,_quality,_novelty_and_reproducibility: As described above, the paper is not at all clear when it comes to connecting motivation to method and to results. The empirical results show modest improvements over the baselines, but these may very well be the result of additional model capacity rather than some form of causal reasoning as advertised.
summary_of_the_review: The paper presents a method that to me is only tenuously connected to the stated motivation. It is hard to connect the stated need for causal reasoning to overcome misleading context and the exact operations carried out by the proposed layer. As such, I believe this paper tells a story that is not quite supported by the method and the experiments. I am open to being corrected as this assessment may be the result of a misunderstanding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The  article propose a method to learn casual image features for contextual bias via an attention mechanism called interventional dual attention (IDA). The mechanism consists of two attention layer with multiple sampling intervention that protects attention from cofounder context. The approach is evaluated on well-known datasets (COCO, VOC) and achieves SotA performance on multi-label classification. 
strength_and_weaknesses: Strengths:
The paper explained the problem  of contextual biases in attention models for solving visual recognition problems.

To address this the proposed interventional dual attention (IDA) consisting attention and multi-sampling intervention.

Evaluation of the proposed approach in well-known datasets COCO and VOC, and comparison to the state-of-the-art.

Ablation study justifying the components IDA and visualization using GRAD-CAM to show the attention map.

Weakness:
I have doubt about the novelty aspect of the approach. Although the paper claims it is novel but it is more re-packaging of attention mechanism to extract class-specific information. The causal intervention seems to be incremental. The final eqn 7 is more of standard weighted attention?

The approach is evaluated using heavy backbones (ResNet-101 or Swin-Base) with image resolution of 384, 448 or 576. Is there any justification to use heavier backbones or large resolution? Experiments should have been carried out with lighter model ResNet-50 or NAS-Net with input resolution of 224 which is widely used for various visual recognition problem.

The claim “attention does not capture the more accurate regions of targets than the baseline.” This could be for a specific image example. Generalizing this claim requires thorough examination which is not done in the evaluation process.

The spatial class-aware attention is sensitive to spatial augmentations?  The impact of spatial augmentation (scale, crop, rotation, etc.) on SCA is not discussed.

I feel there are some recent work on spatial attention by exploring semantic regions for fine-grained visual recognition such as CAP (Behera et al., AAAI 2021), SR-GNN (Bera et al., TIP 2022) exploring GNN and Wharton et al. (BMVC 2021) exploring relationships between hierarchical structures via GCN and could be cited.  

Where is the table containing FLOPS and parameters, and comparison to the baselines and state-of-the-art models? In supplementary, it says Table A.2 which is pointing to the Table 8 that says M
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. It has given a background on what is the problem in most of the attention-driven approaches for visual recognition and justification on how the proposed approach advances in this direction.

There is some element of novelty but it is unclear to me. I have raised this in the weakness section. It would be nice if the authors clarify this in rebuttal and I will reconsider my decision.

The quality is good. Experimental evaluations are carried out to justify the proposed approach. In the weakness, I have mentioned about the use of heavier backbones. Why? why not light-weight model with standard resolution of 224.

The paper explains well about the implementation. The source code is attached as supplementary and the authors have promised to release the source code. Thus, there is little doubt on reproducibility.
summary_of_the_review: The motivation of proposing interventional dual attention for visual recognition is very good but the novelty aspect on the proposed mechanism is not fully explored. The proposed approach is thoroughly evaluated and compared to the state-of-the-art (SotA). It lacks the description on the computational complexity and critical discussion on performance of the model in comparison to the SotA. Please refer to the sections 2/3 for the details. This is a good work I would like to see authors rebuttal to my posed questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to reduce contextual bias in multi-label classification. To this end, the paper aims to remove the effect of unseen context $C$ from the prediction. Specifically, given a casual structure {$C \to X, C \to Y, X \to Y$}, the usual classifier predicts $P(Y|X)$ which implicitly reflects the unseen $C$. Instead, the paper aims to predict $P(Y|do(X))$ to discard the contextual bias. This probability can be computed by backdoor adjustment (Eq. (1)) and inverse probability weighting (Eq. (4)). This term can be estimated by setting the context $c$ as a pair or class and image $(k,x)$, leading to the final estimator (Eq. (7)). The proposed method shows good performance on various multi-label classification benchmarks.
strength_and_weaknesses: ### Strength

- The motivation in Figure 2 is interesting. Addressing contextual bias via modifying the prediction from $P(Y|X)$ to $P(Y|do(X))$ is clear.
- The proposed method outperforms the baselines for multi-label classification.

### Weakness

**Concerns in presentation**

My major concern is the overclaiming presentation. The proposed method is just an improved Transformer block for multi-label classification. However, I did not catch this point until carefully looking at the method. The current title, abstract, and introduction look like the proposed method can improve the robustness of ViT on the contextual bias, e.g., applied for general classification [1] or even object detection. The paper should clarify its scope.

The method is also overly complex and missing some important details. If I understand correctly, isn't spatial class-aware attention (SCA) just an average pooling of spatial class-wise features (which are widely used [2])? So the baseline SCA is just pooling the class-wise feature and applying a binary classifier for each class vector. Then, the proposed method is just an interaction layer of this class vector: convert a R^{KxD} vector to a refined R^{KxD} vector. Figure 3 and the explanation in Section 4 could be significantly improved. Currently, the overview of the method says the structure to be (1) SCA, (2) MS-SCA, (3) DPA. However, it does not match the title of the subsection, which confuses the readers. Also, neither subsection 4.2 nor 4.3 explains what the MS-SCA is exactly -- one can merely guess that Eq. (7) is MS-SCA.

In addition, the paper has lots of typos:
- bare -> base in Figure 1 caption?
- Sec.7 -> Eq. 7 in page 4?
- number of c Eq. 4 -> number of c in Eq. 4?

**Concerns in scope and evaluation**

Another main concern is the scope and evaluation. The proposed causal framework is general and may be applied beyond multi-label classification. Extending the method for general classification or even object detection would highly strengthen the paper. Also, I encourage the authors to include common benchmarks for measuring contextual bias such as MetaShift [1].

[1] Liang & Zou. MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts. ICLR 2022.\
[2] Yun et al. Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels. CVPR 2021.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper needs a significant polishing\
Quality: The core idea is good, but current evaluation is limited\
Novelty: I think the proposed framework is novel\
Reproducibility: Code is submitted
summary_of_the_review: I think the overall idea is interesting. However, I'm lean toward borderline reject for the current version of the paper. I hope the revised paper includes:
- Polish presentation -- clarify the scope and clearly explain the proposed components (complex $\neq$ novel)
- Extend the scope more than multi-label classification and add empirical validations for those experiments
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on addressing the over-reliance on contextual biases, and proposes a new method based on the attention mechanism. It draws inspiration from the causal intervention, and leverages attention together with feature sampling/shuffling to realize the function. Experimental results show that the proposed method is able to outperform the corresponding baselines without considering causal relationships for multi-label classification.  
strength_and_weaknesses: As far as I am concerned, the paper has the following strengths:

(1) Developing visual systems that make decisions by faithfully gathering key information instead of relying on spurious correlations is an important topic. 

(2) It provides an extensive ablation study on different components of the method, which facilitates understanding of their contributions.

(3) The proposed method is applicable to a wide range of deep networks, and consistently improves their performance.

However, there are also notable weaknesses:

(1) It is not new to leverage the attention mechanism for addressing issues related to the causal intervention (Yang 2021b and Wang 2021). The paper does not provide a comparison with these closely related studies, which makes it difficult to understand its contribution. I am aware that the paper focuses on a different task (multi-label classification vs standard image classification). However, without a fair comparison, the experiments still fall short of demonstrating the advantages of the method. I would suggest that the authors consider following experimental paradigms in related studies. Note that Wang 2021 not only experiments with NICO, but also other datasets such as ImageNet-A (which makes the statement in Section 2 inaccurate).

(2) Several studies [ref1, ref2, ref3] (see references below) point out that the attention mechanism may not be a good indicator of regions of interest for decision-making. As this paper emphasizes leveraging attention to address biases, I believe it would be interesting to investigate how attention truly benefits the reduction of biases. For instance, what will be the correlation between the permutation of attention and the change of prediction distributions (could be measured by JSD, see details in [ref1])?

(3) Table 1 shows that the proposed method only achieves limited improvements over the compared methods, despite that all of them do not take into account causal relationships. In addition, these results are for standard evaluation and do not offer too many insights in terms of contextual biases. It would be good to move the results in the appendix (e.g., Table 9) to the main paper, which are more relevant to the overall goal. 

(4) Looking at the results in Table 9, it appears that the IoU scores are relatively low (i.e., around 0.2), does it imply that the model still largely relies on contextual biases? 

(5) In terms of the qualitative results shown in Figure 6, attention maps for most of the heads seem to attend to context while only a few (with higher weights) have a stronger focus on the correct objects. Any idea why? Is the observations somewhat connected to the redundancy of multi-head attention [ref4] or their diverse behaviors [ref5, ref6]?

(6) (minor) Is there a reference for spatial class-aware attention? 

(7) (minor) It appears that the paper slightly exceeds the page limit for ICLR. 

References:

[ref1] Attention is not Explanation. NAACL, 2019.

[ref2] Attention is not not Explanation. EMNLP, 2019.

[ref3] Is Attention Interpretable? ACL, 2019.

[ref4] Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL, 2019.

[ref5] Revealing the Dark Secrets of BERT. EMNLP, 2019.

[ref6] What Does BERT Learn about the Structure of Language? ACL, 2019.

clarity,_quality,_novelty_and_reproducibility: (1) Overall, this paper is well written. However, the experiment section could be improved by bringing more relevant analysis to the main paper, and moving the ablation to the appendix.

(2) Due to the lack of comparison with related methods, the paper fails to fully convince me about its novelty and technical contribution.

summary_of_the_review: While this paper proposes an interesting method for tackling the issues of contextual biases, its experiments are somewhat insufficient to demonstrate the advantages of the method. There are also some intriguing results that are closely related to the overall objectives of the reduction of biases, but are not carefully studied in the main paper. I am leaning towards rejecting the paper given its current presentation, but I would consider updating the scores if more in-depth analyses/experiments are provided.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a novel attention mechanism with causal justifications that remove the spurious contextual correlation for robust visual recognition. Experimental results on MS-COCO and PASCAL-VOC demonstrate that the proposed attention outperforms baselines.

Strength:
1. A well-written presentation of motivation.
2. The proposed attention is in line with the causal justification, which makes it well-motivated.
3. Comprehensive ablations.

Weakness:
1. The robustness of the attention needs more quantitative analysis
2. As raised by Reviewer UmuF, context is indeed a part of the reasoning process of visual recognition. Authors should not claim that context is harmful. In fact, the authors should argue that the "removal of context" per se is a good result of exploiting context.
3. Some implementation details are missing.

justification_for_why_not_higher_score: 1. The implementation of the causality is somewhat unclear.
2. Only multilabel classification is demonstrated. The visual attention should be justified in wider range such as semantic segmentation.
justification_for_why_not_lower_score: Overall, realizing the causality in visual recognition is an interesting idea. The proposed method would attract broad interest in the CV community.
Decision: Accept: poster