In Pursuit of Causal Label Correlations for Multi-label Image Recognition
Authors: Zhao-Min Chen
Xin Jin
YisuGe
Sixian Chan
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-label
Label correlation
Causal intervention
Primary area: deep_learning_architectures
TL;DR: Utilizing casual intervention theory to capture casual label correlations.

Abstract: Multi-label image recognition aims to predict all objects present in an input image. A common belief is that modeling the correlations between objects is beneficial for multi-label recognition. However, this belief has been recently challenged as label correlations may mislead the classifier in testing, due to the possible contextual bias in training. Accordingly, a few of recent works not only discarded label correlation modeling, but also advocated to remove contextual information for multi-label image recognition. This work explicitly explores label correlations for multi-label image recognition based on a principled causal intervention approach. With causal intervention, we pursue causal label correlations and suppress spurious label correlations, as the former tend to convey useful contextual cues while the later may mislead the classifier. Specifically, we decouple label-specific features with a Transformer decoder attached to the backbone network, and model the confounders which may give rise to spurious correlations by clustering spatial features of all training images. Based on label-specific features and confounders, we employ a cross-attention module to implement causal intervention, quantifying the causal correlations from all object categories to each predicted object category. Finally, we obtain image labels by combining the predictions from decoupled features and causal label correlations. Extensive experiments clearly validate the effectiveness of our approach for multi-label image recognition in both common and cross-dataset settings.

In Pursuit of Causal Label Correlations for Multi-label Image Recognition
Zhao-Min Chen',
Xin Jin
Yisu Ge Sixian Chan
Key Laboratory of Intelligent Informatics for Safety & Emergency of Zhejiang Province,
Wenzhou University
Samsung Electronic (China) R&D Centre, Samsung Electronic
The College of Computer Science and Technology, Zhejiang University of Technology cherzhaammnn133@mma1. com, ysgduzu. edu.
Abstract
Multi-label image recognition aims to predict all objects present in an input image. A common belief is that modeling the correlations between objects is beneficial for multi-label recognition. However, this belief has been recently challenged as label correlations may mislead the classifier in testing, due the possible contextual bias in training. Accordingly, a few recent works not only discarded label correlation modeling, but also advocated to remove contextual information for multi-label image recognition. This work explicitly explores label correlations for multi-label image recognition based on a principled causal intervention approach. With causal intervention, we pursue causal label correlations and suppress spurious label correlations, as the former tend convey useful contextual cues while the later may mislead the classifier. Specifically, we decouple label-specific features with Transformer decoder attached the backbone network, and model the confounders which may give rise to spurious correlations by clustering spatial features of all training images. Based on label-specific features and confounders, we employ roos-attention module to implement causal intervention, quantifying the causal correlations from all object categories to each predicted object category. Finally, we obtain image labels by combining the predictions from decoupled features and causal label correlations. Extensive experiments clearly validate the effectiveness of our approach for multi-label image recognition in both common and cross- cross-dataset settings.
Introduction
Multi-label image recognition a fundamental task computer vision, aiming to predict all objects present in image. It has widespread applications including object detection 16), medical imag- ing [39], and person re-identification [3I. However, this task challenging as the combinations of labels can be tremendous. Modelling label correlations to reduce the search space is believed to be essential for multi-label image recognition [4]
In the research of lubti-label image recognition, an implicit yet common assumption is: the training and test sets follow independent and identically distributions (i.i.d.). and the label correlations consistent. Under this setting, a deep backbone network can implicitly extract context- features that are beneficial for object recognition, and furthermore, explicit label correlation modeling can explore contextual cues more deeply to improve the recognition accuracy. Technically, graph structures 12 or attention mechanisms [10 (25 have been successfully employed to model label
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pasee
0.59
Pancs
6:99
Thilkg
Deg
Texting
Dog
0.43
Car
0:1
Ca
0.1
Figure 1: Illustration of the concept and effect of contextual bias in training. It is common that "Person", "Dog", and "Cat" co-occur in training images (we only show one image), while the test image may only contain "Person" and "Dog". Excessive reliance on the label co-occurrence in the training set may lead the recognition model predict the "Cat" solely based on the presence of "Person" and "Dog".
correlations. However, these methods may fall short when there exists contextual bias in the training set. As illustrated by Fig. [ "Person", "Doo", and "Cat" co-occur frequently in the training set, but test image may only contain "Person" and "Dog". Consequently, due to the learned label correlations, the multi-label recognition model predicts a high probability of "Cat", solely based on the presence of "Person" and "Dog".
Recently, a few researchers [4 17 have uncovered the contextual bias issue, and discarded explicit label correlation modeling. Furthermore, they attempted to alleviate the effects of contextual bias by decorrelating the feature representations of a category from its co-occurring context [14] or removing the contextual bias in features with causal mechanisms [17]. Despite improved accuracy. these methods neglect to model label correlations. In this paper, we ask: is it possible to model label correlations for multi-label recognition, with the purpose of preserving good and suppressing bad contextual contents? This work attempts to answer this question, as well as quantifying the goodness of contexts (corresponding to all pre-defined categories in the dataset) for recognizing certain semantic category from the perspective of causal theory [22).
In this paper, we explore label correlations with principled causal intervention approach for multi- label image recognition. Causal intervention aims measure the causal effect from one cause variable to another effect variable, by "physically" putting the effect variable at any context to remove the effects of confounders in current image. Our key motivation lies on the realization that the causal label correlations (in the probability-raiste when contextual bias does exist in the training set. We pursue causal correlations (e.g., from "Person" to "Clothes") to mine contextual cues for recognition, while suppressing spurious correlations (e.g.. from "Person" " to "Cat") which are associated by confounders (e.g., the overall scene) and may mislead the classifier in testing.
sense [22]) are stable in both training and testing, even
We design an end-to-end framework that carefully integrates causal intervention into multi-label recognition. Specifically, we decouple label-specific features with a Transformer decoder attached to the backbone network, and model the confounders (imaginary contextual contents) by clustering spatial features of all training images. Based on label-specific features and confounders, we employ cross-atter attsenatttnionn module to implement causal intervention for all pre-defined categories, quantifying the causal label correlations from all object categories each predicted object category. Finally, we combine the predictions from decoupled features and causal correlations multi-label prediction.
In summary, our main contributions are as follows:
We propose conceptually simple, yet effective label correlation modeling approach based on causal intervention to tackle the issue of contextual bias for multi-label image recognition. It allows to capture causal label correlations (good contextual contents) to improve recognition accuracy, while suppressing the effects of spurious correlations (possible bad contextual contents) that may mislead the classifier.
We conduct comprehensive experiments with contextual bias to evaluate the effectiveness of the proposed method for multi-label image recognition. Under both common and cross- dataset settings, method consistently demonstrates advantages over existing methods.
2
Related Work
Correlation modeling for multi-label image recognition. Modeling label correlations is crucial for multi-label image recognition. Early approaches achieved this by embedding label correlations using

---

plate
Figure 2:
Illustration of causal label correlations and spurious correlations revealed by causal intervention, in a probability-raisine sense that if ((Y]do(X)) > P(Y), then causal correlation exists from X ("Person") to Y (categories in this figure).
but the performance is affected by the order in which the labels are predicted. To overcome the sequential issue, researchers attempted capture label correlations with graph structures, enabling simultancous prediction of the label sets [4] 28 12 25
Recurrent Neural Networks (RNNs) 32
Besides graph structures, several other approaches have been developed to establish label correlations. DER [5] employs metric learning to pull related label-specific features closer and push unrelated label features apart. C-Tran [15] introduces label mask training approach for general multi-label image classification, indirectly constructing label correlations by predicting masked labels. Q2L [9] employs Transformers decompose label features and utilizes self-attention mechanisms establish label correlations. SST 7 employs Transformers to simultaneously capture both spatial and semantic
label correlations.
From a broader view context modeling, label correlation modeling, arguably, can be understood as a strategy enhance label-specific contextual cues. However, contextual information is double-edged sword, and may mislead the classifier in presence of contextual bias in training.
Contextual bias and debiasing. While visual context is widely believed be beneficial for object recognition, recent works show that contextual bias may hurt multi-label image recognition [14] 17]. Such bias happens when an object category frequently co-occurs with some other object categories. Strongly relying on context may mislead the classifier, when typical contextual patterns around an object are absent or an object are absent from its typical context.
Due to the contextual bias issue, some recent works have discarded label correlation modeling for multi-label image recognition, since the contextual priors encoded by learnt label correlations may mislead recognition. Furthermore, these works developed contextual debiasing techniques, by decorrelating feature representations of a category from its co-occurring context [14), , removing contextual bias in features with causal mechanisms [17] We argue that these methods may discard useful contextual cues, leading to inferior accuracy for common objects in common contexts.
Causal intervention in vision. Causal intervention measures the causal relationship between two random variables, by removing the confounders that may associate them [22
Recently, causal
intervention has gained attention in the field of computer vision [17 226] 33
37]. with expectations to address the contextual bias or long-tailed distribution issues. For instance, VC-
34
27
36
RCNN [3 employs causal intervention and proxy tasks to extract unbiased visual features, which can benefit downstream tasks like Image Captioning, Visual Question Answering, and Visual Com- monsense Reasoning. Tang et al. [26] show that the SGD momentum is essentially a confounder in long-tailed classification, and propose to remove bad causal effects by intervention. Wang et B4 reveal that traditional attention module is biased in -iitrr-distribution setting, and propose causal attention unbiased visual recognition.
For multi-labed image recognition, Liu et al. 1 recently propose to remove the contextual bias in features with causal intervention. However, this approach does not consider label correlations, and may discard contextual evidences that are crucial for recognizing obscure instances.
3 Preliminaries and Motivation of Causal Correlations
Causal correlations in probability-raisin sense. Humans can easily understand the causal correlation between the presence of two objects (e.g.. "Person" is the cause of "Clothes"). Sceking
3

---

Cauaal Inerveniaas
Casaal Ennevestion
Figure 3: (a): Causal correlation between label 'nd Y, which is not affected by confounder set C. (b): Spurious correlation, where the co-occurence of X and Y is caused by confounder set C.
for calculable formal definition, we follow the statement in (33 if P( P(Y]do(X)) > P(Y). then causal correlation exists from X to Y in a probability-raising sense. Here do X) is the do-operation, which pursues the causality between the cause X and the effect Y without the confounding effect.
As shown in Fig. 2 our implementation of causal intervention (which will be elaborated later) can reveal the causal correlations between objects in above probability-raising sense. For example, given the observation of "Person", the probabilities of "Clothes" and "Skis" raise. But on the other hand, although "Cat" often co-occurs with "Person", with causal intervention the probability of "Cat" even decreases when observing "Person".
Understanding this requires delving deep into the implication of causal intervention. Causal intervention is reminiscent of randomised controlled trials [2). Based on the presence of X, causal intervention makes the prediction of Y goes beyond the limitation of the context of current test image. It measures the probability of Y. by putting Y at randomised context (which can be simulated by confounder set C) with X. Therefore, if X the cause of Y. it should be able to provide contextual cues to raise the probability of Y, regardless of the contents (except X the cause of Y, it will raise the probability Y based
Why pursuing causal correlations?
and Y) in current test image. If X on causal intervention, although they might co-occur frequently in the training set.
Formally, Fig. shows Structural Causal Model 23) where X and Y represent two labels, and C' represents the confounder set. In causal theory, each directed edge denotes a possible causal relationship between Iwo nodes. Fig. 3 and (b) illustrate two extreme cases: the causal correlation between label X and Y which is not affected by the confounder C, and the spurious correlation, where the co-occurrence of X and Y is caused by the confounder set C.
Causal intervention by backdoor adjustment. As "physical" intervention that puts Y at any typically applied for "virtual" intervention:
context is almost impossible, backdoor adjustment 22
P(Y P(Y|do(X)) = P
Here the key idea is to cut off the link from confounder C' to cause X, and stratify C' into pieces C = {c], making C' no longer correlated with X, and making X have fair opportunity to incorporate every confounder into the prediction of Y, subject to a prior P(c).
Approach
4.1 Overview of Proposed Pipeline
Building upon above analysis, we incorporate causal intervention into explicit label correlation modeling for multi-label image recognition, designing pipeline of two complementary branches (Fig. D the branch of decoupled label-specific features, and the branch of causal label correlations. In particular, the causal correlation branch is built upon decoupled features.
Given an input image, a backbone network (e.g., ResNet-50 [T]) is firstly employed to extract the spatial feature. Then, a Transformer decoder is leveraged to decouple label-specific features from the spatial feature. This branch predicts image labels based on objects themselves, rather than the context label correlations. To take into account the label correlations yet overcoming the effects of contextual bias, we construct a causal intervention branch, which explicitly models causal label correlations, and integrates them into prediction.
Formally, we denote the prediction confidence from the causal intervention branch as Doausats and the prediction from the decoupled feature branch ddecouples then the final prediction confidence

---

Decsupling Speciip Fentures
Summarzzing Causel Libel Correltions
Marcal
Cose-Aessiee FC,
Trrmeformer Decoder
Ingut
Rockbone
PC
Penion
Dog
Surihoard
Figure The overall framework of our proposed method.
Rdccouypl
scores y can be written as:
ÿ 1/2- Yeausal + 1/2 Ydecouple IR
(2)
where N is the number of categories. We utilize standard multi-label recognition loss train the model, which can be written as:
Σ Byt log(ū) - (1 ygt) log(1 9)
where Vgt {0, 1} is ground truth label vector of the input image.
In the following, will detail the designs of the decoupled feature branch and the causal intervention branch. We will also present explanations and discussions about causal correlations in context to facilitate understanding.
4.2 Predicting by Decoupling Label-Specific Features
We firstly decouple label-specific features for input image with two purposes: (i) predicting image labels based on objects themselves; (ii) preparing for label correlation modeling by causal intervention.
For decoupling label-specific features, common approaches include Class Activation Mapping (CAM) 40) and Transformer [30]. We employ a Transformer decoder for this purpose. Specifically, given input image I, we firstly use CNN backbone to extract spatial feature F:
F = fenn (T).
(4)
Then, we employ a standard Transformer decoder to decouple label-specific features X E RR from F:
X feecoder (Q. F).
Here, Q E NxD are learnable label embedding as queries, N and are the number of categories and dimensionality of spatial features, respectively.
Finally, we obtain the prediction confidence Ddecouple from decoupled features by:
deccouple: = a (ff(a(Q)))
where Jra (-) denotes the fully-connected layer, (f.) the sigmoid function. We apply multi-label loss to this branch for decoupled label-specific feature learning.
4.3 Predicting by Summarizing Causal Label Correlations
With label-specific features, we can construct our intervention branch which explicitly models causal label correlations for multi-label image recognition.
To estimate the probability of each category Y, on this causal intervention branch, a straightforward approach is firstly calculating P(Yyldo(X, for each category X,, and then merging them:
Plausat
Jmerge do()), .... (oo,
One might expect complex modeling of P(Y; do)X:)) and fmerge( (.) for calculating Eq. In this work, we introduce a simple yet effective implementation, using one cross-attention layer and one fully-connected layer.

---

Causal intervention based on label-specific features. We hypothesize that whether X; appears in the test image should be encoded by its label-specific feature z;- Given label-specific features z; and y5 for label X; and Yj. and one potential confounder feature c, we model the conditional likelihood [q as:
P X, C= = c) ofss (#4, c)),
(8)
where (-) is the sigmoid function. fu, (-) calculates the logit for label Y, Then, causal intervention can be calculated as:
P(Y (o( = [E (2r,
(9)
Due to the difficulty in directly computing Eq. 9 apply the Normalized Weighted Geometric Mean (NWGM) [3] to approximate the above equation:
P(Y (x))))(X.)) (fy,
= o oBe (ri,
= o = (2i, P(c))
(10)
Here P(c) is the prior of confounder, which can be obtained from data.
Calcualting Eq. 10 requires further modeling of J (IZ; c). However, instead of hypothesizing formulation of Jv, (zy, c) which only intervenes one category X; for Yj. describe an efficient formulation based on cross attention to intervene all categories X; for Yj in one step. This allows us to circumvent the calculation of each fu, (2i, c), making the calculation of Eq. more efficient.
Effective modeling for all fg, (z c) by cross-attention. Formally, following the notation in Eq. 5
let X [x,, xN] E RRN denote all label-specific features. To implement Eq. 7 we seek for a model to combine the information of all label-specific features x; and one potential confounder feature to predict the logit of label Yj. We employ cross-attention mechanism and fully-connected layer for this purpose:
Ze = cc,
Feause f mergg
fmerye((er Σ Fuy (21, P(c))..... (( (ZN, P(c))))
Σ Sus
a
a Σ
(Ujr Ze, Zo)) P(c)),
(11)
where Ze the addition-basee combination all label-specific features and a confounder feature and S/2 is a fully-connected layer applied upon the cross-attention feature to obtain the logit. Algorithm provides the pscudocode of causal intervention process.
Modeling the confounders It remains an open question about cofounder modeling for visual recognition tasks. In VC R-CNN [33], the authors treated objects as confounders, and extract object- level features based on bounding box annotations. However, one hand, there is no bounding box annotation in the typical setting for multi-label image recognition. On the other hand, we argue that cofounders for recognizing certain object are often hard to define and enumerate objects, scene, and even the texture the environment are all potential confounders. For example, suppose that an image contains two objects: "Person" and "Surfboard", with the scene being the "Beach", In VC R- R-CNN, the "Surfboard" is considered the confounders for the "Person". However, in our opinion, the true confounder should the "Beach", although it does not have an associated image label.
Based on above analysis, to characterize these non con-enumerable confounders, high-level spatial features of training images from pre-trained classification CNN provide good choice, as semantic objects/regions are often activated classification features. By clustering spatial features with K-means algorithm, we obtain a compact set of prototypes to represent potential confounders like objects, scenes and textures. We empirically show the effectiveness of this simple approach for modeling the confounders.

---

Algorithm Pseudocode Causal Intervention PyTorch-like style.
# X (NxD): The label specific features
# C (MxD):
: Confounders
# (M) Priors
#
is the number of categories
# D is the dimensionality of spatial features
# M is the number of confounders
2 K.unsequeeze (1) C. unsqueeez (0) NxMxD
y-j x[j,:]
y_causal.j
for in range(M) (M)
y_causal_] +- fc2 fc2 (cross_atten, (y-j, o[;,c,;], ((;,c,;])) P[c]
y_causal_j y_causal_) s.immsal..ssigmodd
Experiments
5.1 Experimental Settings and Implementation Details
Common Setting. The COCO-Stuff [ and DeepFashion datasets are experimented in com- mon setting, where the training and test sets are from the same dataset. We strictly adhere to the evaluation setup employed in [14,, and report the performance under two different test distributions: "Exclusive" denotes virtual co-occurrence where labels appearing simultaneously in the training set do not co-occur in the test set, and "Co-occur" represents the objects co-occurring both the training and test sets. "All" is the average performance of all categories. We report top-3 recall for DeepFashion and mAP for COCO-Stuff.
Cross-dataset Setting. We consider more challenging yet practical setting in real-world applica- tions: the training and test sets are from different datasets, and may suffer from serious contextual bias issue. We simulate this setting by using MS-COCO [16] for training and NUS-WIDE (8] for testing, and vice versa. particular, we select the same categories (14 common classes) from both the MS-COCO and
datasets for experiments. We report the mean Average Precision (mAP) for all categories this setting.
Implementation Details. To fair comparison with previous methods, employ ResNet-50 and ResNet-101 as backbones for the common setting and real-world setting, respectively. We utilize ImageNet [9] for model parameter initialization. For the cross-dataset setting, the input images are resolution 448 448, while the common setting, resolution is set to 224x224. To extract label-specific features, employ 2-1layer Transformer decoder with 4 attention heads. For modeling confounders, we first train a baseline model with standard multi-label loss, then extract spatial features from all the images in the training dataset, and employ К K-means algorithm (default number of clusters is set to 80) to cluster all pixel-level spatial features. The Adam optimizer chosen for model optimization, with weight decay of 2c 2 and (B1, {22 (0.9, 0.9999). The initial learning rate is set to le - 4, and we employ a cyclic learning rate policy to train our model for 80 epochs. All of the experiments processor, 256 GB main memory, and eight GTX-3090 GPUs.
randomly cropped
on computer with AMD EPYC 7542 32-Core
5.2 Comparing to the State-of-the-arts
Common Setting. In Table we report the performance COCO-Stuff and DeepFashion datasets, where the baseline the vanilla Resnet-50 with standard multi-label loss. Comparing with the baseline, we observed a significant improvement (55.0% 60.6% on "All" mAP). Furthermore, our proposed method outperforms all state-of-re of-the-arts on these two benchmarks. For example, obtains +1.5% "Exclusive" and +3.6% "Co-occur" improvements over the feature-split [14] COCO-Stufi dataset.
We also observe that previous methods that directly build label comelations based on graph structures are affected by contextual bias datasets. For example, MI-GGN 4 and SSGRL. (28) both construct label graphs based the training set and utilize the graph structure capture correlations between
7

---

Table I: Performance comparison in the common setting the COCO-Stuff and DeepFashion datasets.
COCO-Stuff (mAP)
Deepfashion (top-3 recall)
Method
Exclusive
Co-occur
All
Exclusive
Co-occur
All
Q2L 19
23.5
67.1
57.2
12.8
26.3
26.1
ADD-GCN 12
20.6
64.8
55.2
8.2
22.6
23.5
ML-GGCN 4
18.6
67.1
55.1
10.3
23.7
24.0
SSGRL 28
18.1
66.6
54.9
7.9
22.8
23.1
C-Tran [15
22.4
65.1
55.4
11.4
24.6
24.8
CCD
23.8
65.3
55.9
11.5
24.2
24.6
TDRG 38
20.0
64.8
56.2
8.1
22.9
23.6
IDA 18
25.2
64.9
57.0
11.3
25.1
25.4
CAM- Based
14
26.4
64.9
feature-split [1
28.8
66.0
9.2
20.1
Baseline (R50)
21.9
65.5
55.0
11.5
24.1
24.1
Ours
29.7
69.6
60.6
14.6
27.4
28.8
Table 2: mAP Performance comparison in the cross cross-dataset setting on the MS-COCO and NUS- WIDE datasets.
Method
MS-COCO - NUS-WIDE NUS-WIDE + MS-COCO
ADD-GCN
81.8
77.2
ML-GCN
81.4
77.2
SSGRL 28
80.2
76.1
C-Tran 15
80.9
76.9
CCD
81.9
78.3
Q2L [19
82.1
78.6
IDA 18
82.3
78.9
CAM-Based
14
81.0
77.8
feature-splii
14
81.9
78.3
Baseline (R101)
81.1
77.1
Ours
83.2
80.2
labels. These two methods cannot achieve significant improvements over the baseline, and even exhibit noticeable performance degradation. We speculate that the graph structure constructed from the training set are not applicable to the test set in presence of contextual bias.
Cross-dataset Setting. Table 2alle@repune reports the results on the MS-COCO and NUS- WIDE datasets, where the baseline the vanilla Resnet-101 with standard multi-label loss. Similar to the common ting, the performance previous methods affected by contextual bias, and our method outperforms all other state-of-the-ar methods. However, compared the baseline, the performance improvement of our method is not as significant as in the common setting. We speculate that, apart from contextual bias in training, the inconsistency data distribution may also affect the performance. Even so, the cross-dataset results can indicate robustness and generalization capabilities of our method.
5.3
Ablation Studies
In this section, we conduct ablation studies by using ResNet-50 as backbone on COCO-Stuff Dataset.
5.3.1
Effectiveness of Different Modules
We investigate the impacts of key modules in our framework. Specifically, there are two essential modules, i.e., decoupling the label feature module (denoted as "Decouple") and causal intervention module (denoted as "Causal"), Table shows the mAP performance by progressively integrating the above two modules. Solely applying "Decouple" on the backbone gives +1.8% "All" mAP, +0.2% "Exclusive" mAP and + +1.5% "Co-occur" mAP improvement. Directly applying decoupling leads to improved "Co-occur" performance but fails to enhance "Exclusive" performance. We speculate

---

Table 4: The impacts of clustering center
Table 3: The impacts of different modules.
ber.
Decouple Causal Exclusive
Co-occur
All
Number
Exclusive
Co-occur
All
21.9
65.5
55.0
20
26.9
68.9
59.6
22.1
67.0
56.8
40
26.8
69.3
60.1
29.7
69.6
60.6
60
29.3
69.8
60.2
80
29.7
69.6
60.6
100
29.5
69.4
60.5
Table 6: The impact of different modeling approaches for confounders.
Table 5: The impact of backbones for clustering.
Confounder Backbone Exclusive
Co-occur
All
Method
Exclusive
Co-occur
All
ResNet-50
29.7
69.6
60.6
Random
22.1
66.3
56.
ResNet- 101
29.6
69.9
60.5
Early
27.8
69.1
60.1
DEIT3-Large 29.4
69.7
60.5
Label
28.0
69.3
60.3
K-means
29.7
69.6
60.6
that the cross-attention mechanism can capture the long-range dependencies, since it can implicitly model causal correlations but cannot eliminate spurious co-occurrence correlations. Then, the causal intervention module, which can remove spurious co-occurrence correlations and capture the causal correlations, bring another 3.8% "All" mAP. These results show the effectiveness of our approach in alleviating label correlations bias.
5.3.2 Investigation of Confounders
Modeling confounders is core step in our approach. In order to investigate the impact of different confounders, we designed comparative experiments at three aspects, i.e., the number of clustering centers, different clustering features, and different approaches for modeling confounders.
The number of clustering centers. In order investigate the influence of the number of clustering centers, we conducted experiments with different numbers of cluster centers: 20, 40, 60, 80, and 100, respectively. The experimental results are presented in the Table certain range, increasing the number of cluster centers does not significantly affect performance. However, performance decline noticeable when the number of clusters reduced to We speculate that the confounders correspond many attributes including semantic, color, texture, and so on. smaller number of cluster centers can only express very limited attributes, resulting in decline in performance.
can be observed that within
Different backbones for clustering. By default, we employ ResNet-50 as the backbone ex- tract features from all images in the training set on COCO-Stuff, and perform clustering to obtain confounders. In order investigate the impact of different clustering features, we utilize distinet backbones for feature extraction. As shown in Table []] the confounders obtained from different backbones have not significant influence on final performance. The role of the confounders is to identify label causal correlations and does not directly participate in recognition. Therefore, the features extracted by a weak backbone seems sufficient identifying causal correlations, and the use of strong backbone does not lead to performance improvement.
Different approaches for modeling confounders. In this paper. we utilize clustering centers to characterize confounders. We compare it with two additional modeling techniques. The experimental results as shown in the Table "Random" means using random vectors to replace cluster centers as confounders. "Early" means fusing features from early epochs directly to obtain the confounder, which is adopted by [17]. "Label" means directly using label- specifcc features as the confounder. Our modeling of confounders using K -means yields the best results. "Random" leads to significant performance degradation (measured by mAP), especially on "Exclusive" subset. Since, we believe that confounders should be modeled at semantic level. The "Early" approach relies solely simple feature fusion, which fails to effectively differentiate various attributes. The "Label" approach only employs object semantics the confounder, leading to the omission of other attributes. By contrast,

---

Table 7: Different implementations of Eq.
Method
Exclusive
Co-occur
All
Linear
27.4 .4
69.1
60.0
Ours 29.7
69.6
60.6
our approach incorporates both object-related information and other contextual information, offering a more expressive set of confounders.
5.3.3 Different Implementations of Eq.
We investigate different implementations Eq. Following [331. we employ a simpler linear approach and average merge method model Eq. 7 As shown in Table [ we can observe that employing linear modeling approach yields competitive results, but our method still outperforms We speculate that while linear modeling can implement Eq. our approach utilizes cross-attention, which possesses the ability to model long-range dependencies, thereby efficiently capturing causal correlations and suppressing spurious correlations.
Conclusions
In this paper, we presented principled approach address the contextual bias issue for multi-label image recognition. Using causal intervention from causal theory, we pursued causal label correlations, and integrated them into multi-label prediction. We evaluate the effectiveness of our approach with both quantitative and qualitative assessments. In the future, we will investigate more advanced structural causal models for better describing the label correlations.
Acknowledgements
This research was supported by the National Natural Science Foundation of China (Grant No. 62202337, 61906168, 62072340), in part by the Zhejiang Provincial Natural Science Foundation of China (Grant No. LQ24F020016, LY23F020023), Key Science and Technology Innovation Project of Wenzhou(Grant No. ZG2022014) 4)), and in part by the Wenzhou Key Laboratory Construction Project (Grant No. 2022HZSY0048).
References
[1] H. Caesar, Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pages 1209-1218, 2018.
[2] T. C. Chalmers, H. Smith Jr, B. Blackburn, B. Silverman, B. Schroeder, D. Reitman, and A. Ambroz. A method for assessing the quality of randomized control trial. Controlled clinical trials, pages 31-49, 1981.
[3] Z. Chen, X.-S. Wei, P. Wang, and Y. Guo. Learning graph convolutional networks for multi-label recognition and applications. IEEE TPAMI, page in press, 2021.
[4] Z.-M. Chen, X.-S. Wei, P. Wang, and Y. Guo. Multi-label image recognition with graph convolutional networks. In CVPR, pages 5177-5186, 2019.
[5] Z.-M. Chen, Q. Cui, X.-S. Wei, Jin, and Y. Guo. Disentangling, embedding and ranking label cues for lulti-label image recognition. IEEE TMM, 23:1827-1840, 2020.
[6] Z.-M. Chen, X. Jin, B.-R. Zhao, X. Zhang, and Y. Guo. Hce: Hierarchical context embedding for region-basse object detection. IEEE TIP, 30:6917-6929, 2021.
[7] 7.-M. Chen, Q. Cui, Zhao, R. Song, Zhang, and O. Yoshie. Sst: Spatial and semantic transformers for multi-label image recognition. IEEE TIP, 31:2570-2583,, 2022.
[8] T.-S. Chua, Tang, R. Hong, Li, 7. Luo, and Zheng. NUS-WIDE: real-world web image database from national university of singapore. In CIVR, pages 1-9, 2009.
10

---

[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, pages 248-255 2009.
[10] B.-B. Gao and H. Zhou. Learning to discover multi-class attentional regions for multi-label image recognition. IEEE TIP, 30:5920-500 -9330005-2222 2021.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[12] Y. Jin, H. Junjun, Xiaojiang, W. Wenhao, and Q. Yu. Attention-driven dynamic graph convolutional network for multi-label image recognition. In ECCV, pages 649-665, 2020.
[13] X. Kelvin, B. Jimmy, K. Ryan, C. Kyunghyun, C. Aaron, S. Ruslan, Z. Rich, and B. Yoshua. Show, attend and tell: Neural image caption generation with visual attention. ICML, pages 2048 2008-2057,200 2015.
[14] S. Krishna, Kumar, M. Dhruv, G. Kristen, L. Yong, Jac, F. Matt, and G. Deepti. Don't 't judge an object by its context: Learning to overcome contextual bias. In CVPR, pages 11070-11078, 2020.
[15] J. Lanchantin, T. Wang. V. Ordonez, and Y. Qi. General multi-label image classification with transformers. In CVPR, pages 16478-16488, 2021.
[16] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740-755, 2014.
[17] R. Liu, H. Liu, G. Li, H. Hou, T. Yu, and T. Yang. Contextual debiasing for visual recognition with causal mechanisms. In CVPR, pages 12755-12765, 2022.
[18) R. Liu, J. Huang, H. L.. Thomas, and G. Li. Causality compensated attention for contextual biased visual recognition. In ICLR, pages 2023.
[19] S. Liu, L. Zhang, X. Yang, H. Su, and J. Zhu. Query2label: A simple transformer way to multi-label classification. arXiv preprint 0rXiv:2107.10834 pages 1-16, 2021.
[20] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, pages 1096-1104, 2016.
[21] J. Nam, E. Loza Mencía, H. J. Kim, and J. Fümkranz.. Maximizing subset accuracy with recurrent neural networks in multi-label classification. In NeurlPS, page 5419-5429, 2017.
[22] J. Pearl. Causal inference in statisties: An overview. Statisties surveys, 31:96-146, 2009.
[23] J. Pearl. Causality. Cambridge university press, 2009.
[24] S. Ramprasaath, R., C. Michael, D. Abhishek, V. Ramakrishna, P. Devi, and B. Dhruv. Grad- cam: Visual explanations from deep networks via gradient-badt based localization. In ICCV, pages 618-626, 2017.
[25] Y. Renchun, G. Zhiyao, Lei, L. Xiang, B. Yingze, and W. Shilei. Cross-modality attention with semantic graph embedding for multi-label classification. In AAAI, pages 1-9, 2020.
[26] K. Tang, J. Huang, and H. Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurIPS, 33:1513-1524 2020.
[27] K. Tang, Y. Niu, J. Huang. J. Shi, and H. Zhang. Unbiased scene graph generation from biased training. In CVPR, pages 3716-3725, 2020.
[28] C. Tianshui, X. Muxin, H. Xiaolu, W. Hefeng, and L. Liang. Learning semantic-specific graph representation for multi-label image recognition. In ICCV, pages 522-531,
[29] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
[30] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, N. Gomez, Kaiser, and I. Polosuukin. Attention is all you need. In NeurIPS, page 6000-6010, 2017.

---

[31] D. Wang and S. Zhang. Unsupervised person re- re-identification via multi-label classification. In CVPR, 2020.
[32] Wang, Y. Yang. J. Mao, Z. Huang, C. Huang, and Xu. CNN-RNN: A unified framework for multi-label image classification. In CVPR, pages 2285-2294, 2016.
[33] T. Wang. J. Huang, H. Zhang, and Q. Sun. Visual commonsense r-cnn. In CVPR, pages 10760-10770, 2020.
[34] T. Wang, C. Zhou, Q. Sun, and H. Zhang. Causal attention for unbiased visual recognition. In ICCV, pages 3091-3100, 2021.
[35] W. Wang, H. Bao, L. Dong, Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed,
S. Singhal, S. Som, and F. Wei. Image as foreign language: BEiT pretraining for vision and vision-language tasks. CVPR, pages 19175-19186, 2023.
[36] Z. Yue, H. Zhang. Q. Sun, and X.-S. Hua. Interventional few-shot learning. In NeurlPS, pages 7734-2746, 2020.
[37] D. Zhang, H. Zhang, J. Tang, X.-S. Hua, and Q. Sun. Causal intervention for weakly-supervised semantic segmentation. In NeurlPS, pages 655-666, 2020.
[38] J. Zhao, K. Yan, Y. Zhao, X. Guo, F. Huang, and J. Li. Transformer-base dual relation graph for multi-label image recognition. ICCV, pages 163-172. 2021.
[39] Z. Zhao, Y. Qiang. Yang. X. Hou, J. Zhao, and K. Song. Two-stream vision transformer based multi-label recognition for tem prescriptions construction. CIBM, 170: 107920, 2024.
[40] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2016.
[41] Zhu, H. Li, W. Ouyang, N. Yu, and X. Wang. Learning spatial regularization with image-level supervisions for multi-labe image classification. In CVPR, pages 5513-5522. 2017.
12

---

A Supplemental Material
A.1 Limitations
Despite improved accuracy, our causal intervention-based multi-label image recognition algorithm still has several limitations. First, we model confounders by clustering the spatial features extracted from pre-trained classification CNN. On one hand, this approach is dataset-dependent, and on the other hand, although our modeled confounders mainly concerne with object-level, texture-level and scene-level concepts, it is not possible to determine the specific semantics of these concepts, resulting in limited interpretability. Second, we only integrate the causal intervention technique in causal theory in maulti-lbbe image recognition pipeline. In the future, we will consider higher-level causality such as counterfactual reasoning.
A.2 Experiments
A.2.1 Datasets
COCO-Stuff [ covers 171 classes and contains 82,783 training images and 40,504 testing images. DeepFashion [20] contains 209,222 training images and 40,000 testing images. Following [14), for DeepFashion we only consider the 250 most frequent attributes in the training data since other attributes do not contain sufficient samples.
We also evaluate method in more challenging cross-datased setting: training MS-COCO [16), but testing on WUS-WIDE 8 dataset. The MS-COCO dataset contains 122,218 images and covers 80 common objects, with average of 2.9 labels per image. The NUS-WIDE dataset has 269,648 images and 81 concepts, with an average of 2.4 concept labels per image.
A.2.2 Intra-dataset Comparisons MS-COCO
In our main paper, we present cross-dataset experiments show the effectiveness of our method for contextual bias and distribution shift. That is, we train on MS- MS-COCO [16], but test on NUS- WVS-IDD [8]; or train on NUS- WIDE [8] but test on MS-COCO [6]. Here, we also report the accuracy of our method for general multi- recognition, by both training and testing on MS-COCO.
As shown in Tab although our approach is designed to solve the contextual bias issue for multi-label recognition, it can achieve competitive accuracy for general multi-label recognition. In particular, our method outperform CCD (17) by 0.9 mAP, which also integrates causal mechanisms for contextual de-biasing, but discards the explicit modeling for label correlations. Combining the cross- cross-dataset and intra-datasct experimental results, our approach shows advantages over both general multi-labe
image recognition algorithms 7 and recent algorithms that explicitly considers contextual de- de-biasing
A.2.3 Effect of different backbones.
To investigate the impact of different backbones performance, we conducted experiments using ResNet-50, ResNet- 101, and BEIT-Large [35] as backbones. The results are shown in Table 9 which reveal that even with stronger backbone, our method can still effectively improve performance. For example, our method with the BEiT-Large backbone achieves a 72.2% mAP, which outperforms the baseline by 2.9% mAP. This experimental result demonstrates that our method can be generalized to stronger backbones.
A.2.4 Visualizations
For qualitative verification, we employ Grad-CAM baseline and our proposed method. As shown in Fig. 5 although the baseline can activate relevant object locations, also activates the regions with spurious correlations for the target object. In contrast, our method activates solely at the object's location. In Fig. (a), besides activating label "Cell Phone", the baseline method additionally activates label "Person", whereas our method solely activates "cell phone." Regarding negative labels, when an input image lacks specific label, the baseline might erroneously infer false labels through other labels due to excessive reliance on label co-occurrences. Our method mitigates this phenomenon effectively. In Fig. 5 labels
visualize label-specific features for the
13

---

Table 8: Comparisons with state-of-the-ar MS-COCO [6], where the models all trained on MS-COCO dataset. Here, means the re-produced results by ourselves.
Methods
Resolution
mAP
CNN-RNN 224
61.2
SRN 4
224x224
77.1
ResNet-101
448x448
79.1
SSGRL# 28
448x448
81.9
DER
448x448
82.8
ADD-GCN
12
448x448
82.8
ML-GCN
448x448
83.0
C-Tran 15
448x448
83.1
P-GCN
448x448
83.2
MCAR 10
448x448
83.8
MS-CMA
25
448x448
83.8
CCD 17
448x448
84.0
SST
448x448
84.2
TDRG 38
448x448
84.6
IDA 18
448x448
84.8
Q2L 19
448 x448
84.9
Ours
448x448
84.9
Table 9: The impacts of different backbones on COCO-Stuff.
Backbone
Ours
Exclusive
Co-occur
All
21.9
65.5
55.0
ResNet-50
V
29.7
69.6
60.6
23.7
68.5
58.9
ResNet-101
V
31.0
70.3
61.4
46.8
80.4
69.3
BEIT3-Large
56.6
81.2
72.2
"Person", "Baseball Glove", and "Baseball Bat" co-occur in the training set, the baseline infers the presence of "Baseball Bat" through "Person" and "Baseball Glove", even if it is absent in the input image. This eventually leads the baseline to incorrectly activate spatial features associated with the label "Basehall Bat". In contrast, visualization results further validate our motivation, iee., our approach effectively removes spurious co-occurrence correlations and captures the causal correlations.
method nearly avoids activating such features. The
Input
Baseline
Ours
Input
Baseline
Ours
Postive Label:
Negutive Label: Bechalll Bal
Coll Phone
Poctivv Label:
Negative Label:
Dog
Sporn
Figure 5: Visualization for the spatial feature map of baseline and our proposed method using the Grad-CAM 24
A.2.5
(-SNE Visualization of Features
In Fig 6 we show the t-SNE 29 visualization of our confounders and two typical label-specific features ("Baseball Glove" and "Person").
Without causal intervention, we observe that "Baseball Glove" and "Person" are mixed with each other, indicating that they often co-occur in the training set. Furthermore, the confounders are around
14

---

Cloe
Dockel Ciae
Pone
(a) Baseline
Figure 6: t-SNE 29 visualization for confounders and label-specific features of "Baseball Glove" and "Person".
label-specific features, since they are obtained by clustering the spatial features of pre-trained classification backbone.
With causal intervention branch, we empirically find that "Baseball Glove" and "Person" pulled apart, since they have very weak causal correlations in the probability- sense. On the other hand, the label-specific features are pulled away from the confounders, suggesting that causal intervention can remove the influence of confounders for causal representation learning.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Lines 7 to 11 well lines 47 to 55 elaborate on the contributions of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Due to the page limitations, we provide detailed analysis of the limitations of our method in the supplementary materials.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification:
All theories presented in our paper are supported by corresponding references
proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4 and Section 5.1 provide all details to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httaa//gguudeee/eeeeabiiiisssss /Cc////oddiiisssseeeeeee ssiiiiiiscyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section 5.1 provide all experimental settings and implomentation details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The dataset partition and random seed are fixed, thus eliminating the possibility error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Section 5.1 provide the information computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics, and this paper conforms it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper solely utilizes causal theory to establish causal relationships between image labels, hence it may not have any societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper has appropriately referenced the original assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "xLjHlqwVai", "reviewer": "Reviewer_NEe5", "summary": "This paper proposes a causal intervention mechanism for multi-label image classification, where causal label correlations are pursued and spurious label correlations are suppressed. To achieve this, the authors frame a pipeline consisting of a branch for decoupling label-specific features and a branch for summarizing causal label correlations. The results from both branches are combined for final predictions on image labels. Comparative experiments and ablation studies demonstrate the effectiveness of the proposed causal intervention mechanism.", "questions": "- How is the operation $f_{merge}$ removed from the second line of Eq.11?  Why is the summation over all confounders $c$ also removed? Even if it can be removed, which confounder does $c$ in the last line of the formula refer to? The Eq.11 is confusing and needs further clarification. It would be best if the authors add a pseudocode to illustrate this process.z\n- Are the results in Table 1 and Table 2 reported by re-training these models on the relevant datasets? If so, the authors should clarify the experimental details for fair comparison.\n- According to the Table 8, in terms of the intra-dataset comparisons on MS-COCO, Q2L achieves the same performance in mAP as the proposed method. However, Q2L only requires a Transformer decoder for decoupling label-specific features. Therefore, we question the generalizability of the proposed causal intervention mechanism on multi-label image classification task, wondering whether it is only effective on specific datasets.", "limitations": "See Weakness and Questions.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is generally well written with clear motivation and objectives.\n- Causal intervention is technically novel and well motivated in terms of multi-label image classification. \n- Experimental results are impressive, outperforming sub-optimal methods by a considerable margin. Ablation studies are aslo well designed to showcase the contribution.", "weaknesses": "- Line 175: 'Correaltions' -> 'Correlations'.\n- Line 237: 'Transformer encoder' -> 'Transformer decoder'.\n- $f_{fc}$ in Eq.6 and $f_{fc}$ in Eq.11 should be different if their parameters are not shared.\n- In Figure 4, in the causal label correlation branch, the confounder features are added into label-specific features. However, the outputs are not seen to be used in subsequent steps, and it seems that only the label-specific features are utilized for causal intervention. The diagram of this module needs to be improved.\n- More experimental evidence should be provided to verify the effectiveness of the confounder modeling. For example: using random vectors to replace cluster centers as confounders. Only the feature visualization and ablation study on clustering center number are unconvincing.\n- Although this paper is well motivated, the modeling process, especially Equation 11, is confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer NEe5 for the constructive comments and suggestions. In the following, we present our responses addressing the raised concerns. Should our rebuttal effectively address the concerns, we kindly hope you can raise your score.\n\n**(Weakness 1 and 2)** Thanks for your reminding, and we will correct these typos in our revised manuscript.\n\n**(Weakness 3)** To differentiate these two different fully-connected layers, we will use $f_{fc1}$ and $f_{fc2}$ in Eq. 6 and Eq. 11, respectively.\n\n**(Weakness 4)** Thanks for your comments, which remind us that the current diagram for causal label correlation modeling in Fig. 4 lacks a crucial arrow to “feed” the added features into the cross-attention operation. We update our Fig. 4 (see our attached pdf file) to show this.\n\n**(Weakness 5)** Thanks for your kind suggestion, and we agree that random confounders should be the simplest baseline for comparison. We conduct experiments for this random version of confounders: it leads to significant performance degradation (measured by mAP), especially on the “Exclusive” subset of COCO-Stuff. We believe that confounders should be modeled at semantic level (which cannot be achieved by random vectors), and thus present our K-means based solution upon semantic features extracted by a pre-trained classification backbone network.\n\n| Method | \tExclusive |\tCo-occur\t | All |\n| --- | --- | --- | ---\n| Random\t|  22.1\t |66.3\t | 56.1 |\n| K-means |\t29.7 | \t69.6\t | 60.6 |\n\n**(Weakness 6)** We agree with you that our modeling process (as formulated in Eq. 11) should be further clarified. Please refer to our responses to Question 1 in the following.\n\n**(Question 1)** Regarding the $f_{merge}$ in the first two rows in Eq 11, it represents our high-level idea that “We seek for a model to combine the information of all label-specific features $x_i$ and one potential confounder feature c to predict the logit of label $Y_j$. From the third row in Eq.11, we merge all label-specific features $x_i$ into a feature matrix $X$, and thus $f_{merge}$ can be removed. \n\nRegarding the summation over all confounders, we thank for your kind reminding: it cannot be removed in Eq. 11. We will update Eq. 11 in our revised paper as the following.\n\n$$\n\\begin{align}\n  {Z}\\_c &= {X} + c \\,, \\\\\\\\\n  \\hat{y}\\_{causal}^j &= f_{merge}([P(Y_j|do(X_1), ..., P(Y_j|do(X_N)]) \\\\\\\\\n              &= f_{merge}([\\sigma(\\sum_{c} f_{y_j}(x_1, c) \\cdot P(c)), ..., \\sigma(\\sum_{c} f_{y_j}(x_N, c) \\cdot P(c))]) \\\\\\\\\n              &\\approx \\sigma(\\sum_{c} f_{y_j}({X}, c) \\cdot P(c)) \\\\\\\\\n              &= \\sigma(\\sum_{c} f_{fc2}(f_{cross\\\\_atten}(y_j, {Z}_c, {Z}_c))\\cdot P(c))\n\\end{align}\n$$\nwhere ${Z}\\_c$ is the addition-based combination of all label-specific features ${X}$ and a confounder feature $c$, \nand $f\\_{fc2}$ is a fully-connected layer applied upon the cross-attention feature to obtain the logit.\n\nWe provide the following pseudocode to illustrate the implementation process of Eq. 11.\n\n---\nDefine $X \\in \\mathbb{R}^{N \\times D}$, $C \\in  \\mathbb{R}^{M \\times D}$, $P \\in  \\mathbb{R}^{M}$ is the label-specific features, confounders and priors.\n\nZ = X.unsqueeze(1) + C.unsqueeze(0) # $Z \\in \\mathbb{R}^{N \\times M \\times D}$\n\nyj = X[j, :]\n\ny_causal_j = 0\n\nfor c in range(M):\n\n&nbsp; &nbsp; &nbsp; &nbsp; y_causal_j += fc(cross_atten(yj, Z[:,c,:], Z[:,c,:])) * P[c]\n\ny_causal_j = y_causal_j.sigmoid()\n\n---\n\n**(Question 2)** To ensure fair comparisons, we retrain these models based on the codes released by authors.\n\n**(Question 3)** Honestly, we present our results on MS-COCO for completeness, rather than for showing the advantage of our method. This is because MS-COCO generally satisfies the i.i.d assumption, while our method aims to improve practical multi-label recognition where the training and test images may not follow the i.i.d assumption, as the co-occurence between objects might change in testing. We follow the basic experimental settings in [1] which also aims to overcome contextual bias (but not from a causal intervention perspective), but also present our results under challenging cross-dataset setup. In this sense, our main experimental results are sufficient to validate the core contributions of this work: our method significantly outperforms existing methods under non-i.i.d settings, and can still achieve competitive results for i.i.d setting.\n\n[1] Singh, Krishna Kumar, et al. \"Don't judge an object by its context: Learning to overcome contextual bias.\" CVPR. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2BA0lmWea", "reviewer": "Reviewer_wRCW", "summary": "This paper presents a novel approach to addressing label correlations in multi-label image recognition by using causal intervention. The method involves decoupling features, modeling confounders, and implementing causal interventions to capture useful contextual information while suppressing spurious label correlations. This approach is highly innovative and has significant potential applications.", "questions": "1.  I noticed another paper titled \"Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training\" that also employs causal inference to address multi-label image classification. The methods in these papers differ in implementation and theoretical basis. Could you further elaborate on the main differences and advantages of your approach compared to this work?\n2.  Your paper does not provide a detailed analysis of the hyperparameters involved in the proposed method. Could you explain the rationale behind the chosen hyperparameters and their impact on the model's performance?\n3. There is a lack of discussion on the limitations of your proposed method. In what scenarios might your method underperform, and how could future work address these limitations?\n4. Could you explain why certain confounders were selected for modeling in your approach? How does the choice of confounders impact the effectiveness of causal intervention in your model?\n5. How does your method handle cases where the causal relationships between labels are weak or not well-defined? Does this affect the model's accuracy, and if so, how?\n6. How does your approach ensure robustness against noise and variability in the data? Are there any specific strategies employed to handle noisy or incomplete labels?\n7. Could you provide more details on how the feature decoupling using the Transformer decoder specifically contributes to reducing contextual biases in multi-label image recognition?", "limitations": "The effectiveness of the proposed method relies heavily on accurately modeling the confounders. If the selection of confounders is not precise or representative of the underlying data distribution, the causal intervention may not effectively distinguish between useful contextual information and spurious correlations. This could potentially limit the method's performance in scenarios where confounder selection is challenging.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Approach:**\n   The paper introduces a novel method that applies causal intervention to model label correlations in multi-label image recognition. This innovative approach addresses the challenge of spurious label correlations and captures useful contextual information, which is a significant advancement in the field.\n\n2. **Comprehensive Methodology:**\n   The proposed framework integrates several complementary techniques, including feature decoupling with a Transformer decoder, confounder modeling through clustering, and causal intervention using cross-attention mechanisms. This comprehensive methodology enhances the robustness and accuracy of multi-label image recognition models.\n\n3. **Thorough Experimental Validation:**\n   The paper conducts extensive experiments across multiple datasets, demonstrating the effectiveness of the proposed method. The results consistently show improvements over existing approaches, particularly in scenarios with contextual biases, underscoring the practical value of the method.", "weaknesses": "1. **Lack of Hyperparameter Analysis:**\n   The paper does not provide a detailed analysis of the hyperparameters involved in the proposed method, such as the number of clusters for confounders or the parameters of the cross-attention module. A sensitivity analysis of these hyperparameters would be beneficial to understand their impact on model performance and to guide practitioners in tuning the model effectively.\n\n2. **Insufficient Discussion on Method Limitations:**\n   The paper lacks a thorough discussion on the limitations of the proposed method. It would be valuable to include an analysis of scenarios where the method might not perform well, such as when the selection of confounders is inaccurate or when the causal relationships between labels are weak. Addressing these limitations can provide a more balanced view of the method's applicability and robustness.\n\n3. **Limited Ablation Studies:**\n   Although the paper includes some ablation studies, the number and depth of these experiments are not comprehensive enough. More detailed ablation studies are needed to analyze the independent contribution of each module (e.g., feature decoupling, confounder modeling, and causal intervention) to the overall performance. This would help in understanding the importance and effectiveness of each component of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer wRCW for the detailed feedbacks on our work. In the following, we present our responses addressing the raised concerns. Should our rebuttal effectively address the concerns, we kindly hope you can raise your score.\n\n**(Weakness 1)** We agree with you that a detailed analysis of the hyperparameters is crucial to validate the robustness of our method. Regarding the number of clusters for confounders as you suggested, **we have already presented the results in our main paper**. Please refer to Tab. 4 and Sec. 5.3.2 for details. As for the parameters, in the following Table, we compare our method to the baseline (naive ResNet-50) and Q2L (SOTA method). Our method significantly outperforms the baseline, as well as the SOTA Q2L method which has much larger model size. In particular, our causal module only adds marginal parameters, but can improve the accuracy (mAP) on COCO-stuff from 56.8 to 60.6, demonstrating the effectiveness of our causal label correlation modeling.\n\n| Decouple |\tParam.\t| mAP on COCO-stuff All |\n| --- | --- | --- |\n| Q2L\t |175.3M |\t57.2 |\n| Baseline\t | 26.9M |\t55.0 |\n| Baseline+Decouple |\t41.1M |\t56.8 |\n| Baseline+Decopule+Causal (Ours) |\t45.3M |\t60.6 |\n\n**(Weakness 2)** We agree with you that a discussion about the limitations is valuable for a more complete understanding of our method. Actually, **we have discussed the limitations of our current confounder modeling and causal intervention in Sec. A.1** (Supplemental Material). We plan to add such discussions to the main body of our paper.\n\nRegarding the confounders, **we have investigated the effects of different confounder modeling methods (see Tab. 6)**. In addition, we also add the comparisons of our K-means based confounders and random confounders, which clearly show that inaccurate confounders will lead to significant performance degradation.\n\n|Method |\tExclusive |\tCo-occur |\tAll |\n| --- | --- | --- | ---|\n|Random |\t22.1 |\t66.3 |\t56.1 |\n|K-means |\t29.7 |\t69.6 |\t60.6 |\n\n**(Weakness 3)** We agree with you that comprehensive ablation study is helpful for understanding the importance and effectiveness of each component of the proposed method. Actually, **we have presented ablation experiments to validate the effectiveness of each component of our method.** Please refer to Tab 3 and Sec. 5.3.1.\n\n**(Question 1)** Thanks for suggesting this reference paper. There are two core differences between our method and the method proposed in this paper. Firstly, the meaning of the nodes in the Structural Causal Model is different: we consider two different target objects as $X$ and $Y$, while the reference paper considers the target object as $X$ and model prediction as $Y$. Secondly, our high-level considerations in confounder modeling are different. The reference paper considers co-occurring objects as confounders. While in our paper, we argue that purely modeling confounders with object-level features is insufficient, and the confounders should also include background or text information. We will add discussions about our paper and this reference paper in our revised paper.\n\n**(Question 2)** Please refer to our response to weakness 1.\n\n**(Question 3)** Please refer to our response to weakness 2.\n\n**(Question 4)** **In Sec 4.3 (line 203~217) we discuss our understanding about confounder modeling in context of multi-label image recognition.** We argue that confounders should not only consider the objects with labels defined by the dataset, but also include other types of objects, image background and even image textures. Based on this understanding, we present to model the confounders by clustering the spatial features extracted by a pre-trained classification network, which may characterize confounders with object-level, texture-level and scene-level concepts. In Tab. 6, we investigate the impact of different confounder modeling methods, where our method achieves the best result.\n\n**(Question 5)** Our method aims to suppress spurious label correlations and enhance causal label correlation. Therefore, if the correlation between two labels are spurious, our method can always suppress their label correlations for multi-label image recognition. We would like to highlight that this does not mean that our method can always make right prediction, but means that it can always suppress spurious label correlations to reduce the probability of error prediction caused by it.\n\n**(Question 6)** How to handle noisy or incomplete labels is an important research topic in machine learning. However, this is not the focus of our paper, and our current method cannot address the partial label issue. Thanks for your suggestion, and we consider extending our method to the partial label setting as our future work.\n\n**(Question 7)** We would like to highlight that pure feature decoupling using the Transformer decoder cannot reduce contextual biases. To reduce contextual biases, we explicitly model causal label correlations based on the decoupled features and confounders, with the guidance of causal intervention in causal theory.\n\n**(Limitation)** On one hand, confounder modeling is a crucial step in our method, and bad choices may lead to obvious performance degradation.  But, on the other hand, we would like to highlight, it remains an open question about confounder modeling for visual recognition tasks. In this work, we present our understanding about confounders (see line 203~217), and then develop a simple but effective modeling approach based on pre-trained classification network and K-means clustering. In our ablation experiments (Tab. 6), we show the advantages of our confounder modeling approach.\n\nIn short, what we would say is that for this open question (confounder modeling), we have presented our rational analysis, as well as a simple approach with experiments validating its effectiveness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQGIgVoEyS", "reviewer": "Reviewer_BNe7", "summary": "This paper proposes a simple yet effective method based to address the issue of contextual bias for multi-label image recognition. It utilizes the casual intervention theory to pursue causal label correlations and suppress spurious label correlations. It utilizes the k-means to model the confounders, and employs the cross-attention mechanism to achieve the causal intervention. Experimental results demonstrate the efficacy of this approach.", "questions": "The authors should provide more detailed explanations and experiments about confounders.\n\nThe authors should provide a description of the inference process.\n\nThe authors should clarify how to obtain a prior of confounders.", "limitations": "The authors have addressed the limitation of this method. This method is dataset-dependent, and it can not determine the specific semantics of confounders, resulting in limited interpretability. Besides, it only considers the causal intervention, and does not consider causal reasoning.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and easy to understand.\n\nThis method seems easy to implement.\n\nThe approach achieves good results.\n\nThe problem is interesting in multi-label recognition tasks.", "weaknesses": "Why the k-means algorithm is used to build confounders, the author should give further explanation.\n\nIn the paper, the number of cluster centers is only calculated to 100, and what will happen if it continues to increase?\n\nRegarding inference time, how many forward passes does the method require?\n\nIn L191, how to obtain P(c) from the data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer BNe7 for the positive comments on our work. In the following, we present our responses addressing the raised concerns.\n\n**(Weakness 1)** In this work, we apply K-means clustering on the spatial features extracted from a pre-trained classification network for confounder modeling. This approach is based on our realization that the cofounders for recognizing certain object are often hard to define and enumerate – objects, scene, and even the texture of the environment are all potential confounders. On the other hand, a pre-trained classification CNN tends to activate the discriminative regions (which could be objects, scene and even the texture) in an image. Therefore, K-means clustering on CNN spatial features can produce a compact set of prototypes to represent potential confounders like objects, scenes and textures.\n\n**(Weakness 2)** Thanks for your suggestion. We conduct additional experiments regarding the number of the clustering centers, as illustrated in Fig. 2 in our attached pdf file. Our experiments show that 80 clusters are sufficient to represent potential confounders in our datasets, and 400 clusters does not bring significant accuracy gain. We point out that this conclusion still remains as an empirical observation. And, we hypothesize that for more complex multi-label image classification datasets (which might be collected in the future), further increasing the number of clusters may be necessary for modeling complex confounders and achieving higher accuracy.\n\n**(Weakness 3)** As the confounders are pre-computed, only one forward pass is required to obtain the final prediction.\n\n**(Weakness 4)** With K-means clustering, each spatial feature will be assigned to a certain cluster $c$. We calculate the number of spatial features for each cluster, divide it by the total number of spatial features, and then obtain $P(c)$ for each cluster.\n\n**(Question 1)** Thanks for your suggestion. We plan to add more explanations about confounders (e.g., our response to Weakness 1) and more experiments (e.g., other implementation choice for confounders like random vectors) in our revised version.\n\n**(Question 2)** Since the confounders (and the prior $P(c)$) are pre-computed, the inference process of the causal branch (formulated by Eq. 11) only requires one forward pass. As a result, the inference process of our whole pipeline (Eq. 2) also only requires one forward pass. We will further detail the inference process of our method in the revised version.\n\n**(Question 3)** Please refer to our response to Weakness 4.\n\n**(Limitations)** As described in Sec A.1, our current approach still has limitations in interpretability and higher-level causal modeling. We consider these as our future works, and will try our best to solve these limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful analysis and feedbacks, which are invaluable for understanding how to improve our paper.  We address the questions and concerns raised by each reviewer point-by-point in the respective threads below.  We also attach a PDF containing one updated Figure in response to Reviewer **NEe5,** and one additional Figure ****for Reviewer **BNe7.**

In general, reviewers agree that we present a novel effective pipeline that integrates causal label correlation modeling that can improve practical multi-label recognition where the training and test images may not follow the i.i.d assumption, as the co-occurence between objects might change in testing.  Specifically, we appreciate their assessment of our paper writing as **“well written with clear motivation and objectives”** (NEe5) and **“well-written and easy to understand”** (BNe7). The reviewers concur that **“Causal intervention is technically novel and well motivated in terms of multi-label image classification”** (NEe5). They give positive comments on our experiments that **“Experimental results are impressive, outperforming sub-optimal methods by a considerable margin. Ablation studies are also well-designed to showcase the contribution”** (NEe5), **“extensive experiments across multiple datasets, demonstrating the effectiveness of the proposed method. The results consistently show improvements over existing approaches, particularly in scenarios with contextual biases, underscoring the practical value of the method”** (wRCW), and **“This method seems easy to implement. The approach achieves good results. The problem is interesting in multi-label recognition tasks”** (BNe7).

Most of the reviewers expect more in-depth analysis and comparisons of the **confounder modeling** in our pipeline. Here, we would like te state the motivation, approach, validation, and limitation of our confounder modeling in detail.

**Motivation**: On one hand, confounder modeling is crucial for causal intervention, as evidenced by our new experiments that compare random confounders and our method (see our response to Weakness 5 for reviewer **NEe5**). But on the other hand, based on our survey, it remains an open question about cofounder modeling for visual recognition tasks. In this work, we present our understanding about cofounders modeling. That is, confounders for recognizing a certain object are often hard to define and enumerate – objects, scene, and even the texture of the environment are all potential confounders.

**Approach**: We model the confounders by clustering the spatial features extracted with a pre-trained classification network, as the classification network tends to activate object-level, texture-level, or scene-level semantic concepts [1]. By clustering spatial features with K-means, we obtain a compact set of prototypes to represent potential confounders like objects, textures and scenes. Our approach is significant different from previous works like VC R-CNN [2] that relies on bounding box annotations (which are often absent for image classification task), and only considers pre-defined objects as confounders.

**Validation**: In the main body of our paper, we validate the effectiveness of our confounder modeling approach in Tab. 6, and investigate the impact of the number of clustering centers in Tab. 4. Furthermore, we compare our confounder modeling approach with simple random confounders during rebuttal (response to Weakness 5 for reviewer **NEe5**).

**Limitation**: As described in Sec. A.1 in our Supplemental Material, our confounder modeling approach is dataset-dependent, although our cross-dataset multi-label image classification experiments can justify its generalization ability to a large extent. On the other hand, although our modeled confounders are often concerned with object-level, texture-level and scene-level concepts, it is difficult to determine the specific semantics of these concepts, resulting in limited interpretability. We leave better confounder modeling approach as our future work.

[1] Zhou, Bolei, et al. "Learning deep features for discriminative localization." CVPR. 2016.

[2] Wang, Tan, et al. "Visual commonsense R-CNN." CVPR. 2020.

Meta-review: After the rebuttal and the authors-reviewers discussion period, the paper got 2 borderline rejects and 1 strong accept. Overall, the reviewers think the proposed method is novel and interesting. The proposed method uses a causal intervention approach to identify the causal label correlations and suppress spurious label correlations. The rebuttal answered most of the concerns about the experimental model analysis. After reading the paper, the reviews, and the rebuttal, the AC recommends to accept this paper. The AC acknowledges that some parts of the experimental section could be improved by adding the new results from the rebuttal, but the overall technical contribution is above the acceptance bar.  The AC strong encourages the authors to update the paper by adding the key results from the rebuttal.
Decision: Accept (poster)