In Pursuit of Causal Label Correlations for Multi-label Image Recognition

Authors: zhao-min  chen
jin  xin
yisu  ge
sixian  chan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 08:07:01

Abstract: Multi-label image recognition aims to predict all objects present in an input image. A common belief is that modeling the correlations between objects is beneficial for multi-label recognition. However, this belief has been recently challenged as label correlations may mislead the classifier in testing, due to the possible contextual bias in training. Accordingly, a few of recent works not only discarded label correlation modeling, but also advocated to remove contextual information for multi-label image recognition. This work explicitly explores label correlations for multi-label image recognition based on a principled causal intervention approach. With causal intervention, we pursue causal label correlations and suppress spurious label correlations, as the former tend to convey useful contextual cues while the later may mislead the classifier. Specifically, we decouple label-specific features with a Transformer decoder attached to the backbone network, and model the confounders which may give rise to spurious correlations by clustering spatial features of all training images. Based on label-specific features and confounders, we employ a cross-attention module to implement causal intervention, quantifying the causal correlations from all object categories to each predicted object category. Finally, we obtain image labels by combining the predictions from decoupled features and causal label correlations. Extensive experiments clearly validate the effectiveness of our approach for multi-label image recognition in both common and cross-dataset settings.

Main paper: 
 introduction: 
 Multi-label image recognition is a fundamental task in computer vision, aiming to predict all objects present in an image. It has widespread applications including object detection [6] , medical imaging [39] , and person re-identification [31] . However, this task is challenging as the combinations of labels can be tremendous. Modelling label correlations to reduce the search space is believed to be essential for multi-label image recognition [4] .In the research of multi-label image recognition, an implicit yet common assumption is: the training and test sets follow independent and identically distributions (i.i.d.), and the label correlations are consistent. Under this setting, a deep backbone network can implicitly extract context-aware features that are beneficial for object recognition, and furthermore, explicit label correlation modeling can explore contextual cues more deeply to improve the recognition accuracy. Technically, graph structures [4, 12] or attention mechanisms [10, 25] have been successfully employed to model label Figure 1 : Illustration of the concept and effect of contextual bias in training. It is common that "Person", "Dog", and "Cat" co-occur in training images (we only show one image), while the test image may only contain "Person" and "Dog". Excessive reliance on the label co-occurrence in the training set may lead the recognition model to predict the "Cat" solely based on the presence of "Person" and "Dog". correlations. However, these methods may fall short when there exists contextual bias in the training set. As illustrated by Fig. 1 , "Person", "Dog", and "Cat" co-occur frequently in the training set, but a test image may only contain "Person" and "Dog". Consequently, due to the learned label correlations, the multi-label recognition model predicts a high probability of "Cat", solely based on the presence of "Person" and "Dog".Recently, a few researchers [14, 17] have uncovered the contextual bias issue, and discarded explicit label correlation modeling. Furthermore, they attempted to alleviate the effects of contextual bias by decorrelating the feature representations of a category from its co-occurring context [14] , or removing the contextual bias in features with causal mechanisms [17] . Despite improved accuracy, these methods neglect to model label correlations. In this paper, we ask: is it possible to model label correlations for multi-label recognition, with the purpose of preserving good and suppressing bad contextual contents? This work attempts to answer this question, as well as quantifying the goodness of contexts (corresponding to all pre-defined categories in the dataset) for recognizing a certain semantic category from the perspective of causal theory [22] .In this paper, we explore label correlations with a principled causal intervention approach for multilabel image recognition. Causal intervention aims to measure the causal effect from one cause variable to another effect variable, by "physically" putting the effect variable at any context to remove the effects of confounders in current image. Our key motivation lies on the realization that the causal label correlations (in the probability-raising sense [22] ) are stable in both training and testing, even when contextual bias does exist in the training set. We pursue causal correlations (e.g., from "Person" to "Clothes") to mine contextual cues for recognition, while suppressing spurious correlations (e.g., from "Person" to "Cat") which are associated by confounders (e.g., the overall scene) and may mislead the classifier in testing.We design an end-to-end framework that carefully integrates causal intervention into multi-label recognition. Specifically, we decouple label-specific features with a Transformer decoder attached to the backbone network, and model the confounders (imaginary contextual contents) by clustering spatial features of all training images. Based on label-specific features and confounders, we employ a cross-attention module to implement causal intervention for all pre-defined categories, quantifying the causal label correlations from all object categories to each predicted object category. Finally, we combine the predictions from decoupled features and causal correlations for multi-label prediction.In summary, our main contributions are as follows:• We propose a conceptually simple, yet effective label correlation modeling approach based on causal intervention to tackle the issue of contextual bias for multi-label image recognition. It allows us to capture causal label correlations (good contextual contents) to improve recognition accuracy, while suppressing the effects of spurious correlations (possible bad contextual contents) that may mislead the classifier. • We conduct comprehensive experiments with contextual bias to evaluate the effectiveness of the proposed method for multi-label image recognition. Under both common and crossdataset settings, our method consistently demonstrates advantages over existing methods. Recurrent Neural Networks (RNNs) [32, 21] , but the performance is affected by the order in which the labels are predicted. To overcome the sequential issue, researchers attempted to capture label correlations with graph structures, enabling simultaneous prediction of the label sets [4, 28, 12, 25] .Besides graph structures, several other approaches have been developed to establish label correlations. DER [5] employs metric learning to pull related label-specific features closer and push unrelated label features apart. C-Tran [15] introduces a label mask training approach for general multi-label image classification, indirectly constructing label correlations by predicting masked labels. Q2L [19] employs Transformers to decompose label features and utilizes self-attention mechanisms to establish label correlations. SST [7] employs Transformers to simultaneously capture both spatial and semantic label correlations.From a broader view of context modeling, label correlation modeling, arguably, can be understood as a strategy to enhance label-specific contextual cues. However, contextual information is a double-edged sword, and may mislead the classifier in presence of contextual bias in training.Contextual bias and debiasing. While visual context is widely believed to be beneficial for object recognition, recent works show that contextual bias may hurt multi-label image recognition [14, 17] . Such bias happens when an object category frequently co-occurs with some other object categories. Strongly relying on context may mislead the classifier, when typical contextual patterns around an object are absent or an object are absent from its typical context.Due to the contextual bias issue, some recent works have discarded label correlation modeling for multi-label image recognition, since the contextual priors encoded by learnt label correlations may mislead recognition. Furthermore, these works developed contextual debiasing techniques, by decorrelating feature representations of a category from its co-occurring context [14] , or removing contextual bias in features with causal mechanisms [17] . We argue that these methods may discard useful contextual cues, leading to inferior accuracy for common objects in common contexts.Causal intervention in vision. Causal intervention measures the causal relationship between two random variables, by removing the confounders that may associate them [22] . Recently, causal intervention has gained attention in the field of computer vision [17, 26, 33, 34, 27, 36, 37] , with expectations to address the contextual bias or long-tailed distribution issues. For instance, VC-RCNN [33] employs causal intervention and proxy tasks to extract unbiased visual features, which can benefit downstream tasks like Image Captioning, Visual Question Answering, and Visual Commonsense Reasoning. Tang et al. [26] show that the SGD momentum is essentially a confounder in long-tailed classification, and propose to remove bad causal effects by intervention. Wang et al. [34] reveal that traditional attention module is biased in out-of-distribution setting, and propose causal attention for unbiased visual recognition.For multi-label image recognition, Liu et al. [17] recently propose to remove the contextual bias in features with causal intervention. However, this approach does not consider label correlations, and may discard contextual evidences that are crucial for recognizing obscure instances.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Lines 7 to 11 as well as lines 47 to 55 elaborate on the contributions of our paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Due to the page limitations, we provide a detailed analysis of the limitations of our method in the supplementary materials.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries and motivation of causal correlations: 
 Causal correlations in probability-raising sense. Humans can easily understand the causal correlation between the presence of two objects (e.g., "Person" is the cause of "Clothes"). Seeking for a calculable formal definition, we follow the statement in [23] : if P (Y |do(X)) > P (Y ), then a causal correlation exists from X to Y in a probability-raising sense. Here do(X) is the do-operation, which pursues the causality between the cause X and the effect Y without the confounding effect.As shown in Fig. 2 , our implementation of causal intervention (which will be elaborated later) can reveal the causal correlations between objects in above probability-raising sense. For example, given the observation of "Person", the probabilities of "Clothes" and "Skis" raise. But on the other hand, although "Cat" often co-occurs with "Person", with causal intervention the probability of "Cat" even decreases when observing "Person". Formally, Fig. 3 shows a Structural Causal Model [23] , where X and Y represent two labels, and C represents the confounder set. In causal theory, each directed edge denotes a possible causal relationship between two nodes. Fig. 3 (a) and (b) illustrate two extreme cases: the causal correlation between label X and Y which is not affected by the confounder C, and the spurious correlation, where the co-occurrence of X and Y is caused by the confounder set C.WhyCausal intervention by backdoor adjustment. As "physical" intervention that puts Y at any context is almost impossible, backdoor adjustment [22] is typically applied for "virtual" intervention:EQUATIONHere the key idea is to cut off the link from confounder C to cause X, and stratify C into pieces C = {c}, making C no longer correlated with X, and making X have a fair opportunity to incorporate every confounder c into the prediction of Y , subject to a prior P (c).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will release the code after the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
overview of proposed pipeline: 
 Building upon above analysis, we incorporate causal intervention into explicit label correlation modeling for multi-label image recognition, designing a pipeline of two complementary branches (Fig. 4 ): the branch of decoupled label-specific features, and the branch of causal label correlations.In particular, the causal correlation branch is built upon decoupled features.Given an input image, a backbone network (e.g., ResNet-50 [11] ) is firstly employed to extract the spatial feature. Then, a Transformer decoder is leveraged to decouple label-specific features from the spatial feature. This branch predicts image labels based on objects themselves, rather than the context or label correlations. To take into account the label correlations yet overcoming the effects of contextual bias, we construct a causal intervention branch, which explicitly models causal label correlations, and integrates them into prediction.Formally, we denote the prediction confidence from the causal intervention branch as ŷcausal , and the prediction from the decoupled feature branch as ŷdecouple , then the final prediction confidence Person Dog Surfboard < l a t e x i t s h a 1 _ b a s e 6 4 = " A 5 P 7 scores ŷ can be written as:6 o z M 2 Z 9 h D Q s K W v E h F E w K g / 8 = " > A A A C D X i c b V C 7 T s M w F H V 4 l v I K M L J E F C S m K k G 8 x g o W x i L R h 9 R E l e P e t F a d h + w b R B X l B 1 j 4 F R Y G E G J l Z + N v c N s M 0 H I k y 0 f n 3 H v t e / x E c I W 2 / W 0 s L C 4 t r 6 y W 1 s r r G 5 t b 2 + b O b l P F q W T Q Y L G I Z d u n C g S P o I E c B b Q T C T T 0 B b T 8 4 f X Y b 9 2 D V D y O 7 n C U g B f S f s Q D z i h q q W s e u g O K m Y v w g H 4 w v T l m o z z P u 1 k P W J w m A v K u W b G r 9 g T W P H E K U i E F 6 l 3 z y + 3 F L A 0 h Q i a o U h 3 H T t D L q E T O 9 L y y m y p I K B v S P n Q 0 j W g I y s s m 2 + T W k V Z 6 V h B L f S K 0 J u r v j o y G S o 1 C X 1 e G F A d q 1 h u L / 3 m d F I N L L + N R k i J E b P p Q k A o L Y 2 s c j d X j E h i K k S a U S a 7 / a r E B l Z S h D r C s Q 3 B m V 5 4 n z Z O q c 1 4 9 u z 2 t 1 K 6 K O E p k n x y Q Y + K Q C 1 I j N 6 R O G o S R R / J M XZ i H y 8 G I = " > A A A C C 3 i c b V D J S g N B E O 1 x j X G L e v T S J A i e w o y 4 H Y N e P E Y w C 2 T C U N P p S Z r 0 L H T X i G G Y u x d / x Y s H R b z 6 A 9 7 8 G z v L Q R M f F P V 4 r 4 r u e n 4 i h U b b / r a W l l d W 1 9 Y L G 8 X N r e 2 d 3 d L e f l P H q W K 8 w W I Z q 7 Y P m k s R 8 Q Y K l L y d K A 6 h L 3 n L H 1 6 P / d Y 9 V 1 r E 0 R 2 O E t 4 N o R + J Q D B A I 3 m l s j s A z F z k D + g H 0 y 4 w G + V 5 7 m U M U g 0 y 9 0 o V u 2 p P Q B e J M y M V M k P d K 3 2 5 v Z i l I Y + Q S d C 6 4 9 g J d j N Q K J j k e d F N N U + A D a H P O 4 Z G E H L d z S a 3 5 P T I K D 0 a x M p U h H S i / t 7 I I N R 6 F P p m M g Q c 6 H l v L P 7 n d V I M L r u Z i J I U e c S m D w W p p B j T c T C 0 J x R n K E e G A F P C / J W y A S h g a O I r m h C c + Z M X S f O k 6 p x X z 2 5 P K 7 W r W R w F c k j K 5 J g 4 5 I L U y A 2 p k w Z h 5 J E 8 k 1 f y Z j 1 Z L 9 a 7 9 T E d X b J m O w f k D 6 z P H 6 g N n L o = < / l a t e x i t > ŷ causal < l a t e x i t s h a 1 _ b a s e 6 4 = " V c + J 0 Y l C S U / C F h + i s E B Y M K w j U H o = " > A A A B / H i c b V D L S s N A F J 3 4 r P U V 7 d L N Y B F c l U R 8 L Y t u X L Z g H 9 C G M p l O 2 q G T S Z i 5 E U u I v + L G h S J u / R B 3 / o 3 T N A t t P X C 5 h 3 P u Z e 4 c P x Z c g + N 8 W y u r a + s b m 6 W t 8 v b O 7 t 6 + f X D Y 1 l G i K G v R S E S q 6 x P N B J e s B R w E 6 8 a K k d A X r O N P b m d + 5 4 E p z S N 5 D 9 O Y e S E Z S R 5 w S s B I A 7 v S B / Y I f p D m n U P a z L K B X X V q T g 6 8 T N y C V F G B x s D + 6 g 8 j m o R M A h V E 6 5 7 r x O C l R A G n g m X l f q J Z T O i E j F j P U E l C p r 0 0 P z 7 D J 0 Y Z 4 i B S p i T g X P 2 9 k Z J Q 6 2 n o m 8 m Q w F g v e j P x P 6 + X Q H D t p V z G C T B J 5 w 8 F i c A Q 4 V k S e M g V o y C m h h C q u L k V 0 z F R h I L J q 2 x C c B e / v E z a Z z XEQUATION)where N is the number of categories. We utilize standard multi-label recognition loss to train the model, which can be written as:EQUATIONwhere y gt = {0, 1} N is ground truth label vector of the input image.In the following, we will detail the designs of the decoupled feature branch and the causal intervention branch. We will also present explanations and discussions about causal correlations in context to facilitate understanding.We firstly decouple label-specific features for input image with two purposes: (i) predicting image labels based on objects themselves; (ii) preparing for label correlation modeling by causal intervention.For decoupling label-specific features, common approaches include Class Activation Mapping (CAM) [40] and Transformer [30] . We employ a Transformer decoder for this purpose. Specifically, given input image I, we firstly use a CNN backbone to extract spatial feature F :EQUATIONThen, we employ a standard Transformer decoder to decouple label-specific featuresEQUATIONHere, Q ∈ R N ×D are learnable label embedding as queries, N and D are the number of categories and dimensionality of spatial features, respectively.Finally, we obtain the prediction confidence ŷdecouple from decoupled features by:EQUATIONwhere f f c1 (•) denotes the fully-connected layer, σ(•) is the sigmoid function. We apply multi-label loss to this branch for decoupled label-specific feature learning.With label-specific features, we can construct our intervention branch which explicitly models causal label correlations for multi-label image recognition.To estimate the probability of each category Y j on this causal intervention branch, a straightforward approach is firstly calculating P (Y j |do(X i )) for each category X i , and then merging them:EQUATIONOne might expect complex modeling of P (Y j |do(X i )) and f merge (•) for calculating Eq. 7. In this work, we introduce a simple yet effective implementation, using one cross-attention layer and one fully-connected layer.Causal intervention based on label-specific features. We hypothesize that whether X i appears in the test image should be encoded by its label-specific feature x i . Given label-specific features x i and y j for label X i and Y j , and one potential confounder feature c, we model the conditional likelihood in Eq.1 as:EQUATIONwhere σ(•) is the sigmoid function. f yj (•) calculates the logit for label Y j . Then, causal intervention can be calculated as:EQUATIONDue to the difficulty in directly computing Eq. 9, we apply the Normalized Weighted Geometric Mean (NWGM) [13] to approximate the above equation:EQUATION)Here P (c) is the prior of confounder, which can be obtained from data.Calcualting Eq. 10 requires further modeling of f yj (x i , c). However, instead of hypothesizing a formulation of f yj (x i , c) which only intervenes one category X i for Y j , we describe an efficient formulation based on cross attention to intervene all categories X i for Y j in one step. This allows us to circumvent the calculation of each f yj (x i , c), making the calculation of Eq. 7 more efficient.Effective modeling for all f yj (x i , c) by cross-attention. Formally, following the notation in Eq. 5, let X = [x 1 , ..., x N ] ∈ R N ×D denote all label-specific features. To implement Eq. 7, we seek for a model to combine the information of all label-specific features x i and one potential confounder feature c to predict the logit of label Y j . We employ cross-attention mechanism and fully-connected layer for this purpose:EQUATIONwhere Z c is the addition-based combination of all label-specific features X and a confounder feature c, and f f c2 is a fully-connected layer applied upon the cross-attention feature to obtain the logit. Algorithm 1 provides the pseudocode of causal intervention process.It remains an open question about cofounder modeling for visual recognition tasks. In VC R-CNN [33] , the authors treated objects as confounders, and extract objectlevel features based on bounding box annotations. However, on one hand, there is no bounding box annotation in the typical setting for multi-label image recognition. On the other hand, we argue that cofounders for recognizing certain object are often hard to define and enumerate -objects, scene, and even the texture of the environment are all potential confounders. For example, suppose that an image contains two objects: "Person" and "Surfboard", with the scene being the "Beach". In VC R-CNN, the "Surfboard" is considered the confounders for the "Person". However, in our opinion, the true confounder should the "Beach", although it does not have an associated image label.Based on above analysis, to characterize these non-enumerable confounders, high-level spatial features of training images from pre-trained classification CNN provide a good choice, as semantic objects/regions are often activated in classification features. By clustering spatial features with K-means algorithm, we obtain a compact set of prototypes to represent potential confounders like objects, scenes and textures. We empirically show the effectiveness of this simple approach for modeling the confounders. Common Setting. The COCO-Stuff [1] and DeepFashion [20] datasets are experimented in common setting, where the training and test sets are from the same dataset. We strictly adhere to the evaluation setup employed in [14] , and report the performance under two different test distributions: "Exclusive" denotes virtual co-occurrence where labels appearing simultaneously in the training set do not co-occur in the test set, and "Co-occur" represents the objects co-occurring in both the training and test sets. "All" is the average performance of all categories. We report top-3 recall for DeepFashion and mAP for COCO-Stuff.Cross-dataset Setting. We consider a more challenging yet practical setting in real-world applications: the training and test sets are from different datasets, and may suffer from serious contextual bias issue. We simulate this setting by using MS-COCO [16] for training and NUS-WIDE [8] for testing, and vice versa. In particular, we select the same categories (14 common classes) from both the MS-COCO and NUS-WIDE datasets for experiments. We report the mean Average Precision (mAP) for all categories in this setting.Implementation Details. To fair comparison with previous methods, we employ ResNet-50 and ResNet-101 as backbones for the common setting and real-world setting, respectively. We utilize the ImageNet [9] for model parameter initialization. For the cross-dataset setting, the input images are randomly cropped to a resolution of 448 × 448, while for the common setting, the resolution is set to 224×224. To extract label-specific features, we employ a 2-layer Transformer decoder with 4 attention heads. For modeling confounders, we first train a baseline model with standard multi-label loss, then extract spatial features from all the images in the training dataset, and employ K-means algorithm (default number of clusters is set to 80) to cluster all pixel-level spatial features. The Adam optimizer is chosen for model optimization, with a weight decay of 2e -2 and (β 1 , β 2 ) = (0.9, 0.9999). The initial learning rate is set to 1e -4, and we employ a cyclic learning rate policy to train our model for 80 epochs. All of the experiments are run on a computer with an AMD EPYC 7542 32-Core processor, 256 GB main memory, and eight GTX-3090 GPUs. 
comparing to the state-of-the-arts: 
 Common Setting. In Table 1 , we report the performance on COCO-Stuff and DeepFashion datasets, where the baseline is the vanilla Resnet-50 with standard multi-label loss. Comparing with the baseline, we observed a significant improvement (55.0% v.s. 60.6% on "All" mAP). Furthermore, our proposed method outperforms all state-of-the-arts on these two benchmarks. For example, it obtains a +1.5% "Exclusive" and +3.6% "Co-occur" improvements over the feature-split [14] on COCO-Stuff dataset.We also observe that previous methods that directly build label correlations based on graph structures are affected by contextual bias in datasets. For example, ML-GCN [4] and SSGRL [28] both construct label graphs based on the training set and utilize the graph structure to capture correlations between Method MS-COCO → NUS-WIDE NUS-WIDE → MS-COCO ADD-GCN [12] 81.8 77.2 ML-GCN [4] 81.4 77.2 SSGRL [28] 80.2 76.1 C-Tran [15] 80.9 76.9 CCD [17] 81.9 78.3 Q2L [19] 82.1 78.6 IDA [18] 82.3 78.9 CAM-Based [14] 81.0 77.8 feature-split [14] 81.9 78.3 Baseline (R101) 81.1 77.1 Ours 83.2 80.2 labels. These two methods cannot achieve significant improvements over the baseline, and even exhibit noticeable performance degradation. We speculate that the graph structure constructed from the training set are not applicable to the test set in presence of contextual bias.Cross-dataset Setting. Table 2 reports the results on the MS-COCO and NUS-WIDE datasets, where the baseline is the vanilla Resnet-101 with standard multi-label loss. Similar to the common setting, the performance of previous methods is affected by contextual bias, and our method outperforms all other state-of-the-art methods. However, compared to the baseline, the performance improvement of our method is not as significant as in the common setting. We speculate that, apart from contextual bias in training, the inconsistency in data distribution may also affect the performance. Even so, the cross-dataset results can indicate robustness and generalization capabilities of our method.In this section, we conduct ablation studies by using ResNet-50 as backbone on COCO-Stuff Dataset.We investigate the impacts of key modules in our framework. Specifically, there are two essential modules, i.e., decoupling the label feature module (denoted as "Decouple") and causal intervention module (denoted as "Causal"). Table 3 shows the mAP performance by progressively integrating the above two modules. Solely applying "Decouple" on the backbone gives a +1.8% "All" mAP, +0.2% "Exclusive" mAP and +1.5% "Co-occur" mAP improvement. Directly applying decoupling leads to improved "Co-occur" performance but fails to enhance "Exclusive" performance. We speculate that the cross-attention mechanism can capture the long-range dependencies, since it can implicitly model causal correlations but cannot eliminate spurious co-occurrence correlations. Then, the causal intervention module, which can remove spurious co-occurrence correlations and capture the causal correlations, bring another 3.8% "All" mAP. These results show the effectiveness of our approach in alleviating label correlations bias.Modeling confounders is a core step in our approach. In order to investigate the impact of different confounders, we designed comparative experiments at three aspects, i.e., the number of clustering centers, different clustering features, and different approaches for modeling confounders.The number of clustering centers. In order to investigate the influence of the number of clustering centers, we conducted experiments with different numbers of cluster centers: 20, 40, 60, 80, and 100, respectively. The experimental results are presented in the Table 4 . It can be observed that within a certain range, increasing the number of cluster centers does not significantly affect performance. However, a performance decline is noticeable when the number of clusters is reduced to 20. We speculate that the confounders correspond to many attributes including semantic, color, texture, and so on. A smaller number of cluster centers can only express very limited attributes, resulting in a decline in performance.Different backbones for clustering. By default, we employ ResNet-50 as the backbone to extract features from all images in the training set on COCO-Stuff, and perform clustering to obtain confounders. In order to investigate the impact of different clustering features, we utilize distinct backbones for feature extraction. As shown in Table 5 , the confounders obtained from different backbones have not significant influence on final performance. The role of the confounders is to identify label causal correlations and does not directly participate in recognition. Therefore, the features extracted by a weak backbone seems sufficient in identifying causal correlations, and the use of a strong backbone does not lead to performance improvement.Different approaches for modeling confounders. In this paper, we utilize clustering centers to characterize confounders. We compare it with two additional modeling techniques. The experimental results as shown in the Table 6 . "Random" means using random vectors to replace cluster centers as confounders. "Early" means fusing features from early epochs directly to obtain the confounder, which is adopted by [17] . "Label" means directly using label-specific features as the confounder.Our modeling of confounders using K-means yields the best results. "Random" leads to significant performance degradation (measured by mAP), especially on the "Exclusive" subset. Since, we believe that confounders should be modeled at semantic level. The "Early" approach relies solely on simple feature fusion, which fails to effectively differentiate various attributes. The "Label" approach only employs object semantics as the confounder, leading to the omission of other attributes. By contrast, our approach incorporates both object-related information and other contextual information, offering a more expressive set of confounders.We investigate different implementations of Eq. 7. Following [33] , we employ a simpler linear approach and average merge method to model Eq. 7. As shown in Table 7 , we can observe that employing a linear modeling approach yields competitive results, but our method still outperforms it. We speculate that while linear modeling can implement Eq. 7, our approach utilizes cross-attention, which possesses the ability to model long-range dependencies, thereby efficiently capturing causal correlations and suppressing spurious correlations. 
conclusions: 
 In this paper, we presented a principled approach to address the contextual bias issue for multi-label image recognition. Using causal intervention from causal theory, we pursued causal label correlations, and integrated them into multi-label prediction. We evaluate the effectiveness of our approach with both quantitative and qualitative assessments. In the future, we will investigate more advanced structural causal models for better describing the label correlations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Section 5.1 provide all experimental settings and implomentation details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The dataset partition and random seed are fixed, thus eliminating the possibility of error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Section 5.1 provide the information of computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper solely utilizes causal theory to establish causal relationships between image labels, hence it may not have any societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not have such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: This paper has appropriately referenced the original assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Despite improved accuracy, our causal intervention-based multi-label image recognition algorithm still has several limitations. First, we model confounders by clustering the spatial features extracted from a pre-trained classification CNN. On one hand, this approach is dataset-dependent, and on the other hand, although our modeled confounders mainly concerne with object-level, texture-level and scene-level concepts, it is not possible to determine the specific semantics of these concepts, resulting in limited interpretability. Second, we only integrate the causal intervention technique in causal theory in our multi-label image recognition pipeline. In the future, we will consider higher-level causality such as counterfactual reasoning.A. We also evaluate our method in more challenging cross-dataset setting: training on MS-COCO [16] , but testing on NUS-WIDE [8] dataset. The MS-COCO dataset contains 122,218 images and covers 80 common objects, with an average of 2.9 labels per image. The NUS-WIDE dataset has 269,648 images and 81 concepts, with an average of 2.4 concept labels per image.In our main paper, we present cross-dataset experiments to show the effectiveness of our method for contextual bias and distribution shift. That is, we train on MS-COCO [16] , but test on NUS-WIDE [8] ; or train on NUS-WIDE [8] , but test on MS-COCO [16] . Here, we also report the accuracy of our method for general multi-label recognition, by both training and testing on MS-COCO.As shown in Tab.8, although our approach is designed to solve the contextual bias issue for multi-label recognition, it can achieve competitive accuracy for general multi-label recognition. In particular, our method outperform CCD [17] by 0.9 mAP, which also integrates causal mechanisms for contextual de-biasing, but discards the explicit modeling for label correlations. Combining the cross-dataset and intra-dataset experimental results, our approach shows advantages over both general multi-label image recognition algorithms [7, 19] and recent algorithms that explicitly considers contextual de-biasing [14, 17] .To investigate the impact of different backbones on performance, we conducted experiments using ResNet-50, ResNet-101, and BEIT-Large [35] as backbones. The results are shown in Table 9 , which reveal that even with a stronger backbone, our method can still effectively improve performance. For example, our method with the BEiT-Large backbone achieves a 72.2% mAP, which outperforms the baseline by 2.9% mAP. This experimental result demonstrates that our method can be generalized to stronger backbones.For qualitative verification, we employ Grad-CAM [24] to visualize label-specific features for the baseline and our proposed method. As shown in Fig. 5 , although the baseline can activate relevant object locations, it also activates the regions with spurious correlations for the target object. In contrast, our method activates solely at the object's location. In Fig. 5 (a), besides activating label "Cell Phone", the baseline method additionally activates label "Person", whereas our method solely activates "cell phone." Regarding negative labels, when an input image lacks a specific label, the baseline might erroneously infer false labels through other labels due to excessive reliance on label co-occurrences. Our method mitigates this phenomenon effectively. In Fig. 5 , as labels Table 8 : Comparisons with state-of-the-art on MS-COCO [16] , where the models are all trained on MS-COCO dataset. Here, * means the re-produced results by ourselves.Methods Resolution mAP CNN-RNN [32] 224×224 61.2 SRN [41] 224×224 77.1 ResNet-101* [11] 448×448 79.1 SSGRL* [28] 448×448 81.9 DER [5] 448×448 82.8 ADD-GCN* [12] 448×448 82.8 ML-GCN [4] 448×448 83.0 C-Tran* [15] 448×448 83.1 P-GCN [3] 448×448 83.2 MCAR [10] 448×448 83.8 MS-CMA [25] 448×448 83.8 CCD [17] 448×448 84.0 SST [7] 448×448 84.2 TDRG [38] 448×448 84.6 IDA [18] 448×448 84.8 Q2L [19] 448×448 84.9 Ours 448×448 84.9 "Person", "Baseball Glove", and "Baseball Bat" co-occur in the training set, the baseline infers the presence of "Baseball Bat" through "Person" and "Baseball Glove", even if it is absent in the input image. This eventually leads the baseline to incorrectly activate spatial features associated with the label "Baseball Bat". In contrast, our method nearly avoids activating such features. The visualization results further validate our motivation, i.e., our approach effectively removes spurious co-occurrence correlations and captures the causal correlations. In Fig. 6 , we show the t-SNE [29] visualization of our confounders and two typical label-specific features ("Baseball Glove" and "Person").Without causal intervention, we observe that "Baseball Glove" and "Person" are mixed with each other, indicating that they often co-occur in the training set. Furthermore, the confounders are around With our causal intervention branch, we empirically find that "Baseball Glove" and "Person" are pulled apart, since they have very weak causal correlations in the probability-raising sense. On the other hand, the label-specific features are pulled away from the confounders, suggesting that our causal intervention can remove the influence of confounders for causal representation learning.NeurIPS Paper ChecklistJustification: We have reviewed the NeurIPS Code of Ethics, and this paper conforms it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: After the rebuttal and the authors-reviewers discussion period, the paper got 2 borderline rejects and 1 strong accept. Overall, the reviewers think the proposed method is novel and interesting. The proposed method uses a causal intervention approach to identify the causal label correlations and suppress spurious label correlations. The rebuttal answered most of the concerns about the experimental model analysis. After reading the paper, the reviews, and the rebuttal, the AC recommends to accept this paper. The AC acknowledges that some parts of the experimental section could be improved by adding the new results from the rebuttal, but the overall technical contribution is above the acceptance bar.  The AC strong encourages the authors to update the paper by adding the key results from the rebuttal.

Reviews: review: paper_summary: paper_summary: This paper proposes a causal intervention mechanism for multi-label image classification, where causal label correlations are pursued and spurious label correlations are suppressed. To achieve this, the authors frame a pipeline consisting of a branch for decoupling label-specific features and a branch for summarizing causal label correlations. The results from both branches are combined for final predictions on image labels. Comparative experiments and ablation studies demonstrate the effectiveness of the proposed causal intervention mechanism.
strength_weakness: strength_weakness: - Line 175: 'Correaltions' -> 'Correlations'.
- Line 237: 'Transformer encoder' -> 'Transformer decoder'.
- $f_{fc}$ in Eq.6 and $f_{fc}$ in Eq.11 should be different if their parameters are not shared.
- In Figure 4, in the causal label correlation branch, the confounder features are added into label-specific features. However, the outputs are not seen to be used in subsequent steps, and it seems that only the label-specific features are utilized for causal intervention. The diagram of this module needs to be improved.
- More experimental evidence should be provided to verify the effectiveness of the confounder modeling. For example: using random vectors to replace cluster centers as confounders. Only the feature visualization and ablation study on clustering center number are unconvincing.
- Although this paper is well motivated, the modeling process, especially Equation 11, is confusing.
questions: questions: - How is the operation $f_{merge}$ removed from the second line of Eq.11?  Why is the summation over all confounders $c$ also removed? Even if it can be removed, which confounder does $c$ in the last line of the formula refer to? The Eq.11 is confusing and needs further clarification. It would be best if the authors add a pseudocode to illustrate this process.z
- Are the results in Table 1 and Table 2 reported by re-training these models on the relevant datasets? If so, the authors should clarify the experimental details for fair comparison.
- According to the Table 8, in terms of the intra-dataset comparisons on MS-COCO, Q2L achieves the same performance in mAP as the proposed method. However, Q2L only requires a Transformer decoder for decoupling label-specific features. Therefore, we question the generalizability of the proposed causal intervention mechanism on multi-label image classification task, wondering whether it is only effective on specific datasets.
limitations: limitations: See Weakness and Questions.
paper_summary: paper_summary: This paper presents a novel approach to addressing label correlations in multi-label image recognition by using causal intervention. The method involves decoupling features, modeling confounders, and implementing causal interventions to capture useful contextual information while suppressing spurious label correlations. This approach is highly innovative and has significant potential applications.
strength_weakness: strength_weakness: 1. **Lack of Hyperparameter Analysis:**
   The paper does not provide a detailed analysis of the hyperparameters involved in the proposed method, such as the number of clusters for confounders or the parameters of the cross-attention module. A sensitivity analysis of these hyperparameters would be beneficial to understand their impact on model performance and to guide practitioners in tuning the model effectively.

2. **Insufficient Discussion on Method Limitations:**
   The paper lacks a thorough discussion on the limitations of the proposed method. It would be valuable to include an analysis of scenarios where the method might not perform well, such as when the selection of confounders is inaccurate or when the causal relationships between labels are weak. Addressing these limitations can provide a more balanced view of the method's applicability and robustness.

3. **Limited Ablation Studies:**
   Although the paper includes some ablation studies, the number and depth of these experiments are not comprehensive enough. More detailed ablation studies are needed to analyze the independent contribution of each module (e.g., feature decoupling, confounder modeling, and causal intervention) to the overall performance. This would help in understanding the importance and effectiveness of each component of the proposed method.
questions: questions: 1.  I noticed another paper titled "Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training" that also employs causal inference to address multi-label image classification. The methods in these papers differ in implementation and theoretical basis. Could you further elaborate on the main differences and advantages of your approach compared to this work?
2.  Your paper does not provide a detailed analysis of the hyperparameters involved in the proposed method. Could you explain the rationale behind the chosen hyperparameters and their impact on the model's performance?
3. There is a lack of discussion on the limitations of your proposed method. In what scenarios might your method underperform, and how could future work address these limitations?
4. Could you explain why certain confounders were selected for modeling in your approach? How does the choice of confounders impact the effectiveness of causal intervention in your model?
5. How does your method handle cases where the causal relationships between labels are weak or not well-defined? Does this affect the model's accuracy, and if so, how?
6. How does your approach ensure robustness against noise and variability in the data? Are there any specific strategies employed to handle noisy or incomplete labels?
7. Could you provide more details on how the feature decoupling using the Transformer decoder specifically contributes to reducing contextual biases in multi-label image recognition?
limitations: limitations: The effectiveness of the proposed method relies heavily on accurately modeling the confounders. If the selection of confounders is not precise or representative of the underlying data distribution, the causal intervention may not effectively distinguish between useful contextual information and spurious correlations. This could potentially limit the method's performance in scenarios where confounder selection is challenging.
paper_summary: paper_summary: This paper proposes a simple yet effective method based to address the issue of contextual bias for multi-label image recognition. It utilizes the casual intervention theory to pursue causal label correlations and suppress spurious label correlations. It utilizes the k-means to model the confounders, and employs the cross-attention mechanism to achieve the causal intervention. Experimental results demonstrate the efficacy of this approach.
strength_weakness: strength_weakness: Why the k-means algorithm is used to build confounders, the author should give further explanation.

In the paper, the number of cluster centers is only calculated to 100, and what will happen if it continues to increase?

Regarding inference time, how many forward passes does the method require?

In L191, how to obtain P(c) from the data?
questions: questions: The authors should provide more detailed explanations and experiments about confounders.

The authors should provide a description of the inference process.

The authors should clarify how to obtain a prior of confounders.
limitations: limitations: The authors have addressed the limitation of this method. This method is dataset-dependent, and it can not determine the specific semantics of confounders, resulting in limited interpretability. Besides, it only considers the causal intervention, and does not consider causal reasoning.

Comments: title: Response to Reviewer BNe7
Respond to the authors
Response to Reviewer NEe5
Response to Author Rebuttal
Response to Reviewer NEe5
Response to Author Rebuttal
comment: We are thankful for your acceptance and constructive feedback.
The authors have addressed my concerns, I will raise my score correspondingly.
We sincerely thank you once again for the constructive comments and suggestions. Should our rebuttal effectively address the concerns, we kindly hope you can raise your score, which we believe is vital for the final decision on our work.
Thank you for your quick reply. All my concerns are addressed.
Thank you for your response. **We must emphasize that our approach does not simply add causal intervention to Q2L**. Q2L includes both a Transformer Decoder and a Transformer Encoder, and uses a specially proposed ASL Loss [1]. In contrast, without considering the causal intervention branch, we only utilize the Transformer Decoder with less parameters for feature decoupling, and train our model with general multi-label classification loss.

Furthermore, **Table 8 aims for performance comparison, rather than ablation study. It does not indicate that our causal intervention method gains no performance improvement on MS-COCO.** We conducted an ablation study on MS-COCO as shown in the table below (CMLL was not compared due to lack of experiments at resolution 448x448 and unavailability of code). It can be seen that our causal intervention module still provides some improvements on the MS-COCO dataset. Compared with Q2L, we achieve similar results with fewer parameters (refer to our rebuttal for Weakness 1 of Reviewer wRCW). Additionally, when compared with other methods specifically designed based on causal theory for the MS-COCO dataset, our method still achieves competitive results.

| Method | mAP |
| --- | --- |
| Res101 | 79.1 |
| Res101 + Decouple | 83.7 |
| Res101+Decopule+Causal (Ours) | 84.9 |
| Q2L | 84.9 |
| CCD | 84.0 |
| IDA | 84.8 |
| PAT | 85.0 |

We consider pursuing higher accuracy on MS-COCO as our future work, by combining our causal intervention method with more advanced backbone networks, multi-label loss, or pre-training methods.

[1] Asymmetric loss for multi-label classification, 2020.
Thank you very much for your thorough response to my comments and questions. Most of my doubts are well addressed. However, I still have concerns about the effectiveness of the proposed method, given the fact that the proposed method shares the same label-specific feature learning architecture as Q2L [1], but does not show any performance improvement on MS-COCO with the additional mechanism of causal intervention. In contrast, CCD [2], IDA [3], CMLL [4] and PAT [5] have all demonstrated the effectiveness of causal intervention on this dataset. The author's explanation is not convincing.


[1] Query2Label: A Simple Transformer Way to Multi-Label Classification (arXiv 2021);

[2] Contextual Debiasing for Visual Recognition with Causal Mechanisms (CVPR 2022);

[3] Causality Compensated Attention for Contextual Biased Visual Recognition (ICLR 2023);

[4] Causal multi-label learning for image classification (NN 2023);

[5] Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training (ICML 2024).
Dear Reviewer,

We hope that our rebuttal has effectively clarified your confusion and that the additional experiments we provided have strengthened the validity of our approach. We eagerly await your feedback on whether our response has adequately resolved your concerns, or if further clarification is needed.
Dear Reviewer,

We hope that our rebuttal has effectively clarified your confusion and that the additional experiments we provided have strengthened the validity of our approach. We eagerly await your feedback on whether our response has adequately resolved your concerns, or if further clarification is needed.