Causal Visual-semantic Correlation for Zero-shot Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 10:22:04

Abstract: Zero-Shot learning (ZSL) correlates visual samples and shared semantic information to transfer knowledge from seen classes to unseen classes. Existing methods typically establish visual-semantic correlation by aligning visual and semantic features, which are extracted from visual samples and semantic information, respectively. However, instance-level images, owing to singular observation perspectives and diverse individuals, cannot exactly match the comprehensive semantic information defined at the class level. Direct feature alignment imposes correlation between mismatched vision and semantics, resulting in spurious visual-semantic correlation. To address this, we propose a novel method termed Causal Visual-semantic Correlation (CVsC) to learn substantive visual-semantic correlation for ZSL. Specifically, we utilize a Visual Semantic Attention module to facilitate interaction between vision and semantics, thereby identifying attribute-related visual features. Furthermore, we design a Conditional Correlation Loss to properly utilize semantic information as supervision for establishing visual-semantic correlation. Moreover, we introduce counterfactual intervention applied to attribute-related visual features, and maximize their impact on semantic and target predictions to enhance substantive visual-semantic correlation. Extensive experiments conducted on three benchmark datasets (i.e., CUB, SUN, and AWA2) demonstrate that our CVSC outperforms existing state-of-the-art methods.

Main paper: 
 introduction: 
 Zero-Shot Learning (ZSL) stands as a significant research area in machine learning, which imitates human cognitive patterns to endow computers with ability to recognize new classes [20, 21, 30] . 1 Humans can identify new classes by leveraging prior knowledge, even without direct exposure. ZSL draws inspiration from human cognition, achieving knowledge transfer from seen to unseen classes by correlating visual samples with shared semantic information The ZSL model aims to establish substantive visual-semantic correlation within seen classes, facilitating accurate identification of inherent attributes in images. This enables accurate zero-shot predictions when encountering images from unseen classes by pricely identifying the genuine attributes. (b) Existing ZSL methods inevitably falls into spurious visual-semantic correlation, primarily caused by mismatches between images and their corresponding attribute annotations. Such discrepancy is due to the fact that diverse images within same class do not have the same attributes consistent with complete attribute annotations. Thus, the spurious visual-semantic correlation finally results in poor knowledge transfer.within seen class. Consequently, by solely relying on the semantic information of unseen classes, models can directly recognize samples from unseen classes. The semantic information typically includes attribute annotations [15, 16] , word vectors [23, 28] , document vectors [22, 39] and so on. In this paper, we focus on attribute annotations as the shared semantic information.Existing methods establish the visual-semantic correlation by aligning visual and semantic features, which are extracted from instance-level images and class-level attribute annotations. Specifically, these methods map visual and semantic features into common space and design various objective functions (such as mean squared loss [52] , cross-entropy loss [18] , contrastive loss [17] , etc.) to facilitate alignment between the two. This process ensures that images and their corresponding attribute annotations get similar representations in the common space, thus achieving visual-semantic correlation. Based on the mapping direction between visual and semantic space, existing methods are broadly divided into two categories: visual-to-semantic mapping methods [5, 25, 40, 54] and semantic-to-visual mapping methods [8, 47, 49] . The existing methods overcome the heterogeneity between vision and semantics, correlate the two and achieve knowledge transfer.However, the mismatch between vision and semantics leads to spurious visual-semantic correlation, limiting knowledge transfer, which has not been well addressed by existing works. As shown in Figure 1 (a), the model aims to establish substantive visual-semantic correlations within seen classes, enabling the identification of inherent attributes present in the images. This facilitates correct zero-shot predictions when encountering unseen class images by accurately identifying the attributes genuinely present. However, attribute annotations are typically comprehensive and defined at the class level, whereas instance-level images within same class exhibit different attributes due to varying individuals and observation perspectives. Consequently, vision and semantics cannot exactly match in practice. As illustrated in Figure 1 (b), the model easily falls into spurious visual-semantic correlation due to directly aligning mismatched vision and semantics. Such spurious correlation limits knowledge transfer and results in inferior predictive performance.Substantive visual-semantic correlation is crucial for ZSL. Inspired by the analysis above, we are prompted to utilize attribute annotations effectively to establish visual-semantic correlation, while enhancing the substantiveness of the correlation. Considering that images often contain fewer attributes than those annotated, we significantly penalize the model for predicting attributes beyond the annotations. At the same time, the model incurs minor penalties for missing attributes that present in the annotations but absent from the images, thus reasonably utilizing the attribute annotations. Furthermore, causal inference [34, 35] , as an effective methodology, excels at revealing the causal correlation between variables. Hence, we leverage the tool of causal inference, specifically counterfactual intervention, to assist the model in establishing substantive visualsemantic correlation. By applying counterfactual interventions to intermediate variable within the model's visual-semantic interaction and maximizing its impact, we can enhance the substantiveness of visual-semantic correlation.We propose Causal Visual-semantic Correlation (CVsC) to establish a substantive visual-semantic correlation for ZSL. Specifically, CVsC utilizes the Visual Semantic Attention module to facilitate interaction between vision and semantics, thereby identifying attribute-related visual features. Subsequently, a Semantic Embedding module maps these features to obtain semantic vector and determine the category. Furthermore, we design Conditional Correlation Loss to effectively utilize semantic information as supervision. Finally, we introduce counterfactual causal intervention applied to attribute-related visual features, and maximize their impact on semantic and target predictions to enhance substantive visual-semantic correlation. The comprehensive experimental demonstrates that CVsC effectively mitigates spurious visualsemantic correlation and significantly enhances the performance of ZSL.In summary, our contributions are as follows:• We explore the significance of visual-semantic correlation for ZSL and highlight that direct visual-semantic alignment leads to spurious visual-semantic correlation, thereby constraining knowledge transfer. • We design Causal Visual-semantic Correlation (CVsC) to establish substantive visual-semantic correlation.• Extensive experiments on multiple benchmark datasets demonstrate that CVsC enhances the substantiveness of visual-semantic correlation and significantly improves the performance of ZSL. 
related works 2.1 zero-shot learning: 
 Zero-Shot Learning (ZSL) [20, 30] , which is proposed to address the data dependency issue in machine learning, can directly identify unseen class samples. Based on the classes included in the testing phase, ZSL is typically divided into conventional settings and generalized settings [37] . In conventional ZSL (CZSL), testing samples only contain unseen classes, while in generalized ZSL (GZSL), testing samples come from both seen and unseen classes. Inspired by human cognition, ZSL transfers knowledge from seen to unseen classes by correlating visual samples with corresponding semantic information within seen classes [9, 42, 47, 48] . The correlation between vision and semantics is established by aligning visual and semantic features, which are extracted from instance-level images and class-level semantic information. Existing methods typically map images and semantic information into a common space and then design objective functions to achieve feature alignment. Based on the mapping direction, existing methods are mainly classified into two categories: visual-to-semantic mapping methods [3, 5, 25, 40, 54] and semantic-to-visual mapping methods [8, 42, 47, 49, 57] . These methods overcome the heterogeneity and correlate visual and semantic information, thereby achieving knowledge transfer. Despite significant efforts in this field, the challenge of spurious visual-semantic correlation continues to hamper ZSL performance. Since instance-level images cannot strictly match attribute annotations defined at the class level, the models easily fall into spurious visual-semantic correlation. To address this problem, we propose CVsC, which enhances the substantiveness of visual-semantic correlation by effectively utilizing attribute annotations and introducing causal inference to correlate vision and semantics.In ZSL, refining the association between visual and semantic features by aligning local visual features can lead to improved performance [6, 18] . Early explorations of local representations in ZSL relied on part detection methods [14, 50] , which utilize pre-trained part detectors for local region representation. However, such methods are limited by the additional and costly annotation data required by the part detectors. Attention mechanism, benefiting from its expertise in extracting discriminative local features, has been introduced into ZSL. [53] initially proposed a stacked semanticguided attention method, enabling the model to learn more discriminative local visual features. [58] introduced a semantic-guided multi-attention localization model, which can identify the most discriminative parts of objects without additional supervisory guidance. [18] decoupled class semantic vectors into multiple attribute vectors, and determined attribute-related visual features by computing attention maps between attribute vectors and local visual features. In this paper, we adopt the visual-semantic interaction approach from [18] to design our model. CVsC first takes ViT [13] and Glove [36] to extract regional visual features and attribute vectors, respectively. Next, the Semantic-Visual Attention module facilitates interaction between regional visual features and attribute vectors, identifying attribute-related visual features. Finally, the Semantic Embedding module maps semantic vectors, enabling computation of probabilities for unseen classes. To establish substantive visual-semantic correlation, CVsC employs Conditional Correlation Loss to properly utilize attribute annotations. Additionally, CVsC introduces causal interference, i.e. Semantic Causal Interference and Target Causal Interference, to strengthen the substantive correlation between vision and semantics.Causal inference [32, 34, 35, 41] investigates the effects of variables when some cause is changed, providing researchers with an effective tool to determine the substantive correlation between variables.In recent years, there has been a rapid growth of interest in combining deep learning with causal inference. It has been successfully applied in various fields, including explainable machine learning [29] , natural language processing [45, 51] , computer vision [27, 38, 43] , and others. The substantive visual-semantic correlation is crucial for ZSL. Therefore, in this paper, we introduce causal inference to ZSL to establish a substantive visual-semantic correlation. By imposing counterfactual interventions on attribute-related visual features learned by the model, we can observe their impact on the model. By maximizing the impact of counterfactual intervention, we enhance the effectiveness of attribute-related visual features, thereby establishing substantive visual-semantic correlation. 
methods: 
 In this section, we first revisit ZSL setting. Then, we introduce the detailed design of Causal Visual-semantic Correlation (CVsC), the flowchart of which is illustrated in Figure 2 .ZSL setting ZSL has two sets of classes, i.e. seen classes C 𝑠 and unseen classes C 𝑢 . The corresponding samples are denoted as 𝒙 ∈ X 𝑠 for seen classes and 𝒙 ∈ X 𝑢 for unseen classes, with labels 𝑦 ∈ Y 𝑠 for seen classes and 𝑦 ∈ Y 𝑢 for unseen classes. The classlevel attribute annotations are encoded to semantic vectors. Here, we denote the semantic vector of class 𝑐 as𝑺 𝑐 = 𝑠 𝑐 1 , . . . , 𝑠 𝑐 𝐾 ⊤ ,where 𝑠 𝑘 𝑐 represents the value of the 𝑘-th attribute in class 𝑐, and 𝐾 represents the total number of attributes. The class semantic vectors are available for all classes in both the training and testing phases. The training set D 𝑡𝑟𝑎𝑖𝑛 = {(𝒙, 𝑦)} consists of samples from seen classes and their corresponding labels. ZSL is categorized into two types: conventional ZSL (CZSL) and generalized ZSL (GZSL) based on the scope of the testing set. The goal of CZSL is to predict image labels from unseen classes (𝒙 ∈ X 𝑢 ) in the testing set D 𝑡𝑒𝑠𝑡 = {𝒙}, while GZSL aims to predict image labels from both seen and unseen classes(𝒙 ∈ X 𝑠 ∪ X 𝑢 in the testing set D 𝑡𝑒𝑠𝑡 = {𝒙}).CVsC takes image and attribute as the input, and employs Semantic Visual Attention to interact with vision and semantics, thereby obtaining attribute-related visual features. Subsequently, a Semantic Embedding module maps these features into the semantic space, and calculates the similarity with the class semantic vectors to obtain the class probabilities. In order to achieve substantive visualsemantic correlation, CVsC first employs Conditional Correlation Loss to reasonably utilize semantic information as supervision. Furthermore, we introduce causal interference into CVsC, i.e. Semantic Causal Interference and Target Causal Interference. By imposing counterfactual interventions on attribute-related visual features learned by the model, and maxing their impact, we can get more effective features and strengthen the substantive correlation between vision and semantics.As shown in Figure 2 , the Semantic Visual Attention takes the regional visual features and attribute vectors as input. In particular, The image 𝑥 is divided into 𝑅 patches, and features corresponding to these patches are extracted using the Vision Transformer (ViT) [13] . Here, we utilize the output of the last layer of ViT, removing the cls token, as the regional visual features, denoted as 𝑭 = [𝒇 1 , ..., 𝒇 𝑟 , ...𝒇 𝑅 ] 𝑇 . Simultaneously, we employ Glove [36] , the pre-trained language models, to extract attribute names as attribute vectors, denoted as 𝑨 = [𝒂 1 , ..., 𝒂 𝑘 , ..., 𝒂 𝐾 ] 𝑇 . Subsequently, we utilize attention mechanisms to facilitate interaction between vision and semantics. Here, we use 𝑄, 𝐾, and 𝑉 to respectively denote query, key, and value. They are defined as follows:EQUATIONBy taking the attribute as the query, we compute its correlation matrix with the key (i.e., regional visual feature 𝑭 ) to obtain the attribute attention map 𝑴, and scale it using SoftMax. Here, the attribute vectors 𝑨 are projected to ensure uniform dimensions for 𝑄, 𝐾, and 𝑉 , ensuring their dot product computation. Then, the attribute-related visual features 𝑭 𝐴 are obtained by multiplying the attribute attention map 𝑴 with the 𝑉 :𝑭 𝐴 = 𝑴𝑉 = softmax (𝑄𝐾) 𝑉 .(2)Next, we employ the Semantic Embedding module to map attributerelated visual features 𝑭 𝐴 to the semantic space, obtaining the semantic vector for the image, denoted as 𝑺 = [𝑠 1 , . . . , 𝑠 𝐾 ] ⊤ . Subsequently, the final class probabilities are obtained by computing the similarity between this embedding semantic vector and the class semantic vectors.Here, Semantic Embedding module project the attribute vectors 𝑨 to unify the dimensions with 𝑭 𝐴 , and compute their similarity to obtain the attribute score 𝑠 𝑘 , given by:EQUATIONAfter obtaining the semantic vector 𝑺, class probabilities with scaling can be obtained by calculating the similarity between the mapping semantic vector and the class semantic vector, which can be formulated as:EQUATIONTo optimize the model, existing methods typically take cross entropy loss and calibration loss [6, 18, 26] , which can be expressed as:EQUATIONthe 𝜆 𝑐𝑎𝑙 is the weight to control the weight coefficient of calibration loss.We can find that such objective functions primarily utilize class labels as supervision, while ignoring establishing substantive visualsemantic correlation for ZSL. Therefore, building upon this, we design Conditional Correlation Loss to effectively leverage semantic information, and introduce counterfactual causal intervention (including Semantic Causal Intervention and Target Causal Intervention) to strengthen the substantive visual-semantic correlation.As analyzed in the introduction, the mismatch between images and corresponding attribute annotations hinders substantive visualsemantic correlation. Such mismatch is caused because instancelevel images can not match the complete attribute annotations defined at the class level.Given that the attributes present in images are typically less than attribute annotations, we impose major penalties on the model for predicting attributes beyond the annotations. Simultaneously, minor penalties are applied to the model for failing to predict attributes present in the annotations but absent from the images, thus facilitating effective utilization of attribute annotations. Therefore, we define two semantic difference vectors:EQUATION)Δ𝑺 1 represents the excess of predicted semantic information in the images beyond the predefined semantic information, and Δ𝑺 2 represents the opposite. By imposing differentiated penalties on Δ𝑺 1 and Δ𝑺 2 , we can effectively utilize semantic information. Meanwhile, we introduce a regularization term 𝐽 (𝑺) to constrain the distribution of 𝑺, which maintains the standard deviation of 𝑺 within the same range as 𝑺 𝑦 . Therefore, the Conditional Correlation Loss is formulated as:EQUATIONHere, 𝛼 is the coefficient to control the differentiated penalty weight, we set it to 0.5.Here, we first present the causal perspective within the CVsC, followed by the introduction of Semantic Causal Intervention and Target Causal Intervention that we designed. Causal View of ZSL We introduce the formulation of causality for CVsC by using causal graph (also known as structural causal model [33] ). The causal graph is a directed acyclic graph G = {N, E}, where each variable in the model corresponds to a node in N , and the causal links in E describe how these variables interact. As depicted in Figure 3 𝑆. As for 𝑆 → 𝑌 , it signifies the prediction of image categories based on 𝑆.The variable 𝐹 𝐴 , as the product of visual-semantic interaction, reflects and influences the substantive nature of visual-semantic correlation. Traditional methods primarily rely on seen class labels to optimize models and treat the model as a black box, overlooking the impact of the quality of 𝐹 𝐴 on semantic and target predictions. Causal inference provides a tool to break free from the black box by analyzing causal relationships between variables, aiding us in critical analysis. Therefore, we utilize causality to assess the effectiveness of learned 𝐹 𝐴 , then encourage the model to learn more effective 𝐹 𝐴 to establish substantive visual-semantic correlation.The introduction of a causal graph enables us to analyze the causalities by manipulating variables and observing their effects. Such operation is termed intervention in the causal inference [34, 41] and can be represented as 𝑑𝑜 (•). When investigating the impact of a variable, the intervention involves removing all incoming links to the variable and assigning it a new value. For instance, in our causal graph, 𝑑𝑜 (𝐹 𝐴 = F 𝐴 ) denotes setting the variable 𝐹 𝐴 to F 𝐴 and severing the link 𝐼 → 𝐹 𝐴 , thereby removing the causal influence from its parent variable 𝐼 .Inspired by existing causal inference methods [33, 38, 41] , we incorporate counterfactual intervention to investigate the impact of 𝐹 𝐴 . The counterfactual intervention is achieved by an imaginary intervention altering the state of the variables assumed to be different [38, 41] . In our work, we conduct counterfactual intervention 𝑑𝑜 (𝐹 𝐴 = F 𝐴 , 𝐼 = (𝑭 , 𝑨)) by replacing the learned attribute-related visual features 𝑭 𝐴 with non-existent features F 𝐴 while keeping the input 𝐼 unchanged. Subsequently, we can observe its impacts on semantic and target predictions, as depicted in Figure 3 (a)(b). By maxing these impacts, we can make the model learn more efficient 𝐹 𝐴 . The specific operations will be detailed in subsequent sections. 3.4.1 Semantic Causal Intervention. The variable 𝐹 𝐴 first affects the semantic prediction results. Thus, we observe its impacts on semantic prediction. By substituting the learned attribute-related visual features 𝑭 𝐴 with imaginary features F 𝐴 , we implement a counterfactual intervention 𝑑𝑜 (𝐹 𝐴 = F 𝐴 ). In practice, we replace 𝑭 𝐴 with randomly generated features of the same dimensions. Referring to [33, 38, 41] , we can assess the influence of variable 𝐹 𝐴 by observing the difference between original semantic predictions 𝑆 (𝐹 𝐴 = 𝑭 𝐴 , 𝐼 = (𝑭 , 𝑨)) and semantic predictions under counterfactual intervention 𝑆 (𝑑𝑜 (𝐹 𝐴 = F 𝐴 ), 𝐼 = (𝑭 , 𝑨)):𝑆 𝑒 𝑓 𝑓 = 𝑆 (𝐹 𝐴 = 𝑭 𝐴 , 𝐼 = (𝑭 , 𝑨)) -𝑆 (𝑑𝑜 (𝐹 𝐴 = F 𝐴 ), 𝐼 = (𝑭 , 𝑨)).(8) 𝑆 𝑒 𝑓 𝑓 reflects the impact of 𝐹 𝐴 for the semantic prediction, thus maximizing the 𝑆 𝑒 𝑓 𝑓 can guide the model to learn more effective 𝐹 𝐴 . Additionally, we introduce a regularization term to prevent the model from simply weakening 𝑆 (𝑑𝑜 (𝐹 𝐴 = F 𝐴 ), 𝐼 = (𝑭 , 𝑨)), which utilize class semantic vectors to constrain 𝑆 𝑒 𝑓 𝑓 . Consequently, the Semantic Causal Intervention can be finally formulated as the following objective function:EQUATION)3.4.2 Target Causal Intervention. The variable 𝐹 𝐴 also influences the final target prediction. Therefore, we simultaneously maximizing the impact of 𝐹 𝐴 on target prediction to learn effective 𝐹 𝐴 . Similarly, the influence of 𝐹 𝐴 on the target prediction 𝑌 can be represented by the difference between the original 𝑌 (𝐹 𝐴 = 𝑭 𝐴 , 𝐼 = (𝑭 , 𝑨)) and the counterfactual intervention 𝑌 (𝑑𝑜 (𝐹 𝐴 = F 𝐴 ), 𝐼 = (𝑭 , 𝑨)):𝑌 𝑒 𝑓 𝑓 = 𝑌 (𝐹 𝐴 = 𝑭 𝐴 , 𝐼 = (𝑭 , 𝑨)) -𝑌 (𝑑𝑜 (𝐹 𝐴 = F 𝐴 ), 𝐼 = (𝑭 , 𝑨)).(10) Here, we employ cross-entropy to design the objective function for the Target Causal Intervention, which can be expressed as:EQUATION)Through the collaborative effort between Semantic Causal Intervention and Target Causal Intervention, the model is encouraged to learn effective attribute-related visual features, thereby establishing substantive visual-semantic correlation.To optimize our CVSC, we need to minimize the overall objective function, which comprises the typical loss in 5, Conditional Correlation Loss, Semantic Causal Intervention and Target Causal Intervention. This can be represented as:EQUATIONwhere 𝜆 𝑐𝑐 , 𝜆 𝑠𝑐𝑖 and 𝜆 𝑡𝑐𝑖 are the weight to control the Conditional Correlation Loss, Semantic Causal Intervention, and Target Causal Intervention, respectively. After completing model training, we can directly take the model to predict the unseen images under the CZSL setting. While, for the GZSL setting, where test images come from both seen and unseen classes, the calibration factor [4] is employed to adjust the model bias towards seen classes. Specifically, the GZSL predicted expression is defined as:EQUATIONwhere 𝛿 1 and 𝛿 2 are the calibration factor corresponding to two subnets, and I [𝑐 ∈ C 𝑢 ] is an indicator function. 
experiments: 
 In this section, we introduce the datasets, evaluation protocols, and implementation details. Furthermore, we provide a series of experiment analyses to verify our method. Datasets. We conduct extensive experiments on three widely used benchmark datasets, i.e., AWA2 [46] , CUB [44] and SUN [31] . AWA2 For each dataset, we followed the recommended splits [46] , dividing the classes into seen and unseen, as detailed in Table 2 .Evaluation Protocols. The performance of ZSL is evaluated by testing the average top-1 accuracy for each class. In the CZSL setting, we calculate the accuracy (𝐴𝑐𝑐) by predicting the unseen classes on the test samples. In GZSL, which testing set consists of both seen and unseen samples, we need to evaluate the accuracy separately for the seen classes (𝑆) and unseen classes (𝑈 ). Therefore, the performance of GZSL is ultimately assessed by using their harmonic mean, defined as 𝐻 = (2 × 𝑆 × 𝑈 )/(𝑆 + 𝑈 ) [46] . Implementation Details. We implemented our method by using the PyTorch framework 2 . We employed the ViT [13] pre-trained on ImageNet [12] as the backbone for visual feature extraction. The input images were resized to 224 × 224. The attribute vectors were extracted by using the GloVe model [36] trained on Wikipedia articles. The model was optimized by using the Adam [19] optimizer on an NVIDIA 3090.We compute the performance of CVsC under the CZSL and GZSL settings on the CUB, SUN, and AWA2. The results of CVsC are compared with state-of-the-art methods employing different visual backbones, such as TransZero [6] and MSDN [7] using ResNet as the backbone, and PSVMA [24] using ViT as the backbone. Table 2 shows the comparison results. In the CZSL setting, our CVsC achieves the best average top-1 accuracies of 79.1%, 71.5% and 73.1% on CUB, SUN and AWA2, respectively. And under the GZSL setting, CVsC also gets the best results of harmonic mean, e.g., 75.3%, 53.8% and 76.4% on CUB, SUN and AWA2, respectively. These results demonstrate the superiority of the proposed method, which 2 https://pytorch.org/ To gain further insights into CVsC, we conducted ablation studies to evaluate the effectiveness of its key components, namely Conditional Correlation Loss, Semantic Causal Intervention and Target Causal Intervention.Here, we take the model with the same model architecture but without the above three main modules as the baseline. Subsequently, we compared the baseline with CVsC models without Conditional Correlation Loss (denoted as CVsC w/o CC), without Semantic Causal Intervention (denoted as CVsC w/o SCI), and without Target Causal Intervention (denoted as CVsC w/o TCI). We conducted experiments on CUB, SUN, and AWA2 datasets, and the results are presented in Table 3 . It shows that CVsC exhibits significant performance improvement relative to the baseline, while the absence of the aforementioned three components leads to performance degradation. This indicates that all three components effectively assist the model in constructing substantive visual-semantic correlation, thereby enhancing ZSL performance. In addition to the improvement in accuracy, we conducted quantitative experiments to further demonstrate the ability of our CVSC to establish more substantive visual-semantic correlation, including semantic error visualization and attribute attention map visualization.Inspired by [52] , we calculated the class-averaged semantic error by computing the difference between predicted attribute values 𝑠 𝑘 and annotated values 𝑠 𝑐 𝑘 of class 𝑐 on the test set. We normalize the predicted attribute values in the same manner as the annotated attribute values, mitigating the influence of data outliers on the computation of class-averaged semantic error. Therefore, the class-averaged semantic error can be calculated as follows:EQUATIONwhere N 𝑐 the sample set of class 𝑐 on the test set, and |N 𝑐 | is the number of corresponding samples. We adopt the same baseline as in the ablation experiment and compute the class-averaged semantic error for both the baseline and CVsC on CUB. Then, we visualize the class-averaged semantic errors of both the baseline and CVsC for seen and unseen classes separately using matrices, as illustrated in Figure 4 . It can be observed that CVsC significantly reduces semantic errors for both seen and unseen classes.Furthermore, we statistically analyzed the overall average errors for seen and unseen classes, and calculated the increase in error for unseen classes relative to seen classes, as shown in Table 4 . It can be observed that CVsC not only has lower semantic errors compared to the baseline but also exhibits significantly lower error increases for seen classes. These results demonstrate that our CVsC can enhance the substantiveness of visual-semantic correlation, facilitating knowledge transfer from seen to unseen classes.Map. We visualized the model's attribute attention maps, implemented by computing the attribute attention map 𝑴 defined in Eq. 2, to further demonstrate the role of CVsC in establishing substantive visual-semantic correlation.Here, we conducted visualizations separately for both the baseline and CVsC on CUB. The results shown in Figure 5 , reveal numerous attribute localization errors in the baseline, whereas CVsC significantly improves upon this situation. This observation suggests that although ViT demonstrates strong representational capabilities, deeper layers are more susceptible to attention collapse [56] , leading to challenges in attribute localization compared to convolutional networks. This difficulty impedes the model from accurately associating attributes with specific regions, resulting in spurious visual-semantic correlation and limiting knowledge transfer. However, our CVsC approach facilitates the establishment of substantive visual-semantic correlation, enabling accurate attribute localization.There are three key factors involved in our method, the weights of Conditional Correlation Loss, Semantic Causal Intervention and Target Causal Intervention, i.e., 𝜆 𝑐𝑐 , 𝜆 𝑠𝑐𝑖 and 𝜆 𝑡𝑐𝑖 . To analyze the robustness of our CVsC and select optimal hyperparameters for it, we try a wide range of 𝜆 𝑐𝑐 , 𝜆 𝑠𝑐𝑖 and 𝜆 𝑡𝑐𝑖 evaluated on CUB. The results are shown in Figure 6 . It shows that CVsC is robust when the loss weights are set to small, while the performance drops rapidly when loss weights are set to too large. Because the large loss weights will hamper the balance of various losses. According to the results in Figure 6 , we set the weights {𝜆 𝑐𝑐 , 𝜆 𝑠𝑐𝑖 , 𝜆 𝑡𝑐𝑖 } to {0.5, 2.0, 2.0}. 
conclusion: 
 In this paper, we emphasize the significance of visual-semantic correlation for ZSL, and highlight the presence of spurious visualsemantic correlation caused by mismatches between instance-level images and class-level attribute annotations. This inspires us to propose a novel method termed Causal Visual Semantic Correlation  
 
 Appendix: It is noted that ZSL is typically denoted as zero-shot image classification, and we follow the standard in this paper. 


Decision: In this paper, the authors propose the Causal Visual-semantic Correlation method, which uses a Visual Semantic Attention module and Conditional Correlation Loss to learn meaningful visual-semantic correlations, enhanced by counterfactual interventions, resulting in better performance against SOTA methods on three benchmark datasets. Some issues (more discussions on related works, clearer mathematical notations, improved figure representation) need to be addressed. For more details, please refer to the reviewers' detailed comments and final thoughts.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of Zero-Shot Learning. Existing methods often directly align visual and semantic features but suffer from mismatched correlations. To overcome this issue, the authors propose a novel method called Causal Visual-Semantic Correlation (CVsC). This method introduces counterfactual intervention to attribute-related visual features, maximizing their impact on semantic and target predictions to reinforce substantive correlations. The proposed method achieves the best results on three benchmark datasets, and ablation studies demonstrate the effectiveness of each component.
strength_weakness: strength_weakness: The paper is well-written and easy to follow.

The motivation for introducing counterfactual intervention in this paper is clear and innovative, and the results are excellent.
limitations: limitations: 1 Recent methods in the related work section on Attention Mechanism also need to be discussed, such as [1,2,3]. Similarly, [4] should also be compared and discussed.

[1] MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning. CVPR2022.

[2] Dual Part Discovery Network for Zero-Shot Learning. MM 2022.

[3] DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning. AAAI 2023

[4] Counterfactual Zero-Shot and Open-Set Visual Recognition. CVPR2021

2 There are still questions regarding the design of $\triangle S_1$ and $\triangle S_2$, which require further explanation.

3 From the ablation study, it appears that CCL contributes the most. What could be the possible reasons for this?
paper_summary: paper_summary: This paper aims to learn substantive visual-semantic correlation and eliminate possible spurious visual-semantic correlation in zero-shot learning. Therefore, this paper proposes to tackle this problem from visual-semantic attention mechanism and causal visual-semantic correlation learning with counterfactual intervention. Experiments show some effectiveness.
strength_weakness: strength_weakness: +This paper introduces causal inference in zero-shot learning to solve visual-semantic correlation issue.

+Experiments show some effectiveness of the proposed method.
limitations: limitations: --There lacks experimental evidence or observation to show the spurious visual-semantic correlation in zero-shot learning.

--There is no visualization comparison before and after removing spurious correlation. It is unclear if the improvement in performance is from the spurious correlation removal.

--The attention mechanism between visual-semantic is not novel because this is common in idea due to the wide use of self- and cross- attention.

--Leveraging counterfactual intervention is widely studied in related fields, such as domain generalization. Therefore, the novelty is limited.

--The writing has a large space for improvement. I obverse a number of typos in language. 

--There are some important mathematical symbols missing, such as W_q in Eq.1.

--More ZSL methods are not discussed and compared.
paper_summary: paper_summary: This paper introduces counterfactual intervention applied to ZSL, and maximize their impacton semantic and target predictions to enhance substantive visual-semantic correlation.
strength_weakness: strength_weakness: Extensive experiments on multiple benchmark datasets demonstrate that the method enhances the substantiveness of
visual-semantic correlation and significantly improves the performance of ZSL.
limitations: limitations: The method's backbone does not correlate well with ViT, so experimental results from other backbones, such as ResNet-101, can be provided to provide more detailed comparisons.
paper_summary: paper_summary: Zero-Shot Learning bridges visual data with semantic descriptions to generalize to unseen classes, conventionally aligning visual and semantic features. 
However, individual images rarely encapsulate the full class-level semantics, causing misaligned correlations. 
This tackles this by employing Visual Semantic Attention for attribute-focused feature extraction, Conditional Correlation Loss for guided alignment, and counterfactual interventions to amplify meaningful feature-semantic associations, yielding superior performance on CUB, SUN, and AWA2 datasets compared to state-of-the-art models.
strength_weakness: strength_weakness: The idea  is interesting.

The experiments prove the effectiveness of the method.
limitations: limitations: Some issues need to be addressed:

1. There are some errors in the paper. For example, why is it not Q*K^T in Eq.2?

2. The font in Figure 1 is too small, and the idea is not expressed clearly enough.

3. The author lacks discussion of some related work in the field of zero-shot learning. For example [1].
[1] A causal view of compositional zero-shot recognition

4. Will the code be released publicly?