CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-ShoT Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 03:35:54

Abstract: Zero-shot learning (ZSL) enables the recognition of novel classes by leveraging semantic knowledge transfer from known to unknown categories. This knowledge, typically encapsulated in attribute descriptions, aids in identifying class-specific visual features, thus facilitating visual-semantic alignment and improving ZSL performance. However, real-world challenges such as distribution imbalances and attribute co-occurrence among instances often hinder the discernment of local variances in images, a problem exacerbated by the scarcity of fine-grained, region-specific attribute annotations. Moreover, the variability in visual presentation within categories can also skew attribute-category associations. In response, we propose a bidirectional cross-modal ZSL approach CREST. It begins by extracting representations for attribute and visual localization and employs Evidential Deep Learning (EDL) to measure underlying epistemic uncertainty, thereby enhancing the model's resilience against hard negatives. CREST incorporates dual learning pathways, focusing on both visual-category and attribute-category alignments, to ensure robust correlation between latent and observable spaces. Moreover, we introduce an uncertainty-informed cross-modal fusion technique to refine visual-attribute inference. Extensive experiments demonstrate our model's effectiveness and unique explainability across multiple datasets. Our code and data are available at: https://anonymous.4open.science/r/CREST-1CEC.

Main paper: 
 introduction: 
 Humans frequently possess the talent to grasp novel concepts relying on prior experience without the need to see them beforehand. For instance, a peacock is commonly known as a bird with a colorful fan-shaped tail; if individuals have previous knowledge of birds and fans, they can quickly identify a peacock. However, unlike humans, widely used and studied supervised deep learning models are typically limited to classifying samples belonging to categories seen during training, lacking the capacity to handle samples from unseen categories during training, thus lacking generality and flexibility. Therefore, to further advance Artificial General Intelligence (AGI) [4] and achieve true implementation, Zero-Shot Learning (ZSL) was introduced to identify new classes by leveraging inherent semantic relationships during learning [19, [33] [34] [35] 44] . It is already extensively applied in tasks with broad real-world applications, e.g., image classification [18, 63] , semantic segmentation [5, 17] , video understanding [66, 71] , 3D generation [29, 67] , etc., which also contributes significantly to the robust development of Large Language Models (LLMs) [31, 58] and Embodied AI [26, 56] .In ZSL, attributes stand as key semantic descriptors for visual features of images, representing a widely embraced form of annotation. Unfortunately, the attribute annotations are more often categorical rather than regional [13] . Dense attention interactions do not guarantee that models directly grasp the correspondence between local visual-semantic information and categories, nor do they alleviate the model's epistemic uncertainty when confronted with unseen categories [51] . That is because the skewed distribution of attributes in the real world, as well as the issues arising from attribute co-occurrence shown in Figure 1 .Existing methods overlook the importance of aligning regional features with categories. Models may link specific attributes, like a red bird's bill, to "bill color red" but struggle to deduce the bird's species. This challenge is compounded as attributes across species are often intertwined. Furthermore, real-world images of the same category vary significantly due to factors like camera angles, background, distances, lights and the motions, making it difficult for dense attention to learn hard category-matching patterns. This can increase epistemic uncertainty when merging features for inference, potentially exacerbating modal conflicts and impairing model performance [65] .To this end, we integrate Evidential Deep Learning (EDL) [51] into ZSL for the first time, leading to a novel framework, named Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-ShoT Learning, termed as CREST. Specifically, we employ the Visual Grounding Transformer (VGT) and the Attribute Grounding Transformer (AGT) to extract bidirectional, regionlevel features from images and attributes. Unlike conventional approaches that simply adjust distances within the representation space based on category [13] , our strategy addresses vision variability and feature-category alignment directly. We first introduce instance-level contrastive learning for adaptive vision alignment and employ a technique similar to non-maximum suppression to reduce attribute overlap between categories, facilitating deeper attribute-category insights. To counteract the potential degradation from hard-negative samples [48] , we apply EDL for epistemic uncertainty measure and develop an uncertainty-driven fusion method [23, 24, 38, 65] . This enhances the model's generalization in downstream tasks by merging semantic knowledge across representation spaces. To summarize, our contributions are as follows:• We propose CREST, a novel ZSL framework that considers bidirectional cross-modal representations of attributes and visual features. Moreover, it leverages dual learning pathways, focusing on both visual-category and attributecategory alignments, learning implicit matching patterns between features and categories from fine-grained visual elements and attribute texts. • To the best of our knowledge, CREST is the first in ZSL to apply EDL for measuring epistemic uncertainty and mitigating potential conflicts in cross-modal fusion.• Extensive experiments show that CREST performs competitively on three well-known ZSL benchmarks, i.e., CUB [59] , SUN [46] , and AWA2 [60] . Comprehensive ablations and analyses further validate the effectiveness and explainability of our approach. 
related work 2.1 zero-shot learning: 
 ZSL can be classified into two main categories based on the classes encountered during the testing phase: Conventional ZSL (CZSL) and Generalized ZSL (GZSL), where CZSL is designed to predict classes that have not been seen during training, whereas GZSL extends its predictive capability to both seen and unseen classes [60] .The core concept of ZSL revolves around learning discriminative and transferable visual features based on semantic information, e.g., attribute descriptions [34] , sentence embeddings [49] , and DNA [2] , enabling effective visual-semantic interactions. Among these, attributes stand out as the most commonly used semantic information within ZSL. Early research focused on harnessing visual-semantic interactions to transfer knowledge to unseen categories [1, 37, 55] . These initial attempts, particularly through embedding-based methods, entailed learning a mapping between seen categories and their corresponding semantic vectors, followed by employing nearest neighbor searches within the embedding space to classify unseen categories [68, 70] . Since they primarily rely on seen category samples, the effectiveness was significantly limited due to a bias towards these categories, exacerbating the challenge in GZSL. Novel regularization and space modification strategies have been developed to improve ZSL model generalization [36, 45, 54] . Generative models, including VAEs [11, 12, 50, 57] , GANs [10, 21, 61, 63] , and generative flows [53, 74] , synthetically enhance feature spaces with unseen class characteristics. These methods, aiming to bridge the domain gap, reframes ZSL as a supervised task by providing a means to compensate for the lack of unseen class data. Despite progress, these methods often neglect localized visual cues in favor of global information, overlooking the nuanced, fine-grained attributes essential for dissecting complex semantic categories [6, 14] . This oversight weakens the visual representations obtained, diminishing the efficacy of the visual-semantic knowledge transfer. Subsequently, intricate attentions are integrated into ZSL to prioritize salient features and attributes, improving model discernment [16, 28, 32, 39, 43, 75] . And Recent studies have started experimenting with the deployment of intricate attention to engage with region-level visual-attribute features [6] [7] [8] 13] . These methods highlight distinctive, fine-grained features, evolving towards complex attention modules for deeper semantic understanding. However, due to instance-level visual variability and inter-class attribute coupling, fine-grained representations may not guarantee accurate feature-to-category matching. This paper delves into aligning latent feature and category spaces.EDL enhances machine learning by enabling models to quantify uncertainty, thus bolstering reliability and interpretability. Grounded in subjective logic principles [30] , EDL has emerged as a response to the challenges of model confidence and uncertainty, as highlighted in neural network calibration issues by [20] . The framework's utility was further solidified by [51] , which introduced a method to quantify classification uncertainty, significantly increasing deep learning model trustworthiness. The adaptability of EDL to various data contexts has been demonstrated through applications like open set action recognition [3] , signifying its efficacy in handling new and unseen data types. The scope of EDL further expanded to multi-view classification [23] , showcasing its ability to integrate and reason with information from multiple sources. This integration was further enhanced by introducing dynamic evidential fusion [24] , highlighting EDL's adaptability in complex data environments.Recent advancements, such as adaptive EDL for semi-supervised learning [72] and its application in multimodal decision-making [52] , have marked EDL's progression towards addressing real-world data challenges. Additionally, [65] illustrates EDL's potential in conflictive multi-view learning scenarios, reinforcing its capacity to support reliable decision-making across diverse applications. In ZSL, there exists epistemic uncertainty in the gap between region-level fine-grained latent space and category space. Moreover, dual-stream visual-attribute interactions do not necessarily align representation spaces. Therefore, we apply EDL to assess feature-category alignment uncertainties independently and introduces an uncertaintydriven fusion framework for coherent visual-attribute inference. 
methodology 3.1 problem definition: 
 ZSL equips models to recognize targets in unseen categories. The training set, 𝐷 𝑠 = {(𝑥 𝑠 , 𝑦 𝑠 )|𝑥 𝑠 ∈ X 𝑠 , 𝑦 𝑠 ∈ Y 𝑠 }, consists of samples from known categories, with 𝑥 𝑠 as images labeled 𝑦 𝑠 . The set 𝐷 𝑢 = {(𝑥 𝑢 , 𝑦 𝑢 )|𝑥 𝑢 ∈ X 𝑢 , 𝑦 𝑢 ∈ Y 𝑢 }Feature Extraction: Attributes and Vision. We extract textual features using the pre-trained GloVe model [47] , while employing ResNet-101 [25] as the CNN backbone to distill visual features from images (as depicted in Figure 2 (a)(b)). These features support the development of a bidirectional grounding Transformer.Bidirectional Grounding Transformer. In the decoding phase, the VGT and AGT refine visual and semantic attributes, respectively. The VGT attend semantic features to localize relevant image regions, whereas the AGT interprets semantic information through regional visual features. Both decoders employ a streamlined crossattention module, with the encoder output 𝑈 serving as keys 𝐾 and values 𝑉 , and semantic embeddings as queries 𝑄. This methodology establishes a bidirectional link between images and attributes, enhancing the recognition of unseen categories. The process is concisely described as follows:EQUATIONwhere 𝑊 𝑞 , 𝑊 𝑘 , 𝑊 𝑣 are the learnable weights in cross attention, 𝑑 𝑘 represents the dimension of the features. After 𝑛 layers of iteration, the output F is transformed by a FeedForward layer:EQUATIONThe AGT structure is analogous to the VGT, differing only in the modalities employed as queries in the cross-attention modules.Overall, the features of attribute and visual localization 𝐹 𝐴 , 𝐹 𝑉 can be respectively captured through the application of AGT and VGT. Generally speaking, existing methods achieve implicit alignment with the categorical space by mapping latent semantic matches in text to relevant visual regions in images, subsequently employing fine-grained embeddings. However, in the real world, the images captured often exhibit visual variability due to factors such as angles, backgrounds, distances, illumination, and motion (as shown in Figure 3 ). This variability significantly diminishes the practical effectiveness of textual semantics since the fine-grained visual representations derived from text may not necessarily correspond to the typical categories intended. Conversely, subjects from different categories might appear visually similar due to these influencing factors. To foster proximity among similar entities and distance among distinct categories in the representational space, some approaches might consider employing intra-group category labels for supervision. This method, however, could yield suboptimal solutions due to the vision variability present in an open-world scenario.To this end, we propose the Visual Instance-level Contrastive Learning (VICL) to mitigate the gap between fine-grained visual latent space and intra-category space.EQUATION𝑓 𝜃 =exp 𝐷 ṽ, ṽ+ /𝜏 exp (𝐷 ( ṽ, ṽ+ ) /𝜏) + ṽ-∈ N ( ṽ) exp (𝐷 ( ṽ, ṽ -) /𝜏) ,where ṽ, ṽ+ and ṽrepresent a candidate positive sample, its positive sample and negative sample respectively. And we adopt a strategy that adjusts for intra-category visual variability through a similarity-based selection of positive samples. Given a batch, the similarity score 𝐷 ( ṽ, ṽ+ ) is computed. If no intra-category sample resemble the candidate, we then identify the similar samples across the batch to serve as the positive samples, irrespective of category, based on the similarity score. This approach enables the model to maintain category coherence despite visual discrepancies.Traditional methods typically align attribute features with visual features in a straightforward manner to achieve recognition outcomes. However, as illustrated in Figure 4 , where attributes coupling across categories, posing challenges to accurate identification. As shown in Figure 2 Specifically, in a batch with 𝑁 samples, our model uses the AGT to map the ℎ-dimensional features 𝐹 𝐴 ∈ R 𝑁 ×ℎ to the queries 𝑄 ∈ R 𝑁 ×𝑑 in the latent space of a meta pattern bank with 𝑊 𝑄 ∈ R ℎ×𝑑 and bias 𝑏 𝑄 ∈ R ℎ×𝑑 . These queries compute similarity scores with pattern vectors Φ via dot products. Equation 5 delineates the transformation where 𝑄 (𝑖 ) = 𝐹 𝐴 𝑖 𝑊 𝑄 + 𝑏 𝑄 generates the attention score 𝑎 (𝑖 ) 𝑗 that leads to the sparse attention-weighted feature vector 𝑉 (𝑖 ) * . This vector is subsequently remapped to the latent space of 𝐹 𝐴 , and directly added to it, enhancing the feature set by integrating the weighted information from the latent space. To decouple the attribute-category mapping in this latent space, we embrace the DIGS loss inspired by non-maximum suppression (NMS). It operates on two fronts:(i). The triplet loss component incentivizes the distinction between the closest and second-closest memory pattern vectors. Let 𝑄 (𝑖 ) be the query representation for the 𝑖-th example, Φ[𝑝] the most similar memory pattern (positive sample), and Φ[𝑛] the second most similar memory pattern (negative sample). The triplet loss is then defined as:L tp = 𝑁 ∑︁ 𝑖=1 max 𝑄 (𝑖 ) -Φ[𝑝] 2 -𝑄 (𝑖 ) -Φ[𝑛] 2 + 𝜆, 0 , (6)where 𝜆 is a margin enforcing that the similarity between 𝑄 (𝑖 ) and Φ[𝑝] exceeds that between 𝑄 (𝑖 ) and Φ[𝑛] by at least 𝜆, encouraging the model to focus on positive samples and hard negatives and pull positive samples closer to the anchor 𝑄 (𝑖 ) than negative ones.(ii). The regularization term promotes compact clustering of patterns by minimizing the distance between each query and its most similar memory pattern. This is quantified as:EQUATIONBy synthesizing these components, the DIGS loss is articulated as L DIGS = L tp + L reg . Hence, it ensures that the memory patterns not only cluster tightly but also maintain separation, enabling the model to discern and generalize known patterns effectively while grasping the relational structure of the prototypes. Given two opinions on the same instance, 𝜔 𝐴 = (𝑏 𝐴 , 𝑢 𝐴 , 𝑎 𝐴 ) and 𝜔 𝐵 = (𝑏 𝐵 , 𝑢 𝐵 , 𝑎 𝐵 ), their synthesis 𝜔 𝐴⊕𝐵 combines their beliefs, uncertainty, and evidence as follows:𝑏 𝐴⊕𝐵 𝑘 = 𝑏 𝐴 𝑘 𝑢 𝐵 + 𝑏 𝐵 𝑘 𝑢 𝐴 𝑢 𝐴 + 𝑢 𝐵 , 𝑢 𝐴⊕𝐵 = 2𝑢 𝐴 𝑢 𝐵 𝑢 𝐴 + 𝑢 𝐵 , 𝑎 𝐴⊕𝐵 𝑘 = 𝑎 𝐴 𝑘 + 𝑎 𝐵 𝑘 2 ,where 𝑎 𝐴 , 𝑎 𝐵 represent two different base distribution(e.g. Uniform distribution). The conflict degree 𝑐 (𝜔 𝐴 , 𝜔 𝐵 ) assesses the divergence and shared certainty between 𝜔 𝐴 and 𝜔 𝐵 :EQUATIONEQUATIONEQUATIONwhere 𝑝 represent the linear projected probability distributions of the opinions by Dirichlet parameters (i.e. 𝑏 and 𝑢). This framework facilitates a nuanced analysis of agreement and discord between the opinions.As illustrated in Figure (2), we treat the outputs of VGT and AGT as evidence vectors, which typically involve issues of ambiguous recognition. Employing EDL allows us to precisely quantify these uncertainties, thereby deriving accurate recognition results. For each instance {x 𝑚 𝑛 } 𝑀 𝑚=1 , the modality count 𝑀 encapsulates two modalities in our bidirectional grounding Transformer, namely visual-to-attribute and attribute-to-visual. , where 𝐾 = |C|. Adapting to unimodal evidence-based classification, the traditional cross-entropy loss is intricately tailored for compatibility with this framework:EQUATIONwhere L 𝐴𝐶𝐸 (𝜶 𝑚 𝑛 ) denotes the unimodal adaptive cross-entropy loss for the parameters 𝜶 𝑚 𝑛 of the Dirichlet distribution for a single instance 𝑛. Utilizing the digamma function 𝜓 , the integral is simplified to the expectation of the logarithm of predicted probabilities, where 𝑆 𝑛 represents the sum of Dirichlet parameters for instance 𝑛, reflecting the total evidence across all classes. The objective of this adaptive loss function is to adjust the network's output parameters to accurately represent the inherent uncertainty in predictions, enabling the network to make confident predictions when evidence is ample and maintain a degree of uncertainty when evidence is scarce.Nevertheless, the aforementioned loss function fails to address the issue of insufficient evidence caused by incorrect labels. Therefore, we incorporate a Kullback-Leibler (KL) divergence term into the loss function.EQUATIONwhere 𝐷 (𝑝 𝑛 | 1) represents the uniform Dirichlet distribution, α𝑚 𝒏 = y n + (1-y n ) ⊙ 𝜶 𝒎 𝒏 denotes the Dirichlet parameters after excluding non-misleading evidence from the predicted parameters 𝜶 𝑚 𝑛 for the 𝑛-th instance, and Γ(•) signifies the gamma function.Hence, for the 𝑛-th instance in the single-modality setting with Dirichlet distribution parameter 𝜶 𝒎 𝒏 , the loss is computed as follows:EQUATIONWhere 𝜆 𝑡 = min(1.0, 𝑡/E) ∈ [0, 1] denotes the annealing coefficient, with 𝑡 being the index of the current training epoch and E representing the annealing steps. Gradually increasing the influence of KL divergence in the loss function prevents premature convergence of misclassified instances to a uniform distribution.To ensure consistency across differing perspectives during training, a method to minimize the degree of opinion conflict is employed. The consistency loss for instance x 𝑚 𝑛 𝑀 𝑚=1 is calculated as follows:EQUATIONIn the processes of VGT and AGT, mismatches may arise, linking attribute features to incorrect visual parts, or the reverse. The parameter 𝑐 serves to measure the conflict level between two opinions, where 𝑐 = 0 denotes a lack of conflict and 𝑐 = 1 denotes direct opposition. For the specific instance x 𝑚 𝑛 𝑀 𝑚=1 , the overall EDL loss functions can be given as follows:EQUATION)where α𝒏 shaped by the fusion of modalities driven by uncertainty 𝑢 𝑚 𝑛 (e.g., the uncertainty-weighted average of modalities' 𝜶 𝑚 𝑛 ) calibrates the EDL loss relative to the observed conflict degree.Attribute Reinforced Semantic Integration. We introduce an Attribute ReInforced SEmantic Integration (ARISE) to improve model discrimination by embedding attribute information into 𝑖=1 with their corresponding class semantic vectors 𝑧 𝑐 , L 𝐴𝑅𝐼𝑆𝐸 can be formally represented as follows:EQUATIONwhere 𝑓 (𝑥 𝑖 ) = 𝜇𝜶 𝐴 𝑖 + (1 -𝜇)𝜶 𝑉 𝑖 with a blanced coefficient 𝜇. L 𝐴𝑅𝐼𝑆𝐸 aims to minimize the discrepancy between the predicted and true distributions, taking into account the attribute similarities between categories, serving as a regularization term that encourages the model to learn generalizable features across different categories. Therefore, the overall loss can be obtained as follows:EQUATIONUpon completing the training of CREST, we extract the visual embeddings of a test sample 𝑥 𝑖 in the semantic space relative to VGT and AGT, denoted as 𝜶 𝑉 𝑖 and 𝜶 𝐴 𝑖 . Given that the semanticaugmented visual embeddings from VGT and AGT offer complementary information, we integrate their predictions through combination coefficients 𝜇 for a calibrated test label prediction of 𝑥 𝑖 , expressed as:EQUATIONIn this formula, C 𝑢 /C pertains to the CZSL/GZSL scenarios, respectively. 
experiments: 
 Dataset. Our study investigates three principal zero-shot learning (ZSL) benchmarks: two fine-grained datasets, CUB [59] and SUN [46] , and one coarse-grained dataset, AWA2 [60] . CUB encompasses 11,788 images across 200 bird classes (150 seen, 50 unseen), featuring 312 attributes. SUN includes 14,340 images spanning 717 scene categories (645 seen, 72 unseen) with 102 attributes. AWA2 contains 37,322 images of 50 animal classes (40 seen, 10 unseen), each described by 85 attributes. Evaluation Protocols. Following Xian et al.'s framework [62] ,we evaluated the top-1 accuracy in both CZSL and GZSL setups.In CZSL, accuracy is assessed solely by predicting unseen classes.For GZSL, we compute the accuracy for both seen (𝑆) and unseen (𝑈 ) classes and employ their harmonic mean (defined as 𝐻 = (2 × 𝑆 × 𝑈 )/(𝑆 + 𝑈 )) as the evaluative metric. Implementation Details We adopt the training divisions suggested by [61] . The feature extraction backbone is a ResNet101 architecture, which has been pre-trained on ImageNet and is utilized without further fine-tuning. The optimization is performed using the Adam optimizer, with hyperparameters set to learning rate of 0.0001 and a weight decay of 0.0001. And the batch size parameters is set to 64. Based on empirical evidence, the hyperparameters 𝜆 𝐸𝐷𝐿 and 𝜆 𝐶𝐴𝐿 are fixed at 0.001 and 0.2 across all datasets. Finally, the encoder and decoder layers of our bidirectional grounding Transformer are configured with a single attention head.In our comparative analysis, we have examined 17 representative or state-of-the-art models from the period of 2020-2023, as illustrated in Table 1 . Our CREST model consistently outperforms most models across the three benchmarks: CUB, SUN, and AWA2, in terms of CZSL accuracy. Notably, CREST achieves the highest harmonic mean (H) on both CUB and AWA2 benchmarks, indicating a wellbalanced performance between seen (S) and unseen (U) classes, which is a critical measure in ZSL.Our CREST model exhibits robust performance in the GZSL setting for unseen classes (U) on AWA2, achieving competitive accuracy. This highlights CREST's capability to recognize new categories effectively while maintaining strong performance on seen classes. Furthermore, the results indicate that while some models like TransZero++ [6] exhibit high accuracy in seen classes, they do not necessarily maintain this level of performance in unseen classes. In contrast, CREST delivers a more consistent and superior performance across both classes, emphasizing its efficacy in a more diverse and practical setting. The incremental advances observed with CREST affirm the effectiveness of our approach in addressing the challenges intrinsic to zero-shot learning, specifically in maintaining high discriminative power while effectively handling the domain shift between seen and unseen categories.In the ablation study depicted in the first image, the effectiveness of various components of the CREST model is evaluated on CUB and SUN datasets. The study illustrates the importance of each component to the model's performance in both GZSL and CZSL.The removal of the AGT from CREST results in a notable decrease in harmonic mean (H) and accuracy (ACC), demonstrating AGT's significant role in feature transformation. Without the VGT, the model's performance drops drastically, especially in the GZSL scenario, indicating VGT's critical contribution to visual feature integration. The exclusion of the EDL module also leads to diminished GZSL and CZSL outcomes, suggesting its key part in robust fusion and reinforcement of resilience against hard negatives.Further analysis shows that the VICL and DIGS loss both enhance the GZSL performance, as their absence results in lower H scores. Setting the coefficient 𝜆 𝐶𝐴𝐿 of to zero slightly reduces the H scores but the overall full model displays superior performance in terms of both H and ACC, which solidifies the synergy and necessity of the full complement of CREST in achieving state-of-the-art results.The parameter tuning for the CREST model indicates a clear optimum range for both 𝜆 𝐶𝐴𝐿 and 𝜆 𝐸𝐷𝐿 . Performance peaks at moderate values of 𝜆 𝐶𝐴𝐿 before declining, signifying its critical role in balancing GZSL and CZSL outcomes. The influence of 𝜆 𝐸𝐷𝐿 appears more stable, with only a slight drop at high values, suggesting its robust contribution to the model's consistent performance across diverse visual tasks. These findings highlight CREST's ability to maintain accuracy while effectively generalizing to new categories, marking its strengths in a zero-shot learning context.Dynamic Uncertainty Progressive Reduction Visualizations. Figure 6 showcases the evolution of model uncertainty for both the CUB and SUN datasets over training epochs. The density plots vividly demonstrate how uncertainty decreases as the epochs progress, with a significant shift towards lower uncertainty levels upon model convergence. This provides empirical evidence of CREST's learning stability and its increasing confidence in predicting class attributes over time, reflecting its robustness and efficacy in handling diverse data. Attention Mapping and Confidence Scoring Visualizations. In Figure 7 , attention visualization on the CUB Dataset is coupled with uncertainty quantification in attribute recognition. The descending order of rows from top to bottom corresponds to a decrease in attribute certainty, with each image annotated with attribute labels and scores. This not only confirmsCREST's nuanced understanding of attribute saliency but also illustrates the impact of real-world variables such as background clutter and occlusions on the model's performance. Additionally, the model demonstrates a keen perception of hard negatives, as reflected in higher uncertainty scores for attributes that are ambiguous or potentially misleading, which underscores its advanced capability for self-assessment and adaptability in complex visual scenarios. t-SNE Visualizations. For Figure 9 , the t-SNE visualizations illustrate the distinct clustering capabilities of the CREST model. The separate subfigures (a) and (b) highlight the feature spaces created by the VGT and AGT, respectively. Subfigure (c) reveals how the integration of VGT and AGT, through EDL fusion, enhances the distinctiveness of clusters, successfully separating the 10 randomly selected classes from both seen and unseen categories. This indicates CREST's powerful ability to delineate classes in a shared feature space, essential for ZSL. 
conclusion and future work: 
 In conclusion, CREST introduces a pioneering bidirectional crossmodal framework that adeptly addresses the visual-semantic gap in ZSL. By navigating distribution imbalances and attribute cooccurrence, it employs localized representation extraction and EDLbased uncertainty estimation, enhancing resilience and improving alignment in challenging ZSL scenarios. Our extensive evaluations establish CREST's advanced capabilities, reinforcing its standing as an effective and interpretable ZSL solution.In future work, we intend to integrate CREST with LLMs to further enhance semantic alignment and interpretability. This integration aims to enrich CREST's robustness in handling complex zero-shot learning scenarios, thereby extending its applicability and effectiveness in ZSL tasks. Through this synergy, we seek to unlock deeper semantic insights and cater to a broader range of applications in the ever-evolving landscape of machine learning. 
 
 Appendix:  


Decision: Initially, the paper received one Weak Reject and three Borderline Accept scores. The reviewers acknowledged the good motivation and effectiveness of the proposed CREST approach but raised major concerns about the contribution and the similarity to existing methods like TransZero++, the performance on certain datasets, and the clarity and completeness of the experimental results.

The rebuttal addressed these concerns by providing clarifications and additional insights. After the rebuttal, Reviewer ek1r raised the score from  Weak Reject to Borderline Accept, and the other reviewers maintained their Borderline Accept scores, confirming that their concerns were addressed.

The AC has carefully read the paper, reviews, and rebuttal. The AC found the arguments of Reviewers oJvV and khPv to be the most persuasive and weighted the paper's strengths over its current weaknesses - while the experiments and expression can be further improved, its good motivation and solid modeling have potential implications for future research in related areas. The authors are encouraged to improve the final version by incorporating the reviews into the final version.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a novel bidirectional cross-modal ZSL approach CREST. CREST leverages dual learning pathways, focusing on both visual-category and attribute-category alignments. Furthermore, CREST applies Evidential Deep Learning (EDL) to measure underlying epistemic uncertainty and mitigate potential conflicts in cross-modal fusion.
strength_weakness: strength_weakness: - This paper is overall well-written and easy-to-follow.
- Numerous experiments show the effectiveness of the proposed method.
limitations: limitations: - EDL seems useless for the GZSL setting in CUB. When performing CREST w/o EDL, the metric H even becomes better. Could the authors explain this phenomenon?
- TransZero++[1] also introduce a dual-path learning by incorprating visual-to-semantic and semantic-to-visual mappings based on Transformer. Could the authors emphasize any difference between TransZero++ and the proposed method?

[1] TransZero++: Cross Attribute-guided Transformer for Zero-Shot Learning. T-PAMI 2022.
paper_summary: paper_summary: This paper proposes a bidirectional cross-modal zero-shot learning framework to address the attribute distribution imbalance and co-occurence problems. The model is trained in dual pathways to perform visual-class and attribute-class alignments. In visual-class alignment module, the authors utilized contrastive learning to capture fine-grained visual features while keeping class coherence. In attribute-class alignment module, the authors utilized sparse matching and NMS-style loss function to decouple attribute-class mapping patterns. In addition, an EDL-based fusion module is proposed to quantify and fuse visual-attribute evidences. An additional regularization loss is also proposed to enhance the model’s discriminative ability. The experiments show promising results on popular ZSL datasets and effectiveness of the proposed modules.
strength_weakness: strength_weakness: - Well-motivated idea. The issues that this paper aims to address are practical and reasonable in real ZSL scenes. The motivation is well illustrated in figure 1,3,4.
- Solid modeling. The description of the three proposed modules is well formulated and I think it’s feasible to support the motivation. The introduction of EDL is novel comparing to other ZSL approaches.
- Good experimental results. This paper shows promising results on popular ZSL datasets. The ablation studies and visualizations show the effectiveness of proposed modules and loss functions.
limitations: limitations: - The results on SUN and AWA2 are relatively low comparing to other SOTAs.
- Some expressions are unclear, e.g., in Section 4.2, “In the ablation study depicted in the first image”, which image? Which table does this ablation refer to? In Section 4.3, the figure of results isn’t specified clearly yet. I recommend the authors to polish the paper carefully.
- Can the authors provide some complexity analysis of each module? It may require additional time and space to train the framework comparing to other approaches.
- Unfortunately, the code isn’t available on the provided link.
paper_summary: paper_summary: The authors propose a bidirectional cross-modal ZSL approach CREST. It begins by extracting representations for attribute and visual localization and employs Evidential Deep Learning (EDL) to measure underlying epistemic uncertainty, thereby enhancing the model’s resilience against hard negatives. CREST incorporates dual learning pathways, focusing on both visual-category and attribute-category alignments, to ensure robust correlation between latent and observable spaces. Moreover, the authors introduce an uncertainty-informed cross-modal fusion technique to refine visual-attribute inference.
strength_weakness: strength_weakness: Positive
- The authors proposed a new method targeting at athe zero-shot learning task.
- The authors conduct extensive experiments to demonstrate the effectiveness  of the proposed method.
limitations: limitations: Negative 
- Besides the accuracy performance of the proposed method, the authors should compare the performance of the proposed method and the related methods.
- The motivation of the proposed model is unclear. For example, why did the authors set the the function g()?
- From Table 1, the performance of the proposed method is not significant on the datasets of SUN, and AWA2.
paper_summary: paper_summary: The paper proposes a zero-shot learning framework called CREST that aims to enhance cross-modal visual-attribute alignment for recognizing unseen categories. CREST has three key components: 1) Cross-modal feature extraction using grounding transformers to extract localized visual and attribute representations. 2) Instance-level contrastive learning to align visual representations with categories, accounting for real-world visual variability. 3) Decoupled insight using a meta pattern bank and sparse attention to decouple attribute mappings across categories. It also introduces evidential deep learning for uncertainty measurement and uncertainty-driven visual-attribute fusion. The paper reports experimental results on three benchmark datasets.
strength_weakness: strength_weakness: 1. The paper conducts experiments across three well-known zero-shot learning benchmark datasets (CUB, SUN, AWA2) which is adequate.
2. It uses grounding transformers to extract localized visual and attribute features, going beyond just global image/text features, which is theoretically well-motivated.
3. The formulations and descriptions of the various model components are technically correct and clear.
limitations: limitations: 1. The overall novelty is limited: (1) It looks like an increment of the previous work [1], which presented an attribute→visual Transformer sub-net and a visual→attribute Transformer sub-net (VAT) to learn attribute-based visual features and visual-based attribute features. (2) The use of contrastive learning for aligning vision representations is not novel and has been studied in other works like CE-GZSL [2] (CVPR’21).
2. On the standard ZSL benchmarks CUB, SUN and AWA2, the performance gains over recent previous methods are relatively modest or lower than previous approaches. Considering the limited novelty, the improvement of the method is not significant.
3. The ablation studies could be more comprehensive to tease apart the contributions of different components. The Evidence Deep Learning consists of three important components, but according to Table 2, this paper only analyzes them as a group, and it is not feasible to clearly show the contribution of each component.
4. According to Table 2, The Evidence Deep Learning contributes marginally to performance improvement and there is no analysis for the different components in Evidence Deep Learning, so the novelty and contribution of Evidence Deep Learning in this paper is ambiguous.
5. According to Figure 9, it is not easy to distinguish the performance improvement that comes from the utilization of Evidence Deep Learning since the three visualization results are very similar.

[1] TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning.
[2] Contrastive embedding for generalized zero-shot learning.