Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learning

Authors: xiangyan  qu
jing  yu
keke  gai
jiamin  zhuang
yuanmin  tang
gang  xiong
gaopeng  gou
qi  wu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-18 02:19:50

Abstract: Recent work shows that documents from encyclopedias serve as helpful auxiliary information for zero-shot learning. Existing methods align the entire semantics of a document with corresponding images to transfer knowledge. However, they disregard that semantic information is not equivalent between them, resulting in a suboptimal alignment. In this work, we propose a novel network to extract multi-view semantic concepts from documents and images and align the matching rather than entire concepts. Specifically, we propose a semantic decomposition module to generate multi-view semantic embeddings from visual and textual sides, providing the basic concepts for partial alignment. To alleviate the issue of information redundancy among embeddings, we propose the local-to-semantic variance loss to capture distinct local details and multiple semantic diversity loss to enforce orthogonality among embeddings. Subsequently, two losses are introduced to partially align visual-semantic embedding pairs according to their semantic relevance at the view and word-to-patch levels. Consequently, we consistently outperform state-of-the-art methods under two document sources in three standard benchmarks for document-based zero-shot learning. Qualitatively, we show that our model learns the interpretable partial semantic association. The code is available at https://anonymous.4open.science/r/EmDepart.

Main paper: 
 introduction: 
 Image recognition tasks have achieved significant success relying on enormous manually labeled data. However, it is impractical to collect and annotate all kinds of images. Zero-Shot Learning (ZSL) [31, 41] emerges as a promising paradigm to address this issue. ZSL aims to identify unseen classes by training on a set of seen classes. The key challenge in ZSL is how to leverage auxiliary information to transfer knowledge from seen to unseen classes. Common auxiliary information includes attributes [27, 53] , word embeddings [24, 36, 42, 64] , and category documents [6, 17, 45] . Most work [11, 13, 22, 30, 34] leverages human-annotated attributes as auxiliary information. These methods assume that seen and unseen classes can be defined with the same attributes. However, these attributes are labor-intensive and challenging to scale [47, 65] , which are impossible for many real-world scenarios. To address this issue, other work [3, 21, 46, 63] applies category word embeddings from the pre-trained language model to replace attributes. However, the category name offers limited discriminative information [3, 8] , imposing potential limitations on performance. Recently, some work [28, 37, 38] demonstrates that documents from the encyclopedias serve as valuable sources of auxiliary information, which contain multiple semantic concepts and knowledge from experts.For document-based ZSL, seen and unseen classes are described by a composition of similar semantic concepts (noun phrases that visually describe a class). Aligning basic semantic concepts with corresponding image regions accurately is the key to knowledge transfer. Recent methods [37, 38] apply fine-grained interactions between words (or documents) and image patches to enhance semantic alignment. However, they are designed without considering the partial association between noisy documents and visual-diverse images: 1) Noisy document: Documents from encyclopedias mainly cover many views, e.g. shape, color, habitat, sound, and diet. However, some views may not include visual information (see "9" in the left of Figure 1 ), e.g. sound and diet. These non-visual views are harmful to knowledge transfer. 2) Exhaustive description: Documents comprehensively describe the possible characteristics of the category. However, a single image typically captures part of them. For example, the last image on the right of Figure 1 only shows the shape of the horn, color, and habitat in the antelope document. However, these methods align the entire semantics of documents with images, obtaining a suboptimal alignment. 3) Visual-diverse image content: Due to variations in shooting angles, lighting, locations, and states, images of the same category convey varying semantic concepts from the document (see the right of Figure 1 ). Aligning diverse images with the same document semantics makes it hard to build accurate semantic alignment. Therefore, accurately modeling the partial association between document and image becomes an urgent problem for document-based ZSL.To this end, we propose an Embedding Decomposition and Partial Alignment (EmDepart) network, as illustrated in Figure 2 to extract multi-view semantic concepts from document and image and accurately align the matching concepts. Specifically, the Semantic Decomposition Module (SDM) is proposed to generate multi-view semantic embeddings from visual and textual sides, providing the basic concepts for partial alignment. However, the SDM may generate multiple embeddings with a slight variance, resulting in information redundancy, denoted as feature collapse. To alleviate this issue, we propose the local-to-semantic variance loss to capture unique local details and multiple semantic diversity loss to make each embedding orthogonal to others. Subsequently, we rely on the semantic similarity of every visual and textual view embedding pair to model the partial association. Two losses are introduced to partially align these pairs according to their semantic relevance at the view and word-to-patch levels. Moreover, a novel score is applied to filter out unmatched information to measure semantic similarity accurately at the inference. Since some fine-grained categories are less described in the encyclopedia, we also design a novel prompt strategy to enrich these documents.Our key contributions are as follows. (1) We propose a novel network that decomposes concepts from document and image into multi-view semantic embeddings and aligns them partially according to semantic relevance. This addresses the suboptimal alignment caused by ignoring the partial association in document-based ZSL. It sheds new light on the vision-and-language partial semantic alignment. (2) To solve the issue of information redundancy caused by feature collapse, we introduce the semantic decomposition module with the local-to-semantic variance loss to capture unique local details and multiple semantic diversity loss to enhance orthogonality among the embeddings. These losses also improve the performance of previous methods by 4.1% on average. (3) With comparable training parameters, our model consistently outperforms state-of-the-art methods for document-based ZSL and GZSL settings in three standard benchmarks. It improves performance by 6.0% and 5.8% on average across all metrics under Wiki and Wiki+LLM documents. Moreover, we qualitatively demonstrate that our model learns the interpretable partial semantic association. 
related work: 
 Zero-Shot Learning aims to train on seen classes and generalize to recognize unseen classes [31, 41] . Most work leverages human-annotated attributes [15, 27, 32, 53] as auxiliary information. These methods transfer knowledge by utilizing compatibility functions [1, 2, 9, 44, 58] to map embeddings into a common space, incorporating generative model to generate unseen classes samples [52, 60, 61, 69] , and enhancing semantic alignment between attributes and image regions [11, 12, 22, 30, 34, 50, 62, 66, 67] . However, annotating attributes needs large human resources and deep domain expertise. In contrast, other work [3, 21, 46, 63] applies word embedding from pre-trained language models [24, 36, 42, 64] , which transfers knowledge through the semantic relationship between different categories. Several methods [25, 26, 39, 54] enhance semantic connections by knowledge graphs. However, they achieve poor performance because of the category name with little discriminative information and sensitivity to linguistic issues [3, 8] . In contrast, documents are easy to collect from encyclopedias, which contain multiple semantic concepts. In this work, we improve knowledge transfer for document-based ZSL by accurate partial alignment. Document-based Zero-Shot Learning uses definition-level text corpora from encyclopedias to obtain auxiliary information. Most work [4, 5, 8, 28, 43] utilize document embedding by TF-IDF [45] or large language models [6, 17] as auxiliary information. While several methods [19, 68] enhance embeddings by part detection network, annotated attributes are used to train the detection model. Recently, some work [37, 38] learns fine-grained interactions to enrich semantic embedding. Specifically, I2DFromer [38] trains a model to align image patches with words in global and local compatibility. I2MVFormer [37] aggregates information at the document level to reduce computation cost, aligning with image regions. However, these methods align the entire semantics of documents with images, ignoring the partial association between them. This results in suboptimal semantic alignment. In this work, our EmDepart generates multi-view semantic embeddings and models the partial association to align the matching semantics accurately. Set-based Embedding Methods aims to learn multiple embeddings to alleviate the semantic ambiguity in cross-modal retrieval task, i.e., an image is semantically matched with multiple captions. This is similar to the challenge for document-based ZSL, i.e., the partial association between the document and diverse images. To be specific, PVSE [49] and TVMM [33] learn a set of embeddings by linear combination with local and global features. PCME [14] represents each sample as a probabilistic embedding. The state-of-the-art method [29] explores slot attention [35] to enhance diversity in setbased embeddings and smooth chamfer similarity to solve sparse supervision and set collapsing problems. However, text corpora in document-based ZSL are at definition level (≈ 500 words). It is hard to obtain multiple embeddings with little information redundancy solely through model architecture. Therefore, we propose two losses to enhance the information difference among semantic embeddings, alleviating the problem of feature collapse. 
method: 
 Our Embedding Decomposition and Partial Alignment (EmDepart) network is illustrated in Figure 3 . We first collect documents from encyclopedias and enrich less-described categories by LLMs. The image perceiver and text perceiver extract salient features for ZSL tasks. Then, the visual and textual Semantic Decomposition Modules (SDM) decompose perceived features into multi-view semantic embeddings. We leverage these embeddings to partially align the semantic information at the view and word-to-patch levels. Notations. Zero-Shot Learning (ZSL) aims to train a classifier on seen classes Y 𝑠 to recognize unseen classes Y 𝑢 during the test, where Y 𝑠 ∩ Y 𝑢 = ∅. The training set {(𝒙, 𝒚, 𝒅)|𝒙 ∈ X 𝑠 , 𝒚 ∈ Y 𝑠 , 𝒅 ∈ D 𝑠 } consists of image 𝒙, its label 𝒚 and auxiliary information, i.e., document 𝒅. These documents are from a collection of textual descriptions of seen classes. At test time, another collection of images X 𝑡𝑒𝑠𝑡 , their potential classes Y 𝑡𝑒𝑠𝑡 , and corresponding documents D 𝑡𝑒𝑠𝑡 will be available to evaluate the model. In the ZSL setting, test images are from unseen classes, and for generalized ZSL (GZSL), from seen and unseen classes.Category documents are the theoretical foundation for knowledge transfer in document-based ZSL. Each class (both seen and unseen classes) has a corresponding document that visually describes it. Document Collection from Encyclopedia. Similar to [37, 38] , we leverage the A-Z animals [56] for AWA2 [59] , AllAboutBirds [57] for CUB [53] , and Wikipedia [55] for FLO [40] to collect documents. Following previous methods, we select relevant sections in the encyclopedia to filter potential noises for the ZSL task. Enriching Less-Described Document. Some fine-grained categories are less described in the encyclopedia, such as dogs chihuahua and collie in AWA2, Nighthawk and Green Violetear in CUB, and most classes in FLO. Therefore, we instruct Large Language Models (LLMs) to generate category definitions to enrich these documents by the following prompt:"Now you are a {type} expert. I will give you {type} name, and you need to give detailed visual information about its shape, color, appearance, habitat, etc. I want you to define {class name}. " We use category species as {type}, i.e., animal for AWA2, bird for CUB, and flower for FLO. We concatenate documents from encyclopedias with LLM-generated to serve as the final auxiliary information. To save computation costs, we enrich less-described instead of all categories. More details are shown in the supplementary.Image Perceiver. Given an input image 𝒙, the image perceiver first encodes features by a fixed ViT [18] . Then, a learnable MLP layer with a residual connection maps image features to dimension 𝑟 , extracting crucial visual information for ZSL tasks. The image perceiver outputs [CLS] token 𝑰 𝐶𝐿𝑆 ∈ R 𝑟 as the global image feature and other tokens 𝑰 𝑙 ∈ R 𝑛×𝑟 as patch-wise local image features, where 𝑛 is the number of image patches. Text Perceiver. Given a 𝑚-words document 𝒅, we use GloVe [42] to initiate each word as input features. Similar to previous work [37, 38] , the text perceiver passes these token features through a learnable MLP with dimension 𝑟 (r < 300) to reduce computation cost. We add a learnable [CLS] token to the sequence and input this sequence into the text encoder, which consists of two transformer encoder blocks. The text encoder outputs [CLS] token 𝑻 𝐶𝐿𝑆 ∈ R 𝑟 as global text feature and 𝑻 𝑙 ∈ R 𝑚×𝑟 as word-wise local text features.The semantics in images and documents are from multiple views, such as shape, color, and habitat. We introduce visual and textual semantic decomposition modules (SDM) to aggregate information from the perceived features of each modality and decompose them to generate multi-view semantic embeddings. This process provides the basic semantic concepts for partial alignment.Taking visual SDM as an example, the textual side follows a similar process. As shown in the middle of Figure 3 , SDM contains 𝑙-layer aggregation blocks integrating perceived features through iterations. In the initial iteration, we introduce a set of learnable tokens 𝑬 0 ∈ R 𝑘 ×𝑟 , referred to as view embeddings later, where 𝑘 denotes the number of embeddings (𝑘 ≪ 𝑛). Subsequently, for the 𝑡-th iteration, we feed both 𝑬 𝑡 -1 and local image features 𝑰 𝑙 to the 𝑡-th aggregation block (𝑡 = 1, 2, ..., 𝑙), iteratively refining the visually discriminative information.Each aggregation block aims to aggregate semantics and extract helpful visual information for the ZSL task. For the 𝑡-th aggregation block, we map local image features 𝑰 𝑙 to key 𝑲 and value 𝑽 and view embeddings 𝑬 𝑡 -1 to query 𝑸 by three linear layers. Then, they are fed into an attention mechanism to obtain the Ê𝑡 ∈ R 𝑘 ×𝑟 :EQUATIONwhere 𝑟 ℎ is the dimension of head attention and 𝑾 𝑜 ∈ R 𝑟 ℎ ×𝑟 is a linear layer to map features to the original dimension 𝑟 . Subsequently, we feed Ê𝑡 to a learnable MLP followed by layer normalization, residual connection, and GELU activation [23] , outputting the iterative visual-semantic information refinement of the view embeddings 𝑬 𝑡 :EQUATIONIn the last iteration, we concatenate the 𝑬 𝐿 ∈ R 𝑘 ×𝑟 , the output of the final aggregation block, with 𝑘 repetitions of global image feature 𝑰 𝐶𝐿𝑆 followed by a layer normalization. This operation constraints view embeddings with small within-set variance and outputs the multi-view visual semantic embeddings 𝑩 𝑉 ∈ R 𝑘 ×𝑟 :EQUATIONSimilarly, we obtain textual semantic embeddings 𝑩 𝑇 ∈ R 𝑘 ×𝑟 .Since definition-level corpora contain numerous words (≈ 500), view embeddings are hard to attend diverse semantics only by model architecture. The challenge also appears on the visual side. Specifically, the SDM may generate view embeddings with a slight variance, resulting in information redundancy, denoted as feature collapse. To solve this issue, we introduce two losses in SDM.Local-to-Semantic Variance Loss aims to encourage each view embedding to focus on unique local information (see the first loss in Figure 3 (b) ). Taking the visual side as an example, it enforces different view embeddings to show distinct attention to the same image patch. We make the following variance constraints on attention maps 𝑨 𝑉 in visual SDM, a dot product between 𝑸 and 𝑲 :EQUATIONwhere 𝒂 𝑡 𝑗 denotes the attention between visual view embeddings and the 𝑗-th image patch in the 𝑡-th aggregation block. The 𝑙 and 𝑛 are the number of aggregation blocks and image patches, respectively. We offer a constant value 𝛾 for constraints, which ensures view embeddings with a certain diversity to the attention of the same patch. The 𝜖 is a small scalar to maintain numerical stabilities. Similarly, we penalize attention maps 𝑨 𝑇 between textual view embeddings and each word token. This loss is formulated as:EQUATIONAfter 𝑙 iterations, each visual and textual view embedding carries distinct semantic information, establishing the foundation for decomposing information.Multiple Semantic Diversity Loss aims to enhance the information decoupling among view embeddings. It forces minimal semantic redundancy between view embeddings by making each embedding orthogonal to others, shown in the second loss of Figure 3 (b). Notably, view embeddings of the final output (𝑩 𝑉 and 𝑩 𝑇 ) contain the global feature, which may invalidate the orthogonality constraint. Therefore, we penalize redundancy among the output of the final aggregation block, i.e., 𝑬 𝐿 . Specifically, we normalize each 𝑬 𝐿 and calculate the cosine similarity between them, yielding the redundant matrix 𝑴 𝑉 = 𝑬 𝐿 (𝑬 𝐿 ) 𝑇 . The loss penalizes the visual and textual redundant matrix (denoted as 𝑴 𝑇 ) to approximate the identity matrix I ∈ R 𝑘 ×𝑘 via an 𝑙 2 -norm minimization:EQUATIONwhere 𝑴 𝑇 is computed in a similar way. This objective ensures that different view embeddings maintain orthogonality, characterized by non-diagonal cosine similarity values converging towards zero.To summarize, since the L 𝑣𝑎𝑟 penalizes view embeddings to focus on different local information and the L 𝑑𝑖𝑣 constrains each embedding to be orthogonal to others, the SDM generates decomposed view embeddings with distinct semantic information.To accurately model the partial association between documents and images, we introduce two losses to align the matching semantic concepts at the view and word-to-patch levels. Decomposed Semantic Alignment aims to align decomposed visual and textual view embeddings according to their semantic relevance. We rely on the semantic similarity between visual and textual view embedding pairs to model the partial association by the Smooth Chamfer [29] function. We first review the Smooth Chamfer, which assigns distinct weights to every document-image embedding pair based on similarity:EQUATIONwhere LSE(𝒃, 𝑩) is smooth approximation for maximum cosine similarity between vector 𝒃 and elements in set 𝑩. Taking LSE(𝒃 𝑇 , 𝑩 𝑉 ) as an example, it is formulated as:EQUATIONwhere cos(•) is the cosine similarity. We introduce a cross-entropy loss to encourage image 𝒙 and corresponding document 𝒅 to be closer than other pairs:EQUATION)where 𝜏 is a temperature scalar. The L 𝑔𝑙𝑜𝑏𝑎𝑙 is designed to smoothly align each visual embedding in 𝑩 𝑉 with the most similar element from textual embeddings 𝑩 𝑇 , and vice versa (see the third loss in Figure 3(b) ). This process models the partial association between visual and textual spaces, which embodies the fact that an image is reflected as part of semantics in the document. Consequently, we align the two spaces more accurately.Local Semantic Alignment aims to apply interactions between image patches and word tokens for fine-grained semantic alignment (see the fourth loss in Figure 3(b) ). It provides the basic semantic concepts for partial semantic alignment and discriminative information for fine-grained classification. Similar to [37, 38] EQUATIONTraining. Our EmDepart is optimized with the following loss:EQUATIONwhere 𝜆 𝑙𝑜𝑐𝑎𝑙 , 𝜆 𝑣𝑎𝑟 , and 𝜆 𝑑𝑖𝑣 are hyper-parameters. The joint training enhances EmDepart to generate multi-view semantic embeddings with information decoupling and accurately align visual and textual space to a common semantic space according to the matching information, significantly improving knowledge transfer.During the inference, a partial score is proposed to filter out unmatched information to measure semantic similarity accurately. Partial Score Function. In Figure 1 , it is evident that an image semantically matches a subset of the semantic concepts within the document. Similarly, we limit the similarity computation solely to document-image semantic pairs with the highest 𝑝 similarity values, where 𝑝 < 𝑘. Specifically, we select the top 𝑝 similarity values between each visual view embedding and all textual view embeddings, yielding a total of 𝑝 × 𝑘 pairs. Subsequently, a similar process is applied to each textual view embedding, resulting in the selection of 𝑝 × 𝑝 pairs. We denote this process as TopCos(𝒙, 𝒅, 𝑝) function, effectively filtering out unmatched information between documents and images. The smooth chamfer is then used to compute the similarity among these chosen pairs:EQUATION)Inference. Given an input image 𝒙, we obtain a prediction ŷ that yields the highest partial score 𝑆 𝑝 (𝒙, 𝒅 ′ ) among unseen classes for ZSL, i.e., D = D 𝑢 , and among both seen and unseen classes for GZSL, i.e., D = D 𝑠 ∪ D 𝑢 : ŷ = arg max EQUATION) 
experiments: 
 Datasets. We evaluate on three benchmark datasets, i.e., a coarsegrained dataset Animals with Attributes2 (AWA2) [59] , two finegrained datasets Caltech-USCD Birds-200-2011 (CUB) [53] and Oxford Flowers (FLO) [40] . The seen-unseen class division is from Proposed Split [59] . We use documents instead of human-annotated attributes in datasets as auxiliary information. Evaluation Metrics. Following [59] , we measure the average perclass top-1 accuracy (T1) on unseen classes for ZSL. For GZSL, we present the per-class mean accuracy on seen (𝑆) and unseen classes (𝑈 ) as well as their harmonic mean 𝐻 = 2 × 𝑈 × 𝑆/(𝑈 + 𝑆). Implementation Details. Similar to [37, 38] , we utilize the ViT-B/16 [18] pre-trained on ImageNet 1K [16] as the visual backbone. If not noted otherwise, we show the performance of ChatGPT [20] as LLMs. Hyperparameters are optimized by grid search in the validation split. Once the hyperparameters are confirmed, we merge the validation with the training split to obtain the performance on the test split. We also apply calibrated stacking [10] for GZSL to trade-off calibration degrees, reducing the bias towards seen classes. More details are available in the supplementary.Comparison with SOTA in Document-based ZSL. In Table 1 , we compare our EmDepart with state-of-the-art (SOTA) methods in document-based ZSL. For a fair comparison, we evaluate methods with the same text and image perceivers. EmDepart outperforms previous methods across all metrics (T1 and H) regarding ZSL and GZSL settings on all datasets. It confirms that modeling the partial association is beneficial for accurate semantic alignment. The previous SOTA methods [37, 38] align complete semantics in documents with images, thus hindering knowledge transfer. Wiki vs Wiki+LLM Documents. With Wiki documents, EmDepart achieves optimal performance compared to previous methods. Notably, EmDepart with Wiki outperforms SOTA methods with Wiki and LLM regarding T1 and H on AWA2 and FLO. It confirms that modeling the partial association is significant for knowledge transfer. Besides, we see a performance improvement from Wiki to Wiki+ChatGPT. This is because visual descriptions generated by LLMs enrich semantic information in less-described classes.b 1 b 2 b 3 b 4 (a) w/o L 𝑑𝑖𝑣 and L 𝑣𝑎𝑟 . b 1 b 2 b 3 b 4 (b) L 𝑑𝑖𝑣 . b 1 b 2 b 3 b 4 (c) EmDepart.Comparison with SOTA in Set-based Embedding Methods. In Table 2 , we replace SDM with TVMM [33] and S-Chamfer [29] , the SOTA set-based embedding methods. Since documents are at the category level, it is challenging to rely solely on model architecture to produce view embeddings with little redundant information. SDM achieves the optimal performance by incorporating L 𝑑𝑖𝑣 and L 𝑣𝑎𝑟 to enhance information difference among view embeddings. Similarly, TVMM and S-Chamfer improve performance after adding L 𝑑𝑖𝑣 and L 𝑣𝑎𝑟 , facilitating differences between embeddings. In Figure 4 (a), we show the feature collapse between view embeddings, i.e., embeddings have little variance, resulting in information redundancy. It is harmful to model the partial association with the same embeddings. To solve this issue, we introduce L 𝑣𝑎𝑟 to make each embedding attend to distinct local details and L 𝑑𝑖𝑣 to penalize embeddings orthogonal to others. In Figure 4 (a-c), view embeddings carry more distinct information (at more different positions) with the L 𝑣𝑎𝑟 and L 𝑑𝑖𝑣 . Quantitatively, we introduce the circular variance 𝑆 𝑣𝑎𝑟 = 1 -∥ 𝒃 ∈𝑩 𝒃/|𝑩|∥ 2 to analyze the information difference of view embeddings. TVMM [33] and S-Chamfer [29] lead to feature collapse under category-level corpora in Figure 5 . After adding L 𝑣𝑎𝑟 and L 𝑑𝑖𝑣 , we improve these methods performance and increase the information difference among view embeddings.In Figure 6 , we qualitatively show that our model learns the interpretable partial semantic association. It contains the visual-semantic decomposition to offer basic semantic concepts and partial semantic alignment according to the matching information.Visual-Semantic Decomposition. We see that different view embeddings focus on distinct information in each modality. Utilizing the giraffe as an example, EmDepart focuses on appearance (in 𝑏 1and 𝑏 2 𝑉 ), habitat (in 𝑏 4 𝑉 ), and global information (in 𝑏 3 𝑉 ) for the visual side. Similarly, there are textual descriptions on color (in 𝑏 1 𝑇 ), appearance and shape (in 𝑏 2 𝑇 and 𝑏 3 𝑇 ), and habitat (in 𝑏 4 𝑇 ). This verifies that the SDM decomposes semantics from images and documents and generates multi-view semantic embeddings. Partial Semantic Alignment. On the similarity matrix of 𝑆 𝑠𝑐 (𝒙, 𝒅), we observe the accurate semantic alignment between document and image. In particular, since the second giraffe image does not represent the visual content about habitat and body, (𝑏 2 𝑉 , 𝑏 3 𝑇 ) and (𝑏 4 𝑉 , 𝑏 4 𝑇 ) has a high similarity in the first image but low in the second. In the first red tiger lily, (𝑏 2 𝑉 , 𝑏 2 𝑇 ) has the highest score, while (𝑏 1 𝑉 , 𝑏 1 𝑇 ) has the highest score in the second orange one. This is consistent with the fact that 𝑏 1 𝑇 pays more attention to "orange" and 𝑏 2 𝑇 focuses more on "red".We ablate key components of EmDepart in Table 3 . For all models, we leverage documents from Wiki+LLM as auxiliary information.Ablation on Loss Functions. In row b), we see a significant drop in performance on CUB and FLO. This is due to the lack of interaction between patches and words, which offers discriminative information for fine-grained classification. The performance of removing L 𝑑𝑖𝑣 decreases more than L 𝑣𝑎𝑟 in rows c) and d). This is due to L 𝑣𝑎𝑟 constraining the variance in attention blocks, which means feature collapse may exist after the MLP projection in SDM. Row e) shows a further decrease in performance, which indicates the complementary of L 𝑑𝑖𝑣 and L 𝑣𝑎𝑟 . Row f) achieves the worst performance, further verifying the effectiveness of our losses. Ablation on Score Functions. The performance degrades in row g) due to the partial score filtering out unmatched semantic information, measuring similarity accurately in the inference. Rows h) and i) ablate Smooth Chamfer with the average distance in [14] and maximum distance in [49] . Smooth Chamfer performs better because it overcomes problems posed by sparse supervision in maximum distance and feature collapse in average distance. Ablation on Proposed Modules. In row j), we remove the global feature in Eq.3. It performs worse as view embeddings using only local features may introduce a large variance, leading to overfitting on seen classes. In contrast, global features in Eq.3 ensure that the variance remains within a controlled range. In row k), we show the result of removing the SDM and leveraging the global feature to align entire semantics of documents to images like [37, 38] . The performance decreases due to the suboptimal semantic alignment, ignoring the partial association. Besides, performance drops in row l) when the visual perceiver lacks a residual connection, which preserves the original visual knowledge inherent to ViT [18] , a crucial factor for knowledge transfer. Ablation of Different LLMs. In Table 4 , we show the effect of different LLMs, consistently improving the performance compared to Wiki documents. It verifies the effectiveness of enriching lessdescribed documents. The performance in FLO improves significantly due to the lack of detailed descriptions for most classes in Wiki. The ChatGPT [20] achieves the best result, which generates more detailed descriptions with rich semantics. Effect of Loss Weights. In Figure 7 (a-c), as the 𝜆 𝑙𝑜𝑐𝑎𝑙 , 𝜆 𝑣𝑎𝑟 , and 𝜆 𝑑𝑖𝑣 increase, the model consistently improves performance compared to the value set as 0. It confirms the effectiveness of our losses. In Figure 7 (a), the performance rises consistently on two datasets, which verifies that fine-grained interactions are essential for knowledge transfer. In Figure 7 (b-c), we see a uniform performance improvement in AWA2 with the increase of 𝜆 𝑣𝑎𝑟 and 𝜆 𝑑𝑖𝑣 . It verifies that L 𝑣𝑎𝑟 and L 𝑑𝑖𝑣 aid the model to generate multi-view semantic embeddings with information decoupling, facilitating the partial semantic alignment. The results in FLO demonstrate the same conclusion when 𝜆 𝑣𝑎𝑟 ≥ 0.5 and 𝜆 𝑑𝑖𝑣 ≥ 1.0. Effect of Number of View Embeddings 𝑘 in SDM. In Figure 7 (d), we report the H when varying 𝑘 from 1 to 8. Besides, we compare with 𝑘 = 0, the baseline without SDM, i.e., leveraging global feature as the view embedding. As the 𝑘 increases, we see progressive performance improvement across both datasets. This is due to multiview embeddings capturing distinct semantics. However, a high 𝑘 may be biased to seen classes, thus harming the performance.In Table 5 , we compare the trainable parameters, the time for training one epoch and inference single image, and the performance with previous methods. With comparable computation cost, our method outperforms previous methods [37, 38] . It verifies that the performance improvement is due to more accurate semantic alignment instead of increased parameters. Moreover, after adding the SDM, the performance improves significantly with a slight increase in  
conclusion: 
 Our EmDepart models the partial association between documents and corresponding images, accurately aligning visual and textual space based on the matching information. By introducing local-tosemantic variance loss and multiple semantic diversity loss, SDM generates multi-view semantic embeddings. These losses also help the previous methods solve the feature collapse problem. Moreover, we introduce two losses to partially align the semantic concepts between documents and images at the view and word-to-patch levels. In addition, we propose a partial score to filter out unmatched information and evaluate semantic similarity accurately. With comparable training parameters, EmDepart outperforms SOTA methods on three benchmarks for document-based ZSL. Qualitatively, our model learns the interpretable partial semantic association. 
 
 Appendix:  


Decision: This paper proposes a novel network to partially align textual and visual space according to their semantic-relevant degree for Zero-Shot Learning (ZSL). All reviewers unanimously recommend accepting the paper as the rebuttal has properly addressed their concerns.

Reviews: review: paper_summary: paper_summary: This paper focus on the problem of partial association between noisy documents and visual-diverse images in zero-shot learning (ZSL). They propose an Embedding Decomposition and Partial Alignment (EmDepart) network to extract multi-view semantic concepts from document and image and accurately align the matching concepts. Furthermore, to solve the issue of information redundancy caused by feature collapse, the authors introduce the semantic decomposition module with the local-to-semantic variance loss and multiple semantic diversity loss.
strength_weakness: strength_weakness: - This paper is well-written and easy to following.
- The research problem is meaningful and the proposed solution is reasonable.
- The proposed method has a strong performance across common ZSL datasets.
- Extensive experiments further illustrate the effectiveness of the proposed method.
limitations: limitations: - Could the Semantic Decomposition Module (SDM) be adapted to attributes-based ZSL methods?
paper_summary: paper_summary: This paper proposes a novel network, termed Embedding Decomposition and Partial Alignment (EmDepart), that partially aligns textual and visual space according to their semantic-relevant degree for Zero-Shot Learning (ZSL). Specifically, the Semantic Decomposition Module (SDM) is proposed to generate multi-view semantic embeddings from visual and textual sides, providing the basic concepts for partial alignment. And the local-to-semantic variance loss is to capture unique local details and multiple semantic diversity loss is to make each embedding orthogonal to others. Moreover, a novel score is applied to filter out unmatched information to measure semantic similarity accurately at the inference. The experimental results on three benchmark datasets show its effectiveness.
strength_weakness: strength_weakness: 1. This paper is well-written and easy to follow.
2. The paper focuses on the limitations of global alignment between documents and images, which has practical research value, and the method proposed is also relatively novel.
3. The proposed method is effective, and sufficient experiments have been conducted for analysis.
limitations: limitations: 1. There are some details errors, such as some abbreviations (CLSN, IMG) not being mentioned.
2. Local alignment between vision and semantics has been studied in some works using attributes as auxiliary information (such as Transzero [1], DAZLE [2] and Gndan[3]), but the related research has not been comparatively discussed.


> - [1] Chen S, Hong Z, Liu Y, et al. Transzero: Attribute-guided transformer for zero-shot learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(1): 330-338.
> - [2] Huynh D, Elhamifar E. Fine-grained generalized zero-shot learning via dense attribute-based attention[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 4483-4493.
> - [3] Chen S, Hong Z, Xie G, et al. Gndan: Graph navigated dual attention network for zero-shot learning[J]. IEEE transactions on neural networks and learning systems, 2022.
paper_summary: paper_summary: This paper proposes a novel network that partially aligns textual and visual space according to their semantic-relevant degree. 
This addresses the suboptimal alignment caused by ignoring the incomplete semantic equivalence between documents and images.
strength_weakness: strength_weakness: 1. The Semantic Decomposition decomposes features into multi-view semantic embeddings, facilitating the partial alignment of matching semantics.  And two losses are introduced to enhance distinctiveness and orthogonality among embeddings, addressing the issue of feature collapse and improving previous methods.
2. State-of-the-art results are achieved on three benchmarks, demonstrating the effectiveness of the approach.
3. The model learns interpretable partial semantic associations, which are visualized effectively.
limitations: limitations: 1. Limited Comparative Analysis: The paper could benefit from more comparative experiments, such as comparisons with more ZSL methods, to further validate the approach.
2. Robustness Analysis: Evaluating the model's robustness could provide a more comprehensive understanding of its real-world applicability, and a sensitivity analysis on the number of view embeddings k could help in understanding model stability.
paper_summary: paper_summary: This paper proposes a method called EmDepart (Embedding Decomposition and Partial Alignment) for document-based zero-shot learning (ZSL). The proposed method introduces the Semantic Decomposition Module (SDM) to generate multi-view semantic embeddings and uses the local-to-semantic variance loss and multiple semantic diversity loss to capture unique local details and enhance orthogonality among the embeddings. It also applies partial semantic alignment at the view and word-to-patch levels. The experimental results show that EmDepart outperforms state-of-the-art methods in document-based ZSL on three standard benchmarks.
strength_weakness: strength_weakness: 1. EmDepart proposes an innovative approach to solve the problem of partial document-image correlation, introducing the concepts of multi-view semantic embedding and partial alignment, which improves the effect of zero-shot learning.
2. The paper is well-organized and easy to read.
limitations: limitations: 1. The semantic decomposition module relies on perceived features from different modalities, such as shape, color, and habitat.  However, this approach may not capture all the nuances and complexities of the semantic concepts, leading to a limited representation of the semantics.
2. It does not consider the partial association between noisy documents and visually diverse images. This can lead to suboptimal alignment, especially when dealing with non-visual views in documents or diverse visual content in images.