Semantic Distillation from Neighborhood for Composed Image Retrieval

Authors: yifan  wang
lei  li
wuliang  huang
chun  yuan
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:59:53

Abstract: The challenging task composed image retrieval targets at identifying the matched image from the multi-modal query with a reference image and a textual modifier. Most existing methods are devoted to composing the unified query representations from the query images and texts, yet the distribution gaps between the hybrid-modal query representations and visual target representations are neglected. However, directly incorporating target features on the query may cause ambiguous rankings and poor robustness due to the insufficient exploration of the distinguishments and overfitting issues. To address the above concerns, we propose a novel framework termed SemAntic Distillation from Neighborhood (SADN) for composed image retrieval. For mitigating the distribution divergences, we construct neighborhood sampling from the target domain for each query and further aggregate neighborhood features with adaptive weights to restructure the query representations. Specifically, the adaptive weights are determined by the collaboration of two individual modules, as correspondence-induced adaption and divergence-based correction. Correspondence-induced adaption accounts for capturing the correlation alignments from neighbor features under the guidance of the positive representations, and the divergence-based correction regulates the weights based on the embedding distances between hard negatives and the query in the latent space. Extensive experimental results and ablation studies on CIRR and FashionIQ validate that the proposed semantic distillation from neighborhood significantly outperforms baseline methods.

Main paper: 
 introduction: 
 Achieving mutual understanding across diverse modalities (such as images, texts, and videos) [4, 7, 23] has long been a fundamental concern in the field of artificial intelligence research. With the remarkable development of social media platforms, the trend towards utilizing multimodal inputs for user queries is gaining momentum for a more convenient and efficient expression of users' requirements. However, traditional image search [45, 49] and textto-image retrieval [2, 26] could not support this desire to comprehend the multi-modal query directly. The task of Composed Image Retrieval (CIR) [1, 3, 9, 24, 38] naturally arises out of necessity to search for the required images given the input composed of a reference image and a sentence describing the modifications on the query image. Through attaining high-level conceptual comprehension and implicit semantics, researches on the CIR promote the related applications, e.g., vision reasoning [17, 21] and change captioning [36, 37] , and drive the growth of cognitive intelligence.Despite the remarkable advancements [5, 16, 47] , this interactive composed image retrieval still poses a significant challenge. The contents in query images and modifiers complement each other for they form a complete user's requirement, yet they also conflict with each other for the textual information implying modifications on images. Drawing upon the above facts, the model involves an effective combination of the semantics to retain and alter, to bring it closer to the target feature within the shared subspace via appropriate measurement metrics. To tackle the aforementioned obstacles, prior approaches can be categorized into two classes, i.e., compositional learning and target-guided learning. The first category [9, 20, 24, 44] resorts to global feature fusion or synthesizes local alignment clues from the query images and query texts to obtain the unified representations for the hybrid-modal query. Under the guidance of the target images, the second category [13, 27, 41] incorporates the visual features in the target domain into similarity measurements to achieve sufficient interactions within the semantic triplets i.e., query images, modification sentences, and target images. For example, ARTEMIS [13] decomposes the query-totarget similarity into implicit similarity to compare the correlated triplet and explicit similarity to measure the shared characteristics between query text and the target image, yet the guidance of targets usually entails significant time consumption and computation cost especially in testing.Traditional Compositional Methods (eg. Combiner Although the abovementioned methods achieve advancement, several crucial issues have been disregarded in the prior research. Firstly, notable distribution discrepancy as shown in Figure 1 is observed between the distributions of query and target, since the target features originate from the visual modality and the query features are mapped in the mixed-modal composition space. The heterogeneous gap could accumulate statistic errors for the divergent distributions and cause misleading alignments, which further impairs the retrieval performance. Secondly, as the annotators for the modifications mainly concentrate on the differences between one-to-one pairs of the reference images and target images [11, 20] , an abundance of candidate images conveying visual concepts that are coherent with the multi-modal query are unlabeled as positive instances. For example, in Figure 1 , several pictures showing a man seizing the crab that fits the query demands are regarded as negative instances and punished in traditional compositional methods when they are embedded close to the anchor query embeddings. Considering that most previous work exploits batch-based classification loss as the main optimization objective, these false negative samples with semantic correlations would misdirect the model to overfit the noises. Thirdly, subtle semantic differences remain to be explored and a robust model should be aware of the distinctions between the query and the negative examples which are similar to the target image. Yet existing frameworks still make ambiguous predictions when facing hard negative examples. As shown in Figure 1 , though the images with high ranking results (in the purple dashed circle) resemble the query image and target image to some extent, they conflict with the "holding the crab" or "a person" that the text cues in the query domain require. Overall, the above limitations of distribution discrepancy, noisy supervision and model discrimination entail a systematic approach for this task.For addressing the above issues, we present a novel method named SemAntic Distillation from Neighborhood (SADN) to balance the correlation extractions from false negatives and discrimination awareness from neighborhood for composed image retrieval. Figure 2 displays the whole architecture of the proposed approach, which is comprised of three major processes. After extracting features for the target images and the query with reference image and text modifier, we construct neighborhood sampling from the target domain for each query based on the initial similarity ranking. To alleviate misdirecting effects to drive correlated candidates apart from the query for the noisy annotations, the correspondence-induced adaption is introduced to capture correlations from the neighbor representations with the supervision of target representations. Under the adaption constraint from the query to the target, shared characteristics are strengthened during optimization. For promoting the discriminations between matched instances and hard negatives, we exploit Mahalanobis distance measurement in the mixedmodal space to evaluate the distribution divergences of each neighbor sample from the query. The correspondence-induced adaption and the divergence-based correction jointly compose the weights for the neighbor representations, and samples in the neighborhood are adaptively aggregated on the query features to encourage the query-to-target interactions across different domains. Extensive experiments and ablation studies validate the effectiveness of distilling semantics from the constructed neighborhood on Fashion-IQ [42] and CIRR [29] .In summary, the contributions of the SADN are listed as:• We propose a novel Semantic Distillation from Neighborhood approach dubbed SADN for composed image retrieval, which incorporates shared semantic characteristics from the targets into query representations to bridge the distribution discrepancy between queries and targets. • We construct the neighborhood for each query, which captures inherent semantic correlations and filters irrelevant components from the target domain to reconstruct the query features with adaptive weights for improving the robustness of the model. • We apply correspondence-induced adaption with the guidance of target representations to distill alignments between query and target, in collaboration with divergences measurement to evaluate the distribution disparity for refining similarity ranking. 
related works 2.1 composed image retrieval: 
 Composed image retrieval [3, 24, 38, 39, 41] targets identifying the hidden intention of user queries based on the reference images and sentences describing modifications. TIRG [38] was the first to propose the paradigm of composing reference images and modifiers and measuring the distances between candidate images and the query compositions, which was further refined by [20, 44] . Specialized in the fashion domain, FashionVLP [15] utilized the prior knowledge from large multi-modal corpora to guide fashion matching in the transformer structure with multiple visual context layers. Anwaar [1] and Kim [22] implemented the composed image retrieval into symmetrical bi-directional matching with an additional reverse retrieval from the target to query to strengthen the supervision through reversible transformations. TG-CIR [41] introduced knowledge distillation from the target-guided teacher module to lead the similarity distributions in the student network without targets. To deal with uncertainty issue by unlabeled candidates, Chen et al. [10] designed an uncertainty modeling module to learn the tolerable ranges of features with a regularization to ensure the alignments. In summary, most previous works mainly focus on the combinations of query image and modification sentences, and extracting fine-grained matching components from the query to the target, but they ignore the potential correlations lying in the false negative candidates and discriminations between hard negatives and target samples. Through enhancing the neighborhood representations integrated on the query features, the proposed SADN aims to magnify the differences between highly similar instances while addressing the issue of noisy annotations.The goal of deep metric learning is to guarantee that similar inputs are mapped to points close in distance while dissimilar inputs are mapped to points far apart in the embedding space. Due to the high dimensionality of the feature space after deep neural networks, one of the core issues for deep metric learning is the efficient computation of pairwise distances in the hidden space. To address this issue, various approximation techniques have been proposed, including triplet loss functions [33] , siamese architectures [25] , and contrastive learning [18, 31] . The advances in metric learning have led to significant improvements in a range of applications, including image retrieval [6] , representation learning [50] , and clustering. Specifically, Lim et al. [28] designed a novel hypergraph-based loss function to incorporate multiple semantic tuples into the edges, which improved class-discriminative semantic relation learning for the image classification task. To avoid false negative sampling from the negative examples, adaptive false negative elimination and attraction [43] solved the negative sampling issue by selecting instances based on semantic distances from the reference sentence representations. However, there are still issues unsolved when applying deep metric learning in the downstream tasks including the impacts of outliers and noisy labels. In this work, we exploit a flexible trade-off mechanism to balance correcting matching triplets and mining the hard negative samples to promote the discrimination of the model. 
methodology 3.1 problem definition: 
 The task of composed image retrieval could be formulated as minimizing the distances between query embeddings and pairwise target embeddings. To formulate this retrieving process, the reference image, modification sentences, and the candidate images are denoted as I 𝑞 , T 𝑞 , and I 𝑡 , respectively. According to the annotations of the datasets for CIR, I 𝑖 𝑞 , T 𝑖 𝑞 and I 𝑖 𝑡 construct a matched triplet from the query to the target. Note that I 𝑖 𝑞 and T 𝑖 𝑞 are always bound together and the model is expected to find the identical I 𝑖 𝑡 for the 𝑖-th query (I 𝑖 𝑞 , T 𝑖 𝑞 ) during validating and testing. In this work, the images and sentences are encoded through pretrained CLIP visual encoders and textual encoders [32] to obtain the primal representations. The reference image I 𝑖 𝑞 , the modification sentence T 𝑖 𝑞 and the target image I 𝑖 𝑡 are embedded as 𝑰 𝑖 ∈ R 𝑑 , 𝑴 𝑖 ∈ R 𝑑 and 𝑻 𝑖 ∈ R 𝑑 , respectively. Therefore, the goal of this interactive retrieval task could be interpreted as maximizing the similarity measurement between the hybrid-modal query embeddings and the corresponding target embeddings in the latent space, as:EQUATIONwhere 𝑓 (𝑰 𝑖 , 𝑴 𝑖 , Θ) refers to composition functions to generate query features for reference image 𝑰 𝑖 and modification text 𝑴 𝑖 with learnable parameter Θ. 𝑔(𝑻 𝑖 , Ψ) is the visual encoder with parameter Ψ for the target image. In this work, we simply use the original target features for measurement, i.e., 𝑔(𝑻 𝑖 , Ψ) = 𝑻 𝑖 . 𝜅 (𝑓 (𝑰 𝑖 , 𝑴 𝑖 , Θ), 𝑔(𝑻 𝑗 , Ψ)) denotes the kernel functions to calculate the similarity score between the 𝑖-th query (composed of 𝑰 𝑖 and 𝑴 𝑖 ) and the 𝑗-th target. The Eq. 1 is optimized to ensure that 𝜅 (𝑓 (𝑰 𝑖 , 𝑴 𝑖 , Θ), 𝑔(𝑻 𝑖 , Ψ)) > 𝜅 (𝑓 (𝑰 𝑖 , 𝑴 𝑖 , Θ), 𝑔(𝑻 𝑗 , Ψ)), where 𝑗 ≠ 𝑖.Prior models usually design delicate mechanisms to combine features of the reference images and modification texts to obtain the query representations and then rank the distances between the query and target representations in the embedding space. Nevertheless, since the reference images and modification sentences may contain conflict semantics in most cases, we argue that overemphasis on the combination of the multi-modal information in the query could bring redundancy. Since textual information describes the alterations on the query images and an overly complicated fusion scheme could cause more noise and ambiguous semantic learning.In the proposed SADN, we apply a concise compositor to yield the raw query features and calculate the initial similarities between the query and target to select neighbor target instances for each query.The neighborhood construction based on the query-to-target ranking is the prerequisite for fine-grained comparison and measurement in subsequent learning to improve the model discrimination.Besides, the neighbor features selected among the candidate images could mirror the concept of query instances lying in the target image domain and guide the query representation to bridge the distribution differences from the query to the target domain. Learning. Given the extracted visual and textual features from the query, an effective compositor to merge the visual information from the reference images and textual semantics from modifiers is essential for obtaining a unified query representation. Since the contributions of the image and text modalities may vary from instances to instances, a weighted visual and textual representation is an intuitive solution to adjust the proportions of different modalities in composing the query features.In this work, we follow the Combiner [3] to integrate the embeddings of concatenation of query image and query text features on the weighted raw features 𝐼 𝑖 and 𝑀 𝑖 . The composed query feature 𝑄 𝑖 ∈ R 𝑑 could be formulated as:EQUATION= 𝛼𝑀𝐿𝑃 𝐼 (𝑰 𝑖 ) + (1 -𝛼)𝑀𝐿𝑃 𝑀 (𝑴 𝑖 ) + 𝐹𝐶 (⊕(𝑰 𝑖 , 𝑴 𝑖 )),where ⊕ is the concatenation operation and 𝛼 is the weight controlling the contributions of the visual component in the composed query. 𝑀𝐿𝑃 𝑋 and 𝐹𝐶 are the abbreviation of Multi-Layer Perceptron for 𝑋 modality and fully-connected layer respectively.As introduced above, we construct the neighborhood in the target domain for each query through the ranking of the query-to-target similarities to reflect the query features and guide the cross-domain alignments. On the basis of the unified query representations 𝑸 𝑖 , the computation of the initial similarity adopts the widely-used cosine similarity as𝑠 𝑖 𝑗 = 𝑸 ⊤ 𝑖 •𝑻 𝑗 ∥𝑸 𝑖 ∥ ∥𝑻 𝑗 ∥ .For the 𝑖-th query, we simply use 𝑺 𝑖 = {𝑠 𝑖1 , 𝑠 𝑖2 , ..., 𝑠 𝑖𝑁 } ∈ R 𝑁 to refer to the similarity list containing the similarities between the 𝑖-th query and all the 𝑁 candidate images in the minibatch. Through ranking the similarities by the descending order, we would screen the top-𝐾 candidate images that exhibit a significant degree of similarity with the query as the neighborhood setS 𝑁 𝑖 = { 𝑗 |𝑠 𝑖 𝑗 ∈ TOP(𝑠 𝑖 𝑗 , 𝐾)}.Afterward, the corresponding neighbor representations 𝑵 𝑖 ∈ R 𝐾 ×𝑑 for the 𝑖-th query are sampling from the visual features in the target domain based on the neighborhood set S 𝑁 𝑖 , which is constituted as follows:EQUATION)where ⊕ is the concatenation operation and 𝑵 𝑗 𝑖 ∈ R 𝑑 represents the 𝑗-th candidate images features in the 𝑖-th query neighborhood.The unified query feature 𝑸 𝑖 only integrates features from query images and modification texts while it lacks the guidance of the target image to align semantics explicitly. The neighbor features 𝑵 𝑖 obtained in Sec. 3.2, originating in the target domain, possess the following characteristics. On one hand, as the neighbor features share high similarity scores with the query, they also exhibit a strong inclination to be false negative samples for the query. Actually, an inherent issue with this interactive task is that multiple candidate images corresponding with the user intents miss annotations to be labeled as positive samples, which could not be neglected. On the other hand, neighbor features inevitably have some nuance to the target features and the retrieval model is expected to be capable of detecting subtle differences and be aware of the real user requirements when dealing with hard negatives. Considering the above two aspects, we design the Neighbor-Aware Aggregation decomposed as two branches, i.e., Correspondence-Induced Adaption to distill correlated semantics from false negatives in the neighborhood, and Divergence-based Correction to percept the distribution differences from hard negatives.As aforementioned, this branch prioritizes the guidance from the target features to steer the adaption from the hybrid-modal distributions in the query domain towards visual distributions in the candidate image domain. Specifically, we first establish dual alignments including alignment embeddings of each neighbor sample and target feature 𝑬 𝑖 , and the alignment embeddings of each neighbor sample and query feature Ẽ𝑖 , to separately measure the resemblances from the neighborhood to the target and query examples, which is calculated via:EQUATIONwhere ⊙ means pointwise product. We denote 𝑬 𝑖 = {𝑬 1 𝑖 , 𝑬 2 𝑖 , ..., 𝑬 𝐾 𝑖 } ∈ R 𝐾 ×𝑑 and Ẽ𝑖 = { Ẽ1 𝑖 , Ẽ2 𝑖 , ..., Ẽ𝐾 𝑖 } ∈ R 𝐾 ×𝑑 for simplification. Since the alignment embedding 𝑬 𝑖 evaluates the correlations between the neighbor instances and target instance, we could derive the correspondence score to infer the correlations between the neighbor instance and the 𝑖-th query among all the neighbors via:EQUATION)where Conv denotes a 1 × 1 convolutional layer to encode the alignment features into the correspondence score 𝒄 𝑗 𝑖 for the 𝑗-th instance in the 𝑖-th query's neighborhood. A high correspondence score 𝒄 𝑗 𝑖 implies that the 𝑗-th candidate image has great semantic overlap with the 𝑖-th target image. Hence, the 𝑗-th candidate feature in the 𝑖-th query neighborhood is supposed to aggregate with more emphasis on the query feature.Note that in the validating and testing, the guidance of target images is invalid and we substitute the Ẽ𝑖 for 𝑬 𝑖 in Eq. 5. In an ideal form, it behooves the Ẽ𝑖 to be consistent with 𝑬 𝑖 , for the alignments between the neighbors with the queries are in parallel with alignments between the neighbors with the targets. We assign a Kullback Leibler (KL) divergence between Ẽ𝑖 and 𝑬 𝑖 during training to ensure the consistency and strengthen the adaptation between the query and target domain, which is formulated as:L 𝑎𝑑𝑎𝑝𝑡 = 1 𝑁 ∑ 𝑖=1 𝐷 𝐾𝐿 ( Ẽ𝑖 ∥ 𝑬 𝑖 ) = 1 𝑁 𝑁 ∑ 𝑖=1 𝐾 ∑ 𝑗=1 Ẽ 𝑗 𝑖 log Ẽ 𝑗 𝑖 𝑬 𝑗 𝑖 . (6)Therefore, with the supervision of the target representations to capture the correlations from neighbors, the query features would be incorporated with shared semantics from false negatives through the dynamic aggregation weights, driving the refined query features and false negative features closer in the latent space.To enhance the discriminations when comparing highly similar instances among substantial candidate images, the divergence-based correction module is dedicated to the awareness of the distribution divergences from the neighborhood features. From another perspective, neighborhood features in high dimensional latent space could be regarded as nearest feature distributions for the query representations, and precisely depicting the distribution differences could reflect the deviations from the candidate sample to the query. To preserve the inherent distribution characteristics of each representation and measure the intricate distances between high-dimensional representations at a fine level, we apply the squared Mahalanobis distance to measure the neighborhood distributions 𝑵 𝑗 𝑖 and query features 𝑸 𝑖 as:EQUATIONEQUATION)where 𝚺 ∈ R 𝑑 ×𝑑 represents the covariance matrix in the highdimensional embedding space and is initialized by diag(𝑸 𝑖 ). Therefore, the distance 𝑑 𝑗 𝑖 ∈ [0, 1] measures the semantic disparity of neighbor 𝑗 away from the anchor query 𝑖.After distilling the semantic resemblances of the neighborhood from the target and estimate the distribution divergences of the neighborhood to the query, we then integrate the two measurements into one indicator as weight parameters for further aggregation. The adaptive weight 𝒂 𝑖 = {𝑎 1 𝑖 , 𝑎 2 𝑖 , ..., 𝑎 𝐾 𝑖 } is calculated through:EQUATION)where 𝛼 and 𝛽 are learnable parameters.After obtaining the neighborhood distributions for each query, it is expected to apply the representations of neighborhood samples to refine the query features. In this way, the distribution characteristics in the target domain could be introduced to guide the query features to be close to the target features in the embedding space, and generalize the query features to adapt to the externalization variety of semantics when comparing the distances to the candidate images. Meanwhile, based on the adaptive weight, the refined query feature could preserve the primitive semantics and distill the correlated semantics adaptively on account of the correspondences and distribution divergences, which could enhance the sensitivities of comparing the true positive instances and hard negative samples. The computation of the adaptive aggregation features is shown as follows:EQUATIONwhere 𝜙 represents Sum-pooling to aggregate the 𝐾 neighbor embeddings as unified representations. Afterwards, we exploit the residual connection to integrate the adaptive aggregation features EQUATION)where FC is the fully-connected layer to transform the embeddings.3.4.1 Final Similarity. Accordingly, the refined similarity score is computed as the cosine-similarity between the updated query features Q𝑖 and candidate images, as:EQUATION)Objectives. Contrastive loss [3, 31, 38] is commonly used in the composed image retrieval task to enforce the query features to be close to positives and far away from negatives. However, the imbalances between positive and negative samples are severe when dealing with large batch sizes, which may arouse unstable convergences during training. Furthermore, models are inclined to attend to easy negative samples with comparatively low similarity scores for the given query, and the appropriate distances between the query and hard negative instances are not sufficiently improved in the metric learning. Instead, we adopt the improved focal loss to enhance the model's discrimination against hard negative samples. Specifically, the hard negatives are assigned with greater weights to increase the penalty on the high similarity scores between hard negative samples and the query samples. Hence, we design the ranking loss as:EQUATIONwhere 𝜏 is a hyperparameter to allow the similarity score to adjust within certain limits. 𝛾 is the modulating factor to adjust the strength of concentration on the hard negatives.Finally, the overall optimization objective is the integration of the adaptation loss L 𝑎𝑑𝑎𝑝𝑡 and L 𝑟𝑎𝑛𝑘 as follows:EQUATIONwhere 𝜆 weights the proportion of the adaptation loss L 𝑎𝑑𝑎𝑝𝑡 during the training process. We utilize the commonly used evaluation metric Recall-rate at K (R@K) for the two datasets in composed image retrieval. Following [9, 13, 20] , we set 𝐾 as 10 and 50 within dresses, toptees, and shirts in FashionIQ. For CIRR, apart from R@1, R@5, R@10, and R@50, we also report subset ranking results as Recall subset @𝐾. 
implementation details.: 
 The visual and textual encoders initialized in CLIP [32] backbones are implemented as four ResNet-50 [19] models and Transformer frames respectively. We first finetune the visual and textual encoders for 10 epochs with the learning rate of 2 × 10 -6 , and then freeze the parameters in encoders to train the proposed SADN models with the learning rate of 2 × 10 -5 and set the batch size as 2048. The default multi-modal compositor follows the Combiner [3] on the open-source community to obtain the initial similarities. We set the 𝐾 as 10 in neighbor representations in Sec. 3.2. The parameters 𝛼 and 𝛽 are initialized as 0.5 and The comparison of experimental results on FashionIQ dataset shown in Table 1 demonstrates the overall competitiveness of the proposed SADN. The upper column displays conventional methods without CLIP, and the middle column shows methods on the pretrained CLIP. The improvement of the proposed SADN is ascribed to the incorporation of neighbor representations on the query features, which distills semantic alignments from the target domain to enhance the salient correlations in the query. Besides, when comparing our model with MGUR [11] which is devoted to modeling the uncertainty learning in this coarsegrained retrieval task, a consistent recall rate growth could be observed. The growing recall rates implicate the neighborhood construction in SADN could facilitate the generalization and correlation extractions rather than sampling from Gaussian distributions.Table 2 reports quantitative comparison results with other approaches on CIRR dataset. In comparison with BLIP4CIR [30] requiring high computation complexity on fine-tuning BLIP models, our proposed SADN still demonstrates promising retrieval results in a concise architecture by the improvement of 4.1% on the R@1. This comparison implies that the proposed SADN could effectively capture the latent alignments across the query and target to avoid the consumption of huge computation resources. A more remarkable improvement could be seen on R@50 mainly due to the correspondence-induced adaption to strengthen the correlations from potential false negative samples. The gain on matching results about subsets demonstrates SADN is discriminatory while ranking visually similar images with subtle distinctions. 3 presents the R@10 results of ablation designs on the testbed of FashionIQ. "CIA" and "DBC" are the abbreviations for Correction-Induced Adaption and Divergence-based Correction respectively. We come to the following observations from Table 3 : (1) Neighborhood construction is essential for attaining robust retrieval performance, which is confirmed by the recall rates decline when comparing the model without neighborhood to SADN. In this case, the unrelated candidate images prevail and dominate the distilling process when aggregating on the query features and disturb both the correspondence learning and the divergence measurement. (2) Both DBC and CIA could facilitate matching the positive target images and DBC is more effective than CIA as the improvements on the mean score shows. The whole SADN achieves the optimal retrieval performance, which suggests the distance measurement is significant to regulating the semantic distillation from the neighborhoods and suppressing the irrelevant components to aggregate on the query. Note that To test the effectiveness of the proposed architecture, we also report results on different backbones for the compositor in Table 4 , where † means re-implementation. The proposed SADN could serve as a plug-andplay design to be integrated into different models for further improving the matching performance. The unsatisfactory performances on the original "Image Only" and "Text Only" indicate incomplete query and insufficient fusion for the query compositor would undermine the query-to-target matching process. The overall enhancement after aggregation demonstrates the generalization of SADN. To estimate the impacts of the losses on the retrieval performance, Table 5 shows the ablative experiments with various training objectives on R@10 metric. L 𝑎𝑑𝑎𝑝𝑡 is indispensable for the robustness to guide the correlation learning under the guidance of the target features. After joining the L 𝑟𝑎𝑛𝑘 , the recall rates further increase as the reweight factors address the imbalance partition of the positive and negative samples. The results from the "SADN w L 𝐵𝐶𝐸 " and "SADN w L 𝑡𝑟𝑖𝑝𝑙𝑒𝑡 " are inferior to the proposed SADN for the cross-entropy treats all the negatives equally regardless of false negatives and hard negatives, and the model equipped with triplet loss might be subjected to the convergence difficulty and high sensitivity to the noisy samples. Eq. 13 and Eq. 14 on FashionIQ and CIRR. As shown in Figure 3 (a), the recall rates first increase and then flow with the growth of the number of 𝐾, since the limited neighbor representations restrict the semantic extraction from the target domain while the excess neighbors bring redundancy and noise. In terms of the 𝛾 to manage the modulating factor, the optimal results occur at the 𝛾 set as 2 for FashionIQ and 3 for CIRR. The difference in the optimal settings of 𝛾 may come from the probability of the hard negative samples on CIRR being higher than FashionIQ, which requires more concentration on these samples during training. The parameter 𝜆 controls the contribution of the L 𝑎𝑑𝑎𝑝𝑡 when optimizing SADN. An obvious drop could be observed when it is fixed as 0 and the advisable value ranges from 50 to 150 to cooperate with L 𝑟𝑎𝑛𝑘 to promote the semantic distillation and discrimination.Results. We also show the visualization retrieval results with neighbor images and aggregation weights in Figure 4 .The heatmaps generated by GradCAM [34] on the last convolutional layer of the CLIP visual encoders demonstrate that our SADN could attend to the areas that the modifiers require. The target images (in red boxes) rank at the top through dynamically aggregating the neighbor features based on the semantic relations and divergence measurements in SADN. 
conclusion: 
 In this paper, we proposed a novel semantic distillation from neighborhood dubbed SADN for composed image retrieval. To mitigate the heterogeneous gap between the query and target representations, we first constructed a neighborhood from the target domain for each query and introduced the semantic aware aggregation to refine the query features with dynamic weights. Specifically, we applied correlation-induced adaption to capture shared semantics from the neighbor features and divergence-based correction to restrain the irrelevant integration of noise. Experimental results and ablation studies verified the overall superiority and generalization of the proposed SADN. In the future, we would further investigate the feasibility of semantic-aware neighborhood construction on other multi-modal tasks. 
 
 Appendix:  


Decision: This paper has been deliberated and the final decision is converged to be accepted due to the following recommendable facets:
1) Extensive investigation against existing methods kicking around and thus to ensure the proposed method stands out as its own extraordinary. 
2) The proposed method is ideated and developed with high recognition as a reasonable, viable and more intuitive framework.
3) Sufficient experiments are conducted to demonstrate/prove the effectiveness of the proposed method.

Though this paper is praiseworthy, reviewers have spotted several limitations which will turn out to benefit the paper if those can be pulled off. 
1) prominent query is the experimental results which are inconsistent to be reported across two tables.
2) computational overheads and memory costs should be provided.

Reviews: review: paper_summary: paper_summary: This paper proposes a new SemAntic Distillation from Neighborhood (SADN) approach for composed image retrieval, which incorporates the shared semantic characteristics from the targets into query representations to bridge the distribution discrepancy between queries and targets. Specifically, the neighborhood of each query is constructed to capture inherent semantic correlations and filter irrelevant components from the target domain to reconstruct the query features with adaptive weights, which could further improve the robustness of the model. Correspondence-induced adaption with the guidance of the target representations is designed to distill alignments between query and target, in collaboration with divergences measurement to evaluate the distribution disparity for refining similarity ranking. Extensive experiments on public datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: -	The authors have fully analyzed the issues of existing researches on composed image retrieval, including the distribution discrepancy between the distributions of query and target, the unlabeled positives instances in candidate set, and the unexplored subtle semantic differences. 
-	The proposed method, Semantic Distillation from Neighborhood (SADN), incorporates the shared semantic characters from the targets into query representation to bridge the distribution discrepancy between targets. The idea is reasonable and novel to some extent.
-	The neighborhood construction for query captures inherent correlations and filters irrelevant components from target domain to reconstruct the query with adaptive weights, which is proved beneficial for improving the effectiveness of the model.
-	The extensive experiments prove the effectiveness of the proposed method.
limitations: limitations: -	The experimental results (Combiner + SADN) in Tab. 4 are a little different from those in Tab. 1 and 2. Is there any modification in method or network architectures while evaluating in Tab. 4 and Tab. 1,2.
-	The Fig. 2 is too complex to understand. The input of the framework and the loss functions are not clear indicated in the figure. The directions of the arrows in the figure are also confusing. The symbols in the figure are also in different form from those in the main body of the paper, such as C_i, D_i, and A_i are in capital form in Fig. 2, while in lowercase in the main body.
-	The time complexity and memory cost during training and testing should be analyzed in the paper, since real-time and lightweight retrieval is crucial for composed image retrieval in real applications. It seems two rounds of retrieval is essential for the proposed method.
paper_summary: paper_summary: This paper investigates the composed image retrieval task, which aims to address the distribution gap between the multimodal query combination and the target image. To address this problem, this paper proposes SemAntic Distillation from Neighborhood (SADN), which constructs neighbourhood samples for each query from the target domain and further aggregates neighbourhood features using adaptive weights to reconstruct the query representation. Experiments show that the neighbourhood semantic distillation proposed in this paper significantly outperforms existing baselines and achieves better retrieval accuracy. However, this paper still has some obvious problems in terms of methodology discourse and experiments.
strength_weakness: strength_weakness: a.This paper proposes a novel neighbourhood-based semantic distillation method (SADN) for combinatorial image retrieval. The method integrates semantic features from the target image into the multimodal query representation, thereby bridging the distributional differences between the multimodal query and the target image.
b.The framework proposed in this paper constructs a neighbourhood for each multimodal query and captures the inherent semantic relevance. This is achieved by filtering out irrelevant parts from the target domain and reconstructing the multimodal query features using adaptive weighting, thus improving the retrieval performance of the model.
c.This paper adaptively align the multimodal query and the target image under the guidance of the target image and optimise the similarity ranking using divergence metric.
limitations: limitations: a. The integration of related works in this paper is not exhaustive, and some of them with significant relevance to this paper are not analysed as they should be. For example, TG-CIR [1] investigated similarity-guided candidate image ranking, which is analogous to the motivation of this paper. However, the authors did not fully explain the differences between this paper and previous similar works.
b. The experimental part of this paper adopted CLIP as the backbone, but only a few of the selected baselines adopted the same backbone. Furthermore, their performance was not as advanced as that of existing SOTA. In fact, there are many state-of-the-art works (e.g., TG-CIR [1], FAME-ViL [2], and SPIRIT [3]), have adopted CLIP as the backbone. However, the authors have not yet compared their model with these other works, which makes it challenging to demonstrate the superiority of the model proposed in this paper. It is recommended that the authors include comparisons with more advanced backbones. Furthermore, the authors do not explicitly state the specific version of CLIP that was employed, and it is recommended that this information be provided.
c. The analysis of experimental results in this paper requires further strengthening. For instance, in the ablation experiments, the paper does not analyse the anomalous experimental results of FashionIQ-Shirt in P7-L729 and P8-L833.
Reference：
[1] Wen H, Zhang X, Song X, et al. Target-guided composed image retrieval[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 915-923.
[2] Han X, Zhu X, Yu L, et al. Fame-vil: Multi-tasking vision-language model for heterogeneous fashion tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2669-2680.
[3] Chen Y, Zhou J, Peng Y. SPIRIT: Style-guided Patch Interaction for Fashion Image Retrieval with Text Feedback[J]. ACM Transactions on Multimedia Computing, Communications and Applications, 2024.
paper_summary: paper_summary: This paper proposes a semantic distillation from neighborhood called SADN for composed image retrieval. It first constructed a neighborhood from the target domain for each query and introduced the semantic aware aggregation to refine the query features with dynamic weights. The semantic aware aggregation includes a correlation-induced adaption to capture shared semantics and divergence-based correction to restrain the irrelevant integration of noise. Experimental results verified the superiority and generalization on different backbones of the proposed SADN.
strength_weakness: strength_weakness: 1.	This paper is well-written with good readability.

2.	Extensive experiments and ablation studies are conducted to demonstrate the superiority and generalization of the proposed SADN.
limitations: limitations: 1.	Why the designed Correspondence-Induced Adaption can distill correlated semantics from false negatives in the neighborhood? The authors should provide more in-depth discussions and analysis.

2.	What is the setting of baseline in Table 3? Does it mean that SADN w/o Neighbor-Aware Aggregation in Figure 2, and directly use $Q_i$ for similarity ranking? If yes, comparing the baseline with the compared methods in Table 1, why this simple baseline can outperform all of them? Do the baselines in Table 1 have the same backbones for image and text modality with SADN?

3.	The kernel function at Line 378 is not correct. It should be $g(T_j, \Psi)$.
paper_summary: paper_summary: Objective of composed image retrieval. The idea is to give as input an image and a text modifier and to get the corresponding image.
The idea is to propose two modules focusing on either converging features or diverging features to ensure a better adequation between the domain of input features and that of the target features.

The paper deals with the task of composed image retrieval. More specifically, it is the task of retrieving an image similar to an image modified by a text. The objective of this task can be apprehended as a domain adaptation problem, to make sure that features jointly extracted from the query image and the textual modifier correspond to the features of the target image. To this end, the proposed approach introduces two modules aiming towards modifying the query feature for the final retrieval in the target domain to better perform.
strength_weakness: strength_weakness: First of all, the paper is well-written even if a bit dense. The reason why the problem is interesting is well presented, especially the fact that the annotation process is often biased towards dissimilarity. The positioning of the approach is clear and the different types of existing solutions are well defined. 

The problem is well defined and the presentation of the proposed solution is extensive and clear (if a bit dense), especially with the clear overview of Figure 2. The evaluation is extensive against many methods and the improvement is very clear. The detailed ablation study helps prove the impact of all proposed modules.

Points must also be given for the quality of the supplementary material. It is informative and gives details in all aspects of the paper. Furthermore, the presence of the code at submission time is greatly appreciated.
limitations: limitations: Although the quality of the paper is undeniable, some aspects should be addressed in my opinion.

First of all, even if the section 3 is clear and detailed, it remains quite dense to apprehend in one pass. It could be interesting to maybe have a diagram showing the different sets that are being compared, it would help understand the textual description. Furthermore, there seems to be an incoherence at the end of section 3.2 when the set $S_N^j$ is mentioned rather than $S_i^N$.

In the evaluation part, there is first an incoherence at the end of section 4.3.1. Isn’t it the last two lines rather than the last two columns of table 3 that should be compared ? Furthermore, it should be interesting to maybe underline the second best per column in table 1 and 2 because it seems that the proposed approach is mainly the best whereas the second best is divided between many methods. A question also arose regarding the parameter sensitivity, would it change with other datasets ? For instance with many more classes ? Or a nested classification ?

Finally, some information in terms of training time would be appreciated to compare with other methods.

Typos : 
- l. 150-151 : “negative examples” but “is”
- l. 380 :  isn’t it the j-th target ?
- Supp. Mat : Section 4 title