Learning Fine-Grained Representations through Textual Token Disentanglement in Composed Video Retrieval
Authors: Yue WU
Zhaobo Qi
Yiling Wu
Junshu Sun
Yaowei Wang
Shuhui Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Composed Video Retrieval; Fine-grained Representation; Feature Disentanglement
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: With the explosive growth of video data, finding videos that meet detailed requirements in large datasets has become a challenge. To address this, the composed video retrieval task has been introduced, enabling users to retrieve videos using complex queries that involve both visual and textual information. However, the inherent heterogeneity between the modalities poses significant challenges. Textual data are highly abstract, while video content contains substantial redundancy. The modality gap in information representation makes existing methods struggle with the modality fusion and alignment required for fine-grained composed retrieval. To overcome these challenges, we first introduce FineCVR-1M, a fine-grained composed video retrieval dataset containing 1,010,071 video-text triplets with detailed textual descriptions. This dataset is constructed through an automated process that identifies key concept changes between video pairs to generate textual descriptions for both static and action concepts. For fine-grained retrieval methods, the key challenge lies in understanding the detailed requirements. Text description serves as clear expressions of intent, but it requires models to distinguish subtle differences in the description of video semantics. Therefore, we propose a textual Feature Disentanglement and Cross-modal Alignment framework (FDCA) that disentangles features at both the sentence and token levels. At the sequence level, we separate text features into retained and injected features. At the token level, an Auxiliary Token Disentangling mechanism is proposed to disentangle texts into retained, injected, and excluded tokens. The disentanglement at both levels extracts fine-grained features, which are aligned and fused with the reference video to extract global representations for video retrieval. Experiments on FineCVR-1M dataset demonstrate the superior performance of FDCA. Our code and dataset are available at: https://may2333.github.io/FineCVR/.

Published as a conference paper at ICLR 2025
LEARNING FINN-GRAINED REPRESENTATIONS THROUGH TEXTUAL TOKEN DISENTANGLEMENT IN COMPOSED VIDEO RETRIEVAL
Yue Wu1.2.4, Zhaobo Qi Yiling Wu', Junshu Sun', Yaowei Wang2 Shuhui Wang1.2*
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS 2 Pengcheng Laboratory 3 Harbin Institute of Technology, Weihai
University of Chinese Academy of Sciences "Harbin Institute of T'echnology, Shenzhen wuyue221@mali ucas ac gizb@hit.edue edu.en (sunjunshu21s
aeachuuuuljict.accc...o wangywepcl.ac.cm
ABSTRACT
With the explosive growth of video data, finding videos that meet detailed require- ments in large datasets has become a challenge. To address this, the composed video retrieval task has been introduced, enabling users retrieve videos using complex queries that involve both visual and textual information. However, the inherent heterogeneity between the modalities poses significant challenges. Tex- tual data are highly abstract, while video content contains substantial redundancy. The modality gap in information representation makes existing methods strug- gle with the modality fusion and alignment required for fine- iee-grained composed retrieval. To overcome these challenges, we first introduce FineCVR-IM, fine- grained composed video retrieval dataset containing 1,010,071 video-text triplets with detailed textual descriptions. This dataset constructed through an auto- mated process that identifies key concept changes between video pairs to generate textual descriptions for both static and action concepts. For fine- grained retrieval methods, the key challenge lies in understanding the detailed requirements. Text description serves as clear expressions of intent, but it requires models to distin- guish subtle differences in the description of video semanties. Therefore, we pro- pose a textual Feature Disentanglement and Erossl Alignment framework (FDCA) that disentangles features at both the sentence and token levels. At the sequence level, we separate text features into retained and injected features. At the token level, an Auxiliary Token Disentangling mechanism proposed disen- tangle texts into retained, injected, and excluded tokens. The disentanglement at both levels extracts fine- fine-grained features, which are aligned and fused with the ref- erence video extract global representations for video retrieval. Experiments on FineCVR- 1M datasct demonstrate the superior performance of FDCA. Our code and dataset are available at: https:: /t0:::/mmayy330 wttpaay3333.tthhtii. github..b.......... 10/ Fece//inn/////////
INTRODUCTION
The explosive growth of video data poses a challenge for users seeking videos that meet fine-grained requirements within vast video database. This demand can be modeled complex query patterns, i.e., composed video retrieval (CVR) (Ventura et al., 2024 Hummel et al. 2024). CVR combines reference video and modification text as query find visually similar and semantically rele- vant videos with altered objects, attributes, or actions. However, existing methods (Ventura et al.) 2024 Hummel et al.] 2024),, including foundational visual-language models(Radford et al. 2021 Li et al. 2002), perform poorly in meeting fine-grained retrieval requirements (Ventura al. 2024). We identify the primary reason is the semantic granularity gap between textual and visual represen- tations. To address this, first introduce new dataset with more detailed textual descriptions. Second, we decompose the textual representation for -ineggrained fusion and alignment.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Quers
(4)
taking
A person is
chasge to drinking
the penson.
(8)
pesson bus drinking
(e) the person for with lapoop bui drinking.
FDCA
Result
Wrong
buighter border indicates higher sotisfationn RED Retained Component BLUE Injected Componentt GREEN Excluded Component.
Figure 1: A user wants to retrieve videos of similar people "taking a cup not with a laptop and drinking". (a) In video-text retrieval, the results often fail to meet user expectations from a visual perspective due to the limitations of text queries. (b) In composed video retrieval, focusing on altered elements is vague, leading to mismatched results. (c)-(e As the three components (in different colors) are gradually incorporated into the modification text, the retrieved videos become increasingly aligned with the user's intent.
From the dataset perspective, the current CVR datasets include CoVR (Ventura et al. 2024 and EgoCRV (Hummel et al. 2024). However, these datasets fail to convey the fine linengrained user de- mands, whereas the modification texts focus solely the changed element, neglecting to specify the visual information that should remain consistent. This oversight often causes redundant visual in- formation in the reference videos mislead models. Consequently, the returned videos may contain incorrect visual parts. See the example in Figure (b), the phrase "change to drinking" highlights the action of drinking but fails convey the essential visual context from the reference video. As a result, the returned videos either visually resemble the reference video or simply depict drinking, failing deliver satisfactory results. Therefore, it essential introduce more precise modification texts that explicitly convey fine-graine grained requirements to enhance retrieval effectiveness.
To facilitate the fine-gn fine-grained CVR task while minimizing extensive annotating efforts, we highlight the conceptual similarities and differences in text queries and construct new dataset FineCVR- MM, containing over one million triplets. Each triplet consists of pair of similar videos and modification text that precisely describes the conceptual relevance between the videos. To convey the fine-grained user demands, we generate modification texts based on three components, including retained com- ponent that remain consistent in the reference and target videos, injected component that are newly added in the target videos, and excluded component that are expected to be excluded from the tar- get videos. Compared to the simple phrases (Liu al.) | [Wu et al., [221] Ventura al.) [024]] Hummel et al., (2044), modification texts with the three components explicitly express fine-grained user demands complex video content. FineCVR-I constitutes total of 1,010,071 triplets compassing diverse concepts. It can be utilized for developing CVR models and enhancing the fine-grained information-seeking ability of multimodal foundation models.
Given existing coarse-grained CVR datasets, current CVR methods (Ventura et al. 2024 Hum- mel al. 2024 Thawakar et al., 2024) primarily focus on cross-modal alignment and Tusion the sentence level. This treatment not only leads to highly-coupled feature extraction and inferior per- formance on the fine- ine-ggrined CVR task, but also lacks exploration on the Amer-grained demands that can be well described by the above three language components. To facilitate fine-graines retrieval, we propose Textual Feature Disentanglement and Cross-modal Alignment framework (FDCA). FDCA disentangles modification text at both sentence and token levels, where cross- cross-modal align- ment is performed between video and text as a basic objective function. By leveraging both sentence- and token-level semantic analysis, FDCA ensures more accurate alignment between the text and visual content, ultimately improving the performance of ine-grained CVR.
We compare method with several composed retrieval methods the FineCVR- 1M dataset. Experimental results demonstrate that our method outperforms existing methods by a clear margin. Our study provides strong support from both dataset and methodology perspectives in enhancing the
2

---

Published as a conference paper at ICLR 2025
fine tie-grained information seek ability of multi-modal video pre-trained models. To summarize, contribution is threefold:
We construct benchmark FineCVR-1M with IM+ triplets. This benchmark supports the combined query with both reference videos and modification text for fine-grained video retrieval and is publicly available for download.
We propose FDCA that performs text feature disentangling at sentence and token levels to progressively enhance the descriptive power of features of the reference video, facilitating efficient retrieval of target videos that visually and semantically satisfy user expectations.
Extensive experiments show the high quality of the FineCVR-IM dataset and the advantage of FDCA.
2 RELATED WORK
Composed Query Dataset requires generating captions for the differences between the reference and target videos. Two CVR datasets have recently emerged, iee, CoVR (Ventura al., 2024)
and EgoCRV (Hummel et al.] 2024). CoVR employs large language model (LLM) Touvron al. 2023a) to generate captions highlighting the differences between two reference captions, while EgoCRV manually creates a test set for first-person perspective videos. However, the modification texts both datasets do not explicitly describe fine- tene-gra requirements, leading to an inability in supporting fine- see--graidd CVR tasks. In our work, we construct datasets by employing clear textual prompts and the LLM automatically generate detailed text descriptions based on video content, thereby well supporting fine-grained composed retrieval.
Composed Image Retrieval (CIR) aims to get the target image by a composed query including a reference image and modification text. Current CIR models focus mainly feature discrimination and feature composition. Discriminating features requires distinguishing the retained features in the reference images, and the features need to be injected into the target image. Existing methods employ gate mechanism (Vo et al. 2019) or semantic -uiided attention mechanism (Chen et al.p
2020b [ee et al. 2021: Wen et al.] 2021 Hosseinzadeh & Wang 2020 Baldrati et al.) 2023 Saito
et al. 2023) to discriminate these features. The feature composition is addressed by projecting visual and textual features into common space, typically utilizing cross-modal Transformer (Saito al.,
2023 Baldrati et al.) (203). However, due to the lack of temporal information, these CIR methods can only be applied static modifications and cannot be extended to CVR directly.
Composed Video Retrieval (CVR) extends the task of composed retrieval from the image domain the video domain. The CoVR (Ventura et al. 2024) employ the BLIP (Li et 2022) directly to achieve mross- alignment and fusion, while EgoCVR uses a training -fre reranking approach.
To obtain the enhanced semantics from modification text, (Thawakar al.) 2024 leverages an LLM to enrich the modification texts. However, these methods emphasize coarse-level feature alignment. In contrast, our FDCA focuses on discriminating textual features into three meaningful components for fine-grained alignment.
3 FINECVR- IM DATASET CONSTRUCTION
Items in the dataset are organized as triplets [V,, T, V.). where T denotes the modification text describing the differences between reference video V, and a target video V,. We first match similar videos as the reference video and the target video, then generate modification texts automatically. For the static concepts, modification texts are generated by filling different and similar key static concepts into the textual prompts. For action concepts, we directly employ a fine-tuned LLM to generate descriptions of the action differences and similarities.
3.1 VIDEO PREPROCESSING AND PAIRING
To ensure diversity and accuracy, first collect videos from easily accessible datasets. We use four datasets, ActionGenome (i et al. 2020), ActivityNet (Fabian Caba Heilbron & Niebles 2015). HVU (Diba al.) 2020). and MSRVTT IT (Xu et al. 2016) our video source. These videos provide rich information on the visual contents, such as actions, scenes, objects, and attributes. To reduce the redundancy of the videos, we clip videos into different events based on their temporal boundary
3

---

Published as a conference paper at ICLR 2025
Table 1: Statistics of FineCVR-IM and existing CVR datasets. "RC" denotes "Retained Compo- nent", "IC" denotes "Injected Component", and "EC" denotes "Excluded Component". FineCVR- 1M incorporates more diversified concept types and fine- ine-grained components in the descriptions.
Train triplets
Test triplets
Unique word
Attribute- Centric
Object- Centric
Scene- Centric
Action- Centric
Visuals
RC
IC
BC
WebVid-CoVR
1,648,899
2.556
133.219
21,098
85.05 85.0%
15.0%
EgoCVR FineCVR-IM
2,295
2.295
940
21.1%
78.9%
1,000,028
10,043
136,547
20,961
0.5%
22.1%
28.5%
48.9%
4
annotations and remove videos lacking corresponding captions. To match similar video pairs that contain slightly different content, following (Chen et al., 2020a Xu & Wang, 202I), we compute the cosine similarity among the BLIP-2 features (Li al. 2023) of videos and select the top 20 matches for each video. Pairing videos facilitates the comparison of video content and generating their corresponding modification texts.
3.2
MODIFICATION TEXT GENERATION
3.2. STATIC CONCEPTS
Basic Static Concept Detection. Static concepts, in contrast with the action concepts of the objects, refer to contents such as objects, attributes, and scenes. Basic statie concepts in the videos serve as the flags for identifying both the content differences between videos and the contents of query interests. To ensure a comprchensive detection of the static concepts, we extract them from both video frames and their annotations. The former are often visually salient concepts, while the latter are high-quality and semantically significant.
Key Static Concept Selection. In this step, we assign scores for concepts of each video clip to identify the key concepts. Specifically, concepts from human annotations are considered highly eredible and align with human attention, they obtain higher scores. In contrast, concepts from video frames may contain error and noise, so they obtain lower scores. Based on the assigned scores, select the key static concepts with the top highest scores in each type for every video. An example is shown in the Appendix Figure A9
Modification Text Generation. We compare the key static concepts between video pairs and gen- erate descriptions of their differences with three types of prompts. Details are in the Appendix B.2
3.2.2 ACTION CONCEPTS
In addition to the static video concepts, we also generate modification texts regarding the fine- grained action differences by fine-tuning LLaMA2 (Tourron et al. 2023b). Specifically, we con-
struct prompt (further elaborated in Appendix B.2) instruct the ChatGPT (Ouyang et al.) 2022)
to generate text that describes the action differences between caption annotations. The generated descriptions are manually corrected for the fine-tuning of LLaMA2 with LoRA (Hu et al.) 2021)
We apply the fine-tuned LLaMA2 to the matched similar video pairs and produce descriptions of the action differences between videos.
3.3 DATASET PROPERTY
FineCVR-IM consists of 1,000.028 triplets for training and 10,043 triplets for testing. Detailed statistics are presented in Table Given the unavoidable noise in the automatic construction pro- cess, we manually select triples with accurate ground-truth modification text descriptions as the test subset for evaluation. Although the number of triplets is smaller than that of Web Vid-CoVR, FineCVR-IM has larger number of videos and comparable richness the descriptive texts. In comparison with the other two CVR datasets that focus only objects and actions, FineCVR-IM targets four categories (attribute, object, scene, and action) for modification and retains three crucial components (retained component, injected component, and excluded component) in the modifi- cation texts, enabling fine-grained video retrieval tasks. More analysis of FineCVR-IN are shown in Appendix B.3
4

---

Published as a conference paper at ICLR 2025
Image Encoder
Tempural lincoder
CMFA
CMFF
Cas
Target Video
CMFA
CMFF
[CLS]
taking
CUI
pietures
bat
Text Encoder
CMPA
CMFF
1
the
mirror
instead
looking
Retuined Component
Companent
and
Injected Composere
only braising
Modification Text
Figure 2: The pipeline of FDCA involves fine- ane-grained cross modal alignment and fusion through the disentangling of text features. We further enhance this process by introducing token-level disen- tangling, where clustering used generate three types of features, enabling the model focus on fine-grained information.
4
METHOD
Given pair of reference video and modification text (V,, T) query, CVR requires cross-modal alignment to relate the composed query fusion feature with a video feature. This fusion feature can be employed retrieve the target video V. from the video database, which should describe both visual and semantic information. As shown in Figure to capture the fine-grained user demand, we propose Textual Feature Disentanglement and Cross-modal Alignment framework (FDCA), dis- entangling modification text at both sentence and token levels. Specifically, Sentence-level Feature Disentangling (SFD) aims to disentangle the text feature into the retained and injected components, followed by Cross-man modal Feature Alignment (CMFA) and Cross-modal Feature Fusion (CMFF). At the token level, we introduce Auxiliary Token Disentangling (ATD) during training. ATD clusters the token-level features, and disentangles tokens into three types of components. These disentangled tokens are used in three auxiliary loss functions to guide the alignment and fusion process, resulting in semantically enriched video feature for more accurate retrieval.
4.1 SENTENCE-LEVEL DISENTANGLEMENT AND FUSION
The SDF aims to obtain global fused video feature through modification text feature disentangling and cross-moda fusion.
Video and Text Encoding, Given a video with f frames, we first employ an image encoder (Radford al. 2021 et al. 2022) to extract the reference frame features. The features are then fed into a temporal encoder and integrated as the reference video feature E IR 1xxd For the modification
text length / that corresponds the video, text encoder (Radford al. 2021 Li et al., 2022) is deployed to extract the sentence-level text feature E Rixd
Sentence-level Feature Disentangling. Based on the semantic similarity between the modification text and reference video, we design cross-attention strategy that is guided by the reference video feature. It extracts the retained component from the modification text regarding the reference video. Specifically, we project p and into the latent space query, key, and value: Q Vo(p), K T (s), V (s), where VQ VK, V are implemented as linear layers. Then, the retained text V V. We assign the remaining features s as injected text feature 8 = s Sn. which is the newly added semantic content in the target video.
feature $R E R Ixd
d is calculated by the reference- eefrrenee-guieed cross-attention Sn. softmax QK"
va
5

---

Published as a conference paper at ICLR 2025
Cross-modal Feature Fusion. The target video feature should capture the retrained feature in the reference video and fuse with the injected text feature. To align the retained text feature with the reference video feature, input the concatenation of p and Sn into the Cross-modal Feature Alignment module, implemented by a Transformer encoder. This module captures the sentence- level retained component feature zi. for the reference video. To further fuse with the injected text feature the Cross-modal Feature Fusion, achieved by another Transformer encoder and a linear layer, integrates कं and St to produce the global fused video feature o
Metric Learning. We employ the contrastive loss to measure the distance between the global fused video feature o and candidate video features in the database:
exp (o q.
C
=
log
(1)
B
Ejes exp (o
IEB
where the qi is the candidate video feature. B is the batch set.
4.2 TOKEN-LEVEL DISENTANGLEMENT AND FUSION
Compared to the highly-coupled global sentence-level features, token features with more specific concept information are easier to disentangle. Shown in Figure 3 we use clustering to separate token features into retained, injected, and excluded components, enabling more effective capture of the fine-graines details.
Token Feature Extraction. The token feature E Ixd'd is encoded with the text encoder from the original text. Moreover, we also generate positive text by removing negation words like "instead of". Given the positive texts, the text encoder gives rise to the positive text feature ē E (R xd where n denotes the number of negation words.
Auxiliary Token Disentangling. To distinguish the encoded token-level features as different compo- nents, we adopt the Density Peaks Clustering based on K-Nearest-Neighhor (DPC-KNN) (Du et al. 2016 robust clustering technique commonly uti- lized for tokens clustering (Jin et al. 2023a Zeng et al. 2022). Specifically, given tokens of the mod- ification text, to ensure tokens in each cluster have the same component type, we first apply one- dimensional convolutional layer across the word se- quence. Then we use DPC-KNN to identify the two centers of tokens. The remaining tokens are classi-
la
ATD
fied into two sets based on the Euclidean distance from cluster centers. We then average the tokens within each cluster to derive two cluster features.
Figure 3: The ATD mechanism.
To identify the injected component, we first note that compared to the original modification text, the positive text is regarded to carry affirmative descriptions, and thus more semantically similar to the reference video. Therefore, during the component disentanglement at the token level, apply clustering to the positive tokens f under the guidance of the reference video feature p. The clustering gives rise to the positive cluster features f1 E R22d of the two centers. The features of the cluster centers are then aggregated based on their similarity with the reference video feature p. The aggregation operation guided by the reference video learns the initial retained token feature ÜR.1 from the positive text:
Ên softmax(Asim(p, Fi)) F1:
where sim(.) is dot product similarity, softmax with 3 is used to compute soft index of the maxi- mum similarity. Then we can obtain the injected token feature from the positive text as & = 8 fR.1-
For obtaining the retained and excluded components, note that initial retained tokens generated from the reference video may contain excluded tokens. To address this, we further disentangle the initial retained tokens and get the retained and excluded token features. Specifically, the positive texts and the original texts contain reciprocal information about the excluded component. We can first obtain the initial retained tokens in the original text and cluster these tokens into two clusters. By comparing

---

Published as a conference paper at ICLR 2025
the clusters between the original text and the positive text, the most similar pair of clusters belongs to the retained component while the other pair belongs to the excluded component.
Following the clustering the positive text, also obtain the cluster features fi E Rxxd on the original text. The original modification text has similar injected components with its positive counterpart. Therefore, the injected token features obtained from the positive texts t can replace the injected features of the original text. Then, we can extract the initial retained token features R. from fi through:
RR1
softmax (Bsim (G, fi)]]
(3)
Next, further cluster the initial retained tokens ÉR.1 and &R,1, resulting two pairs of clusters f2 E R2xd and Jzz E 12xd, Each constitutes a cluster of clean retained tokens and a cluster of excluded tokens. The clusters of clean retained tokens are more similar to each other. In contrast, the cluster of excluded tokens from the original text is accompanied by negation words, reciprocal to that from the positive text. To distinguish the clusters, similarities between the pairs of clusters are compared, resulting in the clean retained tokens 4R, and the excluded tokens Ég.2
tn. = Σ 1=1 softmax (im (f2, fa))s f2,
(4)
Σ
softmax sim (f2, Fa))s F2.
(5)
where is the index of the two clusters.
Auxiliary Loss Construction. Given the disentangled fine-grained components at the token level, we further propose an auxiliary loss to ensure that the model focuses on fine-graines information. Resembling the sentence-level feature alignment and fusion, we also apply the CMFA and CMFF modules to the token-level features. For the feature alignment, CMFA aligns clean retained tokens ÊR,2 with the reference video, giving rise to the token-level retained component feature z For the feature fusion, CMFF fuses =I with the injected positive tokens f1, generating the fused video feature TT. Similar to the Equation [ the token-level contrastive loss L is calculated between the token-level fused feature o with the candidate video feature to learn fine- gree-griindd token-level features follows:
exp (of q.
CT -
Σ log EE8
B
Ejes exp (o,
For the finer control at the token level, we propose contrastive loss regularize the consistency of the retained components with the reference video. It constrains the distance between the positive retained tokens ÊR, and reference video feature p as:
exp (.....PP
CR
log
B
EjEB exp (E,1,PPTT)
We also adopt the excluded component to generate negative samples and regularize the negation semantics in the target video. Although there are clean excluded tokens fg.2, to avoid the information degradation caused by merging with 2R.2, we use the initial retained tokens ÊR,1 which contains both retained and excluded components to align with p from the reference database through CMFA. The feature of the identified reference video o is then fused with & in CMFF, obtaining the negative fused video feature ,² We employ triplet loss in this regularization term to penalize the presence of negation semantics in videos with margin m.
N max(0, + d(g. o²)) + d(g o))) where d(-) represents the distance metric between the two features.
4.3
OVERALL LOSS
The overall loss function is defined as the combination of sentence-level contrastive loss, token-level contrastive loss, consistency regularization term LR. and negation semantic regularization term Lx:
L C
L + LR ALN.
(9)
7

---

Published as a conference paper at ICLR 2025
Table 2: Performance on our dataset compared with baseline and existing methods.
Method
Backbone
R1
R5
R10
R50
Video-only
CLIP
8.27
27.89
40.69
70.00
Text-only
CLIP
2.86
9.19
14.61
34.15
Video-text sum
CLIP
8.47
24.85
46.08
65.01
Linear layer
CLIP
14.68
41.42
57.82
88.74
Linear layer
BLIP
19.13
51.79
68.60
93.62
TIRG Vo et al. 2019
Resnet
0.00
25.80
41.20
75.92
Artemis (Delmas al. 2022)
Resnet
4.76
17.81
28.39
57.92
CosMo Lee et al. 202
Resnet
7.82
25.38
37.45
70.96
MAAF Dodds et al. 2020
CLIP
2.16
20.61
33.37
70.39
Uncertainty-R (Chen et al. 2022
CLIP
5.23
18.45
28.28
60.78
Pic2word (Saito et al. 2023
CLIP
8.10
25.79
38.32
71.01
TFR- -VR (Hummel et al. 2024
CLIP
15.21
40.12
52.78
81.75
Combiner (Baldrat al. 2022
CLIP
19.57
49.46
65.38
92.11
FreestyleRer (Li et al. 2024)
CLIP
20.39
52.98
68.37
93.03
CoVR /Ventura al.
2024)
BLIP
17.05
41.57
56.60
85.56
FDCA-CLLIP
CLIP
25.84
55.84
70.23
94.33
FDCA-BLIP
BLIP
26.79
63.21
78.65
97.25
where is the weight the negation semantic regularization term. With the help of the three additional losses with clear functionalities, the framework focus both global and fine-grained information. Our loss design enhances the model's s capability to identify relevant information the reference video, ensuring that the fused video features align more accurately with the modification texts.
5
EXPERIMENTS
5.1
EXPERIMENTAL. SETUP
All the experiments are conducted on FineCVR-1M M with recall at 1: as the evaluation metric. We re- produce several composed retrieval methods including CIR and CVR paradigm. Our image encoder is initialized with both CLIP (Radford et al. 2021 and BLIP (Li et al. 2022) weights. Mean-pooling used as the temporal encoder in all CIR methods, including our FDCA, ensure fair comparisons.
5.2
COMPARISON RESULTS
Comparison results between our method, the baseline group and Composed Image/Video Retrieval methods group are shown Table 2
Comparison with the pre-trained models. To demonstrate the capability of pre-trained models for CVR task, we set up several basclines with CLIP feature, including utilizing only the reference video feature, utilizing only modification text feature, summing the video feature and text features directly. The fourth and fifth baselines indicate a linear layer trained for fusing cross-modal features with CLIP and BLIP, respectively. Our method significantly outperforms the above five baseline methods, emphasizing the importance of the combined query. Moreover, the score of video-only baseline surpasses that of the video-text sum baseline, indicating simple feature summation can hardly facilitate the complex CVR task, due to the potential misunderstanding and misrepresentaaton of the modification texts. In contrast, FDCA focuses on disentangling the modification text regarding the reference video at both sentence and token levels, facilitating more elegant semantic component extraction and alignment the combined queries.
Comparison with Existing Methods. FDCA demonstrates superior performance, thanks to the disentangling of both global and fine-grained information. Notably, approaches such as Artemis underperform the baseline methods because they are not developed based on large-scale pre-trained model. Nevertheless, even with the advanced pre- pre-trained model, it is worth highlighting that our FDCA-CLIP even outperforms the CoVR method fine-tuned with BLIP. This observation indicates that a well-designed feature disentanglement pipeline is crucial in the CVR task, which, in a sense, appears be more important than updating stronger large-scale pre-trained model.
8

---

Published as a conference paper at ICLR 2025
5.3 ABLATION STUDIES
We conduct ablation studies to assess the necessity of each component in our model on the FineCVR- 1M test set. We use CLIP as the image encoder for the case study, and similar observations can also be obtained on BL.I..
Ablation Studies on SFD and ATD. Table shows that FDCA leverages the strengths of both SFD and ATD. The approach without dis-
Table Ablation studies on each module.
Method
SFD
ATD
R1
R10
Disent.
22.03
$3.22
69.32
94.33
entangling performs poorly due to misinter-
w/ SFD w/ ATD Trans.
22.76
53.20
68.40
94.19
preted combined queries. While introducing SED helps, it still lacks fine-grained informa- tion modeling. The method with ATD excels on RI by focusing formance on R5, R10, and R50. By combining ATD and SFD, our model effectively balances the global and fine-grained information, achieving strong results across all metries. Moreover, within our FDCA framework, we implement CMFA and CMFF separately using distinct Trans- formers. When we replace these with a unified Transformer, denoted as "w/ Trans.", we find that the alignment-fusion -sstossart paradigm is more effective than implementing alignment-fusio single module. This finding is consistent with previous study [Li al. (2025), and it suggests that aligning the retained information before merging it with the retained features allows the model to better focus on the relevant components.
25.77
50.94
64.46
90.32
70.83
94.75
HDCA
25.84
55.84
70.23
94.33
fine-grained details, but it neglects global information, leading weaker per-
in
Ablation Studies on Loss Functions. In Ta-
Table 4: Ablation studies each loss.
ble each loss function contributes to FDCA, resulting in well-balanced performance across
Method
LR
Ex
RI
R5
R10
R.50
w/ SFD
22.76
53 20
68.40
94,19
all metrics. Specifically, the method employ- ing S solely focuses global sentence- feature disentangling, showing lower scores at R1. Conversely, the method only utilizing L' fails to obtain clean components, leading to the lowest performance. By further introducing the LR and CN. the fusion module can extract fused
w/ SFD+ATD
22.15
52.60
68.13
93.83
w/ SFD+ATD
21.37
51 59
67.42
91.89
w/ ATD
20.83
$1.56
67.35
93.85
SFD+ATD
24.42
53.94
68.95
93.78
SFD+ATD
w/ single cluster
22.01
52. 52.16
67.96
93.81
FDCA
25.84
55.84
70.23
94.33
features that align with the retained and injected components while filtering out the excluded ones, resulting in higher scores. We also implemented method that uses a single clustering on tokens, which yielded poor results. This emphasizes the importance of clustering clean components. Over- all, with the addition of three extra loss functions, FDCA achieves clear disentangled features, re- sulting in better-fusee fused video representations.
5.4 ANALYSIS
Effect of Auxiliary Token Disentangling. In Figure we visualize the cross-modal features ob- tained through the alignment module using image features from the second last layer of the visual encoder. In Sample 1, we can observe that by integrating ATD, the network significantly enhances the focus on the retained component (red), i.e., the kitchen. Moreover, even without the excluded text, the modification text can be distinguishable as retained text (red) and injected text (blue). In
peren als waiding sdeseless but
dh lacken SHRE vegitalle
dellg mencioos but
rantino is changm
.iiiig chai.
Trngi
Torgo Viree
Vike
Tal Sowa
kachen
piepersa is deing curtho, lasj
perien also waiding eeliiiios now mantg
rasiee is change
thssceer where
vegenable
buited
Ty: Booll
greand
Typi Bowk
Oan ATA
Ola ATD
Trpl Bowl
percon la alsa dalao eneeiees hut. Troer
dia watchleg
the kackes is bank
miriiden, ber
preand
Sample Figure 4: Visualization of FDCA with and without ATD. With the help of the ATD, the model can focus the retained component in the reference video.
Sample
Sample 2
Sample

---

Published as a conference paper at ICLR 2025
Sample
Sample
Kample
Target Vidla
Target Video
Target Video
Reforence Video
Reference Vidco
Referrnce Video
the Kicces sume
the some where the table changes the Billlay
the person also sitting st dodk, they ane steexing moted cating sontucch
will bread
Video
Ours Top-1 Video
Figure 5: The visualization of failed cases.
Sample 2, when the model lacks ATD, it concentrates wrongly at the excluded component (red), i.. the ground. With the help of ATD, the model shifts its focus to the action of doing exerci ses, demonstrating the essential role of ATD.
Auxiliary Mechanism or Main Mechanism.
Table 5: Results on different fusion mechanisms.
In FDCA, the ATD exclusively participates in the training process. To maximize the uti- lization of the features from ATD, we vali- date the performance under the prior-fusion and post-fusion settings. Prior-fusion involves the integration of features before the CMFA and CMFF, while post-fusion incorporates features subsequent to these two modules. The results in Table show that our auxiliary mechanism outperforms these two fusion methods. Notably, the prior-fusion method surpasses the post-fusion. This observation suggests that features at the sen- tence and token level exhibit certain level of redundancy, and the subsequent alignment in the prior-fusion effectively mitigates this redundancy. In comparison, our approach directly utilizes the sentence-level feature, thereby circumventing this redundancy issue and achieving superior results.
Method
R1
R5
R10
R50
Prior Fusion
25.70
56.03
70.35
94.29
Post-Fusion FDCA
23.29
53.28
68.11
93.62
25.84
55.84
70.23
94.33
Case Studies. We analyze the failed case that ranks out of 100 via the FDCA-BLIP to show the limitation of our method and the challenge of the FineCVR- IM dataset. As depicted in Figure 5 our dataset primarily poses significant challenge for algorithms in the need to balance visual and semantic similarity. In the first case, the retrieved video may closely resemble the reference video but might not clearly show bread in the first case, possibly due to the small object scale of the bread. In the second case, the algorithm needs to accurately understand the event sneezing but our method finds video that only shows s tt i in action. This highlights the importance of temporal modeling, which involves capturing the dynamic changes in the video over time and understanding how they relate to the semantic content. Overall, addressing these challenges requires algorithms to further deal with the spurious and missing correlation in the appearance and temporal information in the video with a more elegant design.
Different Temporal Encoders. We explore three temporal encoders in Table 6 Follow-
Table 6: Results on different temporal encoders.
ing CLIP4Clip (Luo et al.) 2022). we replace
Method
R1
R5
R10
R50
the temporal encoder with LSTM and Trans-
FDCA-MeanP 25.84
55.84
70.23 94.33
former. The results, consistent with CLIP4Clip, show that mean-pooling yields better perfor- mance. However, this does not imply that the input video is insignificant, as Table A7 in the Appendix shows that using only images leads to worse results. These findings suggest that simple temporal encoder is insufficient for fine-grained temporal modeling. More advanced temporal encoders are needed for fine-grained retrieval.
FDCA-Trans
24.45
54.53
69.07 93.91
FDCA-LSTM 23.17 53.36
67.71 91.65
6 CONCLUSION
This paper introduces new dataset FineCVR-IM to facilitate the study of fine-grained CVR and down-stream adaption of large-scale pre-trained models. Moreover, we propose method FDCA for fine-grained CVR. FDCA extracts cross-modal fused features by disentangling text features at both the sentence and token levels regarding the reference video. Our experiments demonstrate that the dataset we create has good quality and diversity, and our method achieves remarkable performance compared to other competitors including the original pre-trained VLMs and other CVR methods. Further study includes more intriguing modeling of the complex video temporal information and integration into the mudal foundation model development, especially from the perspectives of fine-grained video information seeking and video-centric dialog applications.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide comprehensive details on datasets in Sec- tion 3 and Section B method design in Section Details of the training strategies can be found in Section A
The source code has been submitted supplementary materials.
ACKNOWLEDGEMENT
This work supported in part by the National Key R&D Program of China under Grant 2023YFC2508704, and in part by the National Natural Science Foundation of China: 62236008, 62022083, 62306092, and 62441232. The authors would like to thank Shufan Shen, Zhengqi Pei, and the anonymous reviewers for their constructive comments and suggestions that improved this manuscript.
REFERENCES
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21466-1669 21474, 2022.
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. arXiv preprint arXiv:2303. 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning. pp. 1597-1607. PMLR, 2020a.
Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3001-3011, 2020b.
Yiyang Chen, Zhedong Zheng, Wei Ji, Leigang Qu, and Tat-Seng Chua. Composed image retrieval with text feedback via multi-grained uncertainty regularization, arXiv preprint 1..39221 2022.
Ginger Delmas, Rafael Sampaio de Rezende, Gabriela Csurka, and Diane Larlus. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. arXiv preprint arXiv:2203.081011 2022.
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Lue Van Gool. Large scale holistic video understanding. Computer Fision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 593-610. Springer, 2020.
Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye. Modality-agnostt attention fusion for visual search with text feedback. arXiv preprint arXiv:2007.001 2020.
Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Bii
Systems, 99:135-145 2016.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet:
A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, 2015.
Xintong Han, Zuxuan Wu, Phoenix X. Huang. Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. Automatic spatially-aware fashion concept discovery. In ICCV, 2017.
Mehrdad Hosseinzadeh and Yang Wang. Composed query image retrieval using locally bounded features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 3605, 2020.

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0666 096885 2021.
Thomas Hummel, Shyamgopal Karthik, Mariana-luliana Georgescu, and Zeynep Akata. Egoevr: An egocentric benchmark for fine-grained composed video retrieval. arXiv preprint arXiv:2407.16633 2024.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions com- positions spatio-temporal scene graphs. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10236-10247) 2020.
Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal represen- tation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472-2482, 2023a.
Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. Text-video retrieval with disentangled conceptualization and set-to-set alignment. arXiv preprint 1.Xi::3305.12218. 2023b.
Giorgos Kordopatis-Zilos, Symeon Papadopoulos, loannis Patras, and loannis Kompatsiaris. Visil: Fine-grained spatio-temporal video similarity learning. In Proceedings of the IEEE/CVF interna- tional conference computer vision, pp. 6351-6360, 2019.
Seungmin Lee, Dongwan Kim, and Bohyung Han. Cosmo: Content-style modulation for image retrieval with text feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 802 812, 2021.
Hao Li, Yanhao Jia, Peng Jin, Zesen Cheng, Kehan Li, Jialu Sui, Chang Liu, and Li Yuan. Freestyleret: Retrieving images from style-diversffied queries. In European Conference on Com- puter Vision, pp. 258 274. Springer, 2024.
Huafeng Li, Zengyi Yang. Yafei Zhang, Wei Jia, Zhengtao Yu, and Yu Liu. Mulfs-cap: Multimodal fusion-supervised cross-modality alignment perception for unregistered infrared-visible image fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-18, 2025. doi: 2T00///TP....22555335553
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning. pp. 12888-18 12900. PMLR, 2022.
languugg--n
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 2023.
Zheyuan Liu, Cristian Rodriguez, Damien Teney, and Stephen Gould. Image retrieval real-life images with pre-trained lsiionaadd-aanguage models. In ICCV, 2021.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19305-
19314, 2023.
Omkar Thawakar, Muzammal Naseer, Rao Muhammad Anwer, Salman Khan, Michael Felsberg, Mubarak Shah, and Fahad Shahbaz Khan. Composed video retrieval via enriched context and discriminative embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26906, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2337..92288 2023b.
Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr:
Learning composed video re- trieval from web video captions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5270-5279, 2024.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proceedings of the IEÉE/CVF conference on computer vision and pattern recognition, pp. 6439-6448, 2019.
Haokun Wen, Xuemeng Song, Xin Yang, Yibing Zhan, and Liqiang Nic. Comprehensive linguistic- visual composition network for image retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1369-1378, 2021.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. The fashion dataset: Retrieving images by combining side information and relative natural language feedback. CVPR, 2021.
Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame- level similarity perspective. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 10075-10085, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Wang Zeng, Sheng Jin. Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. Pro- ceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 11 11101- 11111, 2022.
13

---

Published as a conference paper at ICLR 2025
In this Appendix, provide:
1. More experiments in Section
2. More details and samples of the FineCVR-IM dataset in Section B
A EXPERIMENTS
A. EXPERIMENTAL SETUP
Baseline Methods. To assess the necessity of this CVR paradigm, we evaluate the following base- lines by a frozen CLIP:
Video-only uses the reference video input only.
Text-only queries by the modification text.
Video-text sum is to sum reference video feature and modification text feature from CLIP directly.
Linear layer is to employ a linear layer to compose the features of the reference video and modification text from CLIP or BLIP.
Composed Query Retrieval Methods. We compare our FDCA with several reproduced composed query retrieval methods including CTR and CVR tasks, as detailed below:
TIRG (Vo et al.
2019) employs a gated module and a residual module to learn the trans-
formation and preservation features.
MAAF (Dodds et al. 2020) introduces self-attention model to enhance the interaction of the cross-modality features.
CosMo (Lee al. (021) uses content and style modulators to learn the underlying style information and residual information.
Uncertainty-R (Chen al. 2022) extracts the multi-grained feature and introduces uncer- tainty regularization to adapt the matching objective.
Artemis (Delmas et al. 2022) sums the implicit similarity and explicit match scores to produce the final score for retrieval.
Pic2word (Saito et al. 2023) transforms an input image to a language token, then composes the pseudo token with text tokens through 3 fully-connection layers obtain the fused feature.
Combiner (Baldrati et al. 2022 contains linear layers to combine the two modalities feature from the CLIP features.
FreestyleRet (Li al. structure.
2024) propose style-space construction and prompt-tuning strategy
CoVR (Ventura et al.
2024) finetunes the text encoder of the BLIP model directly.
TFR-CVR (Hummel et al. 2024) use an LLM combine the video caption and textual modifier into coherent target caption.
We reproduce all the compared methods in the CVR paradigm by inputting a video and a text. To encode videos that have an additional temporal dimension, we utilize mean-pooling in the temporal dimension as the temporal encoder for all the CIR methods and our FDCA.
Implementation Details. All experiments are conducted on the NVIDIA RTX3090 using PyTorch 640) (Radford et al. 2021) or BLIP large (d 256) (Li et al. 2022) as our video encoder. The model is optimized with Adam with an initial learning rate of Ic-4. We set the batch size to 1024 to maintain the perfor- mance. To avoid overfitting, we train our FDCA for 30 epochs. The in the negation semantic regularization term IN is 0.2, while the weight À of the negation semantic regularization term LN set as 5. We implement the Cross Modsllyyy Feature Alignment (CMFA) and Cross-Modality Fea- ture Fusion (CMFF) modules using six Transformer Encoder layers. Each encoder layer consists of multi-head attention mechanism, Layer Normalization, and
For our proposed method FDCA, we utilize the frozen CLIP Res50x4
Network.
14

---

Published as a conference paper at ICLR 2025
Table A7: Results of Different Query for Video Retrieval on FineCVR-IM test dataset.
Train/Val
Test
R1
R5
R10
R50
img+mod
20.51
43.19
60.71
90.12
img+mod
vdo+mod
22.18
45.82
63.92
91.80
img+mod 15.00
37.10
49.32
76.88
vdo+mod
vdo+mod
25.84
55.84
70.23
94.33
Table A8: Results on WebVid-CoVR dataset.
Method
Finetuned BLIP's encoder
R1
R5
R10
R50
CoVR FDCA-BLIP
53.13
79.93
86.85
97.69
52.23
79.42
86.66
96.91
FDCA-BLIP
54.80
82.27
89.84
97.70
A.2 RESULTS ON DIFFERENT QUERY FOR VIDEO RETRIEVAL
To explore the relative effectiveness of images and videos in the CVR task. conduct validations on the FineCVR- CI--CVR-MMM 1M test dataset using queries that included modification text either composed with an image (img+mod) or video (vdo+mod). The image is the middle frame of the video. As shown in Table A due to the rich action descriptions in our dataset, whether trained with image+mod or video+mod, the test results for mmage+mod are comparatively lower. This indicates that the visual information provided by single image is insufficient to convey complete semantic information. In contrast, videos can not only provide more comprehensive visual information but also offer fine- grained contextual information, making the performance of vdo+mod better.
A.3 RESULTS ON OTHER CVR DATASETS
WebVid-CoVR-Teed Dataset: We employ the CoVR and FDCA-BLIP for training and testing on the WebVid-CoVR dataset, and the results shown in Table A8 The results for CoVR are from the paper (Ventura et al., 2024). It can be observed that our method is slightly inferior to directly fine-tuned CoVR. This may be due to the fact that modification texts in the WebVid-CoVR dataset are often incomplete sentences, which hinders the provision the three essential components, con- sequently impacting the effectiveness of the FDCA method. Additionally, CoVR employs the fine- tuned BLIP directly, while our FDCA only utilizes the frozen BLIP. This may be another reason why ours is slightly inferior CoVR. Therefore, we adjust our training strategy by reducing the weight À = of Lr, setting m = 0.05, and fine-tuning BLIP's text encoder following CoVR's configuration. These modifications led to improved performance.
We also validate our method on the EgoCVR dataset (Hummel et al. 2024 Due to the unavailability of training data, we directly apply our FDCA-BLIP model pretrained on FineCVR-IM for testing. The results presented in Table A9 TFR-CVR (Hummel et al.) 2024) achieves superior performance as leverages pre-trained video-language weights from the Ego dataset and requires no additional training. In contrast, both our method and CoVR require training but lack access to the training set, resulting in relatively lower performance. Nevertheless, our approach still outperforms CoVR, particularly the fine-graines RI metric, demonstrating the effectiveness of our decomposition strategy in fine-grained retrieval tasks.
EgoCVR Test Dataset:
A.4 PARAMETER SENSITIVITY
In the ATD, we leverage the negation semantic regularization term CN to penalize the presence of negation semantics in videos. We study the effects of two parameters including m and X in the negation semantic regularization term Lx the FineCVR- IM validation set.
Effect of m. m is the margin value between positive samples and negative samples. As shown in Figure A6al while the performance variations across different margins small in our FineCVR-
15

---

Published as a conference paper at ICLR 2025
Table A9: Results on Ego-CVR test dataset.
Global
Local
Method
R1
R5
R10
R1
R5
R10
CoVR (Ventura et 2024)
15.2
24.3
33.1
49.5
62.9
FDCA-BLIP
8.7
19.2
27.1
36.3
51.1
63.8
TFR-CVR Hummel et al. 2024
14.1
39.5
54.4
44.2
61.0
73.2
50-
Revall
578
Recall
302
300
302
316
01
(a) Result of different m.
(b) Results of different À
Figure A6: The analysis of parameters in the negation semantic regularization term. (a) Effect of
(b) Effect of A
1M dataset, there is a decline in performance as m increases. The best performance's 's m is located at m 0.2, which is used in our method. This underscores the effectiveness of our ATD in generating negative samples to filter misleading information.
Effect of A. is the weight of negation semantic regularization term Lx. From Figure A6b the results demonstrate an initial upward trend followed by subsequent decline. We choose parameter that yields consistently good overall performance, with À =
A.5 THE IMPORTANCE OF THREF COMPONENTS
The FDCA with ATD only decompose the token into three components including retained, injected, and excluded components. Hence, to explore the importance of the three components, we ablate each component the final fusion module. Table shows that all three components essential. The performance with retained components is not bad, since the retained components can make the model focus the target video which is similar to the retained video feature. Moreover, when the injected component is introduced, the model can produce fused video feature that meets the modified needs of the user. Furthermore, with the help of the excluded component, the model can filter the results that the user doesn't t want. All these components help the model understand the visual and semantic demands of users clearly.
Table A10: Performance on three components by ATD.
Retained Injected Excluded
R1
R5
R10
R50
18.20
42.98
60.12
89.10
24.15
53.29
68.91
92.96
25.84
55.84
70.23
94.33
16

---

Published as a conference paper at ICLR 2025
Table All: Results on better temporal encoding mechanism.
Method
R1
R5
R10
R50
FDCA-Tram IDCA-Trans-Dicomm (Jin et al. 2023b)
24.45
54.53
69.07
93.91
25.18
51.12
64.07
90.25
TDDC--Trans-icoosa-tan
25.79
55.90
69.94
94.05
Table A12: Training time comparison.
Method
Parms Processed data Epoch Training time per epoch Total time
CoVR (Ventura et al. 2024
446M
Image
5
30 h
150 h
Pic2word (Satto et al. 2023
179M
CLIP feature
30
6 min
h
Combiner (Baldrati et
2022
237M
CLIP feature
100
10 min
16 h
FDCA
570M
CLIP feature
30
min
10 h
A.6 EXPLORATION BETTER TEMPORAL ENCODER
Intuitively, since video composed retrieval requires understanding contextual information within videos, incorporating superior temporal encoder would yield better results. Here, we employ the temporal encoding mechanism in DiCoSA (Jin al. 2023b) in Table AII] Its effectiveness is limited to RI metrics. We believe this limitation arises because DiCoSA's ' text information, which directly corresponds to video content, be aligned with videos for weighted sum computation. In contrast, our text information contains modificaation-elatee semantics, and using weighted sums introduced certain biases. When instead use our decomposed residual features for weighted sum computation, we observed significant performance improvements. This validates both the effective- ness of decomposition approach and demonstrates that superior temporal encoders can enhance temporal understanding.
A.7 COMPARISON OF TRAINING AND INFERENCE PERFORMANCE.
We compare training time in Table A12 and inference time in Table A13 Although our model has the most parameters, converges faster and has a faster inference speed. Since only use SFD during the inference stage, the inference speed is only slightly slower than the Combiner (Baldrati al. 2022).
A.8 VISUALIZATION OF ATD
We offer some illustrative examples of ATD results in Figure A7 As observed, ATD can effectively disentangle the modification text into three distinct components: clean retained tokens, injected to- kens, and excluded tokens. Because we guide the disentangling of initial retained tokens through reference video features, the tokens are typically crucial retained components in the reference video, such as concepts like baahhooo............... ample and tokens are the remaining parts of the modification text. Therefore, injected tokens usually include concepts from the target video well grammatical elements, such and the person is ob objects in the first example and the is
but rearranging on the sink in the first ex- singing, but.. in the seventh example. On the other hand, the injected
also in the takes a pill
Table A13: Inference time comparison.
Method
Parms Processed data Inference time per
CoVR (Ventra et al. 2024)
446M
Image
56 pairs/s
Pic2word (Sito al. 2023
179M
Image
149 pairs/s
Combiner [Baldrafi et al. 2022
237M
Image
107 pairs/s
FDCA
570M
Image
84 pairs/s
17

---

Published as a conference paper at ICLR 2025
with net in the sixth example. After the second clustering, the initial retained token be disentangled into clean retained tokens bathroom and excluded tokens but rearranging the sink. Notably, even in the modification text without negation words, the ATD can also detect the excluded tokens such as the boy that should not exist in the target video in the fourth example.
on
Reference Video Target Video
Mod Text
Initial Retained Positive Tokens
(Posiiive)
Injected
Tokeus
Initaal Retained
Tokees
Clean Retained
(Postrive) Tokens
(Clean] Exeluded Tokens
Excluded Tokens wi Negation Wards
the
penson also battroom, but takes pill instead of rearrunging objects the sink.
bathroom
bathroom
of rearranging the sink
but rearranging on the sink
also in the
but instcad of rearranging
..bthrooo....
reatranging sink
objocts
the sink
office study
office study same but
the home office study is same but with laptop
office study...
the home with laptop
same but
the living but
the living same but
the living some but
the Living
with broem....
but with a broom
the changes from
the persen changes a boy weman
boy
boy to a
to ao.....
boy
boy
officee/ srudy same
office
Study
same but with chair
with chair
study same but
study same
but
but
arena same with
aicena same but
is with
aarena.. but
arena same..
..but.
..but..
person also singing. but on the show with a good judge
the singing. but..
singing,
also on the show with good judge
.siininng...
.buu...
..uu...
scene where the vacuum in changes Kitchen
vacuum soene changes
where tho.. the kitchen
vacuum in changes to
vacuum
in changes
changes
the person takes another makes it., see recap of.. well
the person takes another
shot and makes and we
shot and
shot and
shot and
recap that well
the also
the also something. but instead of mustard
spreading bread...
spreading something on bread, but changed to butter instead of
-ppreedinng...................g. bread but to
spreading but
instead of
bread
has changed butter
Figure A7: Examples of ATD results.
18

---

Published as a conference paper at ICLR 2025
B THE FINECVR-IM DATASET
As shown in Figure A8 we first match similar videos as the reference video and the target video, respectively. We then automatically generate modification texts regarding static concepts and action concepts, respectively. The former generates modification texts by filling the different key static concepts into the textual prompts, while the latter leverages a fine-tuned LLM to generate the action difference description directly.
Dara Preprocesiing
Video Pairing
3.L. Busic Static Cuncepi Detection
3.1.2 Key Stati Concept Detection
Maric-Conergt--nnee MTG
Highiihh Samvotil Find
Saran Linken
Theaae wler the figmp chongas in the snokkn.
dkiillng vskng lagiop
ligeep
Senur lipeay
3.2 Auctien-coneep--bee MTG
Mars
buy divnting ilen seceee and eppenal it.
Capelha
Uk peace alo working dlvir Japnay, sttrmonds oponar claser.
Figure A8: Pipeline of dataset construction. We segment long videos into video clips by preprocess- ing and pairing video clips by similarity comparison. Then we use prompts and LLM to generate modification texts regarding static concepts and action concepts, respectively.
B. VIDEO PREPROCESS AND PAIRING.
We begin by extracting videos from public datasets and clipping them based event timestamps provided in the annotations. Videos without annotations are excluded from the dataset.
We establish video pairs from the same video source by utilizing cosine similarity, a widely used metric in video similarity learning (Kordopatis-Zilss et al.] 2019: Xu & Wang [202]] Chen et al. 2020a). Specifically, we uniformly sample 8 frames from each video and compute their frame-level features using BLIP-2 (Li et al., 2023). These features are then mean-pooled to obtain a compact video-level representation. Using cosine similarity between compact video features compute video similarity, we identify the top 20 most similar videos for each query video, forming pairs that serve as the foundation for subsequent text modification text generation. To further ensure dataset quality, we instruct annotators to exclude irrelevant video pairs during test set construction.
B.2 MODIFICATION TEXT GENERATION
For static concepts, we use fixed templates to generate differences and similarities among key concepts. To ensure the high quality of the static concepts, concepts in our FineCVR-1M dataset are obtained from the annotations and captions of four accessible benchmark datasets (Ji et al.) 22220), as well as high-confidence score results by BLIP-2. Figure illlustraee the detailed process of key object calculation. In detail, objects in the caption are more semantically meaningful for humans and represent key concepts. thus receiving higher scores of 1.0. Object annotation is also critical for this task, so we give it 0.5. We believe that the results from BLIP-2 provide basic visual information with low confidence, so we assign a score of 0.1 to the BLIP result. Moreover, the importance of words in the caption is related to their location, with the last word being less important than the first word. Hence, we assign the first object in the caption list as the highest location score, and the last object a location score of O. In summary, our FineCVR-IM IM dataset benefits from the utilization of these four reliable sources of data, resulting in dataset with high-quality key concepts.
Then we compare key static concepts between video pairs and generate descriptions their differ- ences with three types of prompts: (a) identify the scene difference where the same object appears in the video pair like "the scene where the { [***] ] is in changes to the { [***] "; (b) choose a different object the focus of the prompt for videos with the same scene, formed as "the [e*e] is same but with { {***}** "; and (c) replace the attributes of the object in the target video when the key object concept is the same, e.g., "the attribute of the {*HM) is replaced by { [***)".
Furthermore, to explore the impact of fixed templates, we also rewrite 6,043 sample texts using GPT- 4 for the test set. Table A14 shows that flexible texts have minimal impact results, since the
19

---

Published as a conference paper at ICLR 2025
Key Object Concept Score Calculating
CLIP
Location
refrigerator
Caption
Annotation
door
9.
person
food
BR
slove
31
1G
window
Figure A9: A sample of calculating key object scores. The caption is "a person cooks food on stove before looking out of a window", and the CLIP detects that there are [refrigerator, door, person, food] in this video. The objects in the caption are [person, food, stove, window], and the objects in the annotation are [food, stove, window]. After calculating, the key object is "food" with the highest score of 1.62.
Table A14: Test result rewritten flexible modification text by GPT-4.
Method
Dataset
R1
R5
R10
R50
FineCVR-
14.69
46.17
64.87
94.44
FDCA-CLIP
Rewritten
14.23
43.31
61.91
92.45
FineCVR-IM
18.68
52.39
70.99
95.28
FDCA-BLIP
Rewritten
18.08
51.32
70.23
94.89
CLIP BLIP text encoder is robust enough to comprehend the core semantic meaning. Moreover, the gap between the FDCA-CLIP bigger than that between the FDCA-BLIP, which could be attributed to the BLIP encoder being BERT, capable of handling complex sentences. This indicates that for a superior text encoder, the flexibility the text will have little impact on the results as long the content remains consistent.
For action concepts, we use LLM to generate differences and similarities between video caption pairs. We first attempt validation. Specifically, we input the following prompt with caption pairs into ChatGPT to generate 1000 samples into (d) different actions with similarities and (e) different actions:
ChatGPT (Ouyang et al.] (022) generate 1,000 data samples for
I'll give two ordered video captions, and you should compare two videos and complete change captioning task based on something similar and return to me with concluded sentence the following template. If the same action object or event exists in both two ordered videos, you should fill this same thing in the first {} and output: "the person is also {} in both two videos, but () in the second video ". Else you output: "The person changes to O in the second video". (please check carefully, if you use the first format, there must be something that exists in both videos). Only give me an output as one of the above formats.
However, we find that the accuracy is not high (below 60%) since ChatGPT has not previously en- countered this type of task. To ensure that the LLM could effectively adapt to identifying differences and similarities between video caption pairs, we manually revise the responses of the 1,000 samples. We then 606 samples model LLaMA (Touvron et al.) 2023b). As a result, we achieve an accuracy of 92% and use this fine-tuned model generate the action concepts text.
training and 394 as validation set to fine-tune the open-source
20

---

Published as a conference paper at ICLR 2025
paree dh Grillig Dine
Figure A10: We manually select samples that have visual differences and similarities.
Table A15: The statistics of the prompt class in the dataset.
CislD
Proret
Coverage
Esample
Static Concept
the where she [ (wee; de changer (*e))
38.50%
where
the entrysany
the [***) but with foee)
22 08%
bodeoove hut with
atrilatt (2*2) welaaed by (era)
olanter by
Concept
I'll give captions, thould theses
11.70'
frols sonelhing mnsdd frie
captioning based something
30%
changes up clonies instedd shobing Manker
Additionally, we observe that despite instructing the LLM to generate two categories: (d) and (e), still tends to hallucinate similarities that do not actually exist. To address this, we filter out any triplets where the verbs in the identified similarities were not present in the original video captions.
For the test set, we manually select samples on the website shown in Figure A10 that visually exhibit significant differences, ensuring they are suitable for evaluation. A user group of eight individuals, including some of the authors, is employed ensure that the generated texts accurately reflect changes between the videos. We also implement two-round cross-validation process. After completing the manual correction process, we obtain 10,043 triplets as our test set.
B.3 THE PROPERTIES OF FINECVR- IM DATASET
We summarize two properties and challenges of our FineCVR- IM as follows.
Variable Concepts. As mentioned in the main paper, we focus on key concepts in videos in the FineCVR-IM dataset construction. Our FineCVR-IM dataset covers four different concept cate- gories, i.e. objects. scenes, attributes, and actions. Comprising five types of prompts, the FineCVR- 1M dataset's class coverage is detailed in Table A15 Notably, data from Static-Concepp-busee MTG and Action-Conceet-basee our dataset not only encompasses visual concepts but also integrates rich temporal information. For each class, we also provide a word cloud in Figure AIT Specifically, in the (a) and (b) classes, our FineCVR-IM dataset focuses on common outdoor or indoor scenes such as sky, along with nouns representing objects food, elements associated with these scenes window. In the (c) class, peo- ple pay more attention to the color of objects, such as brown hair, and at the same time, some ongoing actions also become specific attributes of objects, such as connecting ing computer. Meanwhile, in the (d) and (e) classes, all the words revolve around human and object actions, such as playing ba 1, showcasing the diversity of actions.
MTG roughly account for half of the entire dataset. This indicates that
Fine-grained Demand Semantic. Compared to the incomplete sentences in existing composed query datasets (Vo et al. 2019, Han et al., 2017: Liu et al., 2021: Wu et al.) 2021 Ventura et al.) 2024), the modification texts in our FineCVR-IN IM dataset are all complete sentences. Figure A 22a indicates that the length of the modification text in most cases is words, which is longer than the 5 words observed in the WebVid-CoWe dataset (Ventura et al. 2024). As shown in Figure A12b our dataset covers nouns, verbs, adjectives, and even adverbs, leading the modification text to rep- resent the complete preference of users. Moreover, we statistics the portion of three components
21

---

Published as a conference paper at ICLR 2025
water nad
fielde
S
spor
hair
no
S0me thine
Lands cap
Kankang
Food
gingon
venue
covoreeds
human
Town
o
Pop Laye
now
black
br
talking
pho
wa walking iig
artien contera
S Sky
Foom
SKY
layered
urs ung
s
around Foundllaan
man
spoft
long
red
Fying
s510e woman
te Mandsssge
(a)
(b)
(c)
(d)
(c)
Figure All: The word cloud of each class in our dataset.
2.02
1.75
Number of Triples
1.52
Average Count
1.2
2.3
1.00
2.0
6.75
1.3
0.52
1.0
0.35
0.02
Noon
Wil
Asjocive
Adval
Number of Words
Part of Speech
(a) The statistics of number of words.
(b) Results of different À
Figure A12: The statistics of modification text. (a) We count the number of words for each modifi- cation text. Most modification texts have between 7 and 13 words. (b) We calculate the number of the part of speech in each modification text.
including retained, injected, and excluded components. In FineCVR-IM. 100% modification texts comprise retained components and injected components, and 11.67% texts comprise excluded com- ponents. While the abbreviated text in Webvid-CoVR only comprises 77.68%, 100%, and 2.80%, respectively. Due to the rich information our video, such as multiple individuals, complete se- manties and rich components can prevent ambiguity caused by lack of subject and avoid returning inaccurate video results.
The Challenges. The main challenge of FineCVR-1M dataset is threefold: extracting concepts from videos exhaustively, understanding the semantics of modification text accurately, and fusing the cross-modal feature adequately. Videos from our FineCVR-IX, not only involve static concepts but also encompass actions. Therefore, addressing the first challenge requires CVR algorithms to employ an enhanced visual encoder. The visual encoder is capable effectively detecting concepts, even for small-scale objects and attributes. Additionally, robust temporal encoder is essential to comprehend the semantic information conveyed by the videos. The second challenge requires algo- rithms with a powerful text encoder to comprehend modification text, including extracting negation meanings and capturing fine-gra fine-grained words that represent users' demands. The third challenge in- volves developing algorithms that can integrate valuable information from both videos and modifi- cation texts. These algorithms should discern which parts in reference videos to preserve and which parts to transform, ensuring optimal fusion of rross-modal features.
B.4 MORE SAMPLES IN FINECVR-IM
We present some examples from our FineCVR- 1M dataset in Figure A13 Thanks to the combined insights from BLIP-2 results, annotations, and fine-tuned LLM. FineCVR-INM dataset offers rich blend concepts including static concepts and actions, showcasing elements like stairs and personal protecti ve equipment. Additionally, the video captions provide fine-grained
22

---

Published as a conference paper at ICLR 2025
the where the vehicle in changes the covered landscape.
the scene where the dog is changes to the stairs.
the sky same but with personal protective equipment.
the kitchen is but with phone.
+ the attribute of the hair is replaced by brown and longer.
attribute the is replaced by black.
the person also opening a closet, but they are taking clothes out and getting scared by someone.
the person throwing, but makes to center.
+ the person changes from demonstrating to giving hula the girl who spins simultancously hula rings.
the person changes from a woman to a man and adds pasta instead of potatoes.
Figure A13: Example tuples FineCVR-IM dataset. The videos on the left represent reference videos, while those the right represent target videos. The modification texts the bottom high- light the differences between the two videos.
information on action changes, capturing details like giving a hula a and spins two hula rings.
23


Reviews:
[{"review_id": "1XJYxfCbdo", "reviewer": "Reviewer_tW2k", "summary": "This paper proposes the FineCVR-1M benchmark, which supports the combined query with both reference videos and modification text for fine-grained video retrieval. The authors also propose FDCA which performs text feature disentangling at sentence and token levels to\nprogressively enhance the descriptive power of features of the reference video, facilitating efficient retrieval of target videos that visually and semantically satisfy user expectations.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is essential to construct fine-grained composed video retrieval datasets and develop the corresponding method. The proposed methodology including collecting similar videos and prompting LLMs for annotation generation is promissing.\n\n2. The proposed FDCA method follows the conventional attention based principle, which has been demonstrated effiective across various tasks.", "weaknesses": "1. Unclear dataset construction details: There remain lots of missing details in the dataset construction. For example, how to handle the video when prompting, such as how many frames are used.\n\n2. Quality control of the proposed dataset. In L160, the authors select the top 20 similar videos for each video based on the cosine similarity between them. It is unclear about the cosine similarity selection accuracy. The authors should clarify this through human mannual check or some other solutions.\n\n3. The proposed method is similari to existing method [1]. These two papers share the similar fine-grained cross-modal alignment and fusion methodology. The authros should cite this paper and discuss the differences.\n\n4. In experiments, the authors only include the results in the proposed dataset. It is unclear about the performance of the proposed method in general video-text retrieval datasets. It is recommended to include the discussion about it.\n\n5. The overall paper writing can be polished. For example, the presented figures should be presented as vector images \n\n[1] Disentangled Representation Learning for Text-Video Retrieval", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKHmli6EPP", "reviewer": "Reviewer_FBRb", "summary": "Video Retrieval is a very challenging task, hence the proposal of composite video retrieval, which uses images and text as retrieval signals to search for video content. Current content retrieval faces two issues:\n(1). A lack of video-retrieval datasets with fine-grained descriptions.\n(2). A lack of effective solutions to implement good video-retrieval. \nIn response to the first issue, the authors proposed a fine-grained large-scale video retrieval dataset, FineCVR-1M, which includes one million video-text pairs. Addressing the second model-side problem, the authors proposed a decoupled text representation-cross modality alignment model. However, the methodology part requires more ablations and the related works are incomplete. It would be better to address these issues for further ranking improvement.", "questions": "1. During the construction of the WebVid-CoVR dataset, did the authors consider the possibility of retrieval mismatch due to video length when they mixed short videos like MSRVTT with longer ones like ActivityNet?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors have provided a high-quality, fine-grained dataset in the field of video retrieval, which has made significant contributions to the community.\n\n2. The authors analyzed the advantages of this new dataset over the WebVid-CoVR dataset, which is also proposed for the composed video retrieval task. CoVR uses LLM to describe the differences between two videos and does not support fine-grained details. What we generate are fine-grained text descriptions.\n\n3. The authors' dataset structure is quite ingenious, as it introduces LLM to label the three important components between two video pairs: retained component, injected component, and excluded component, to generate fine-grained text descriptions.", "weaknesses": "1. In the abstract section, the challenges of this composed video retrieval task are described too simplistically. The difficulty of fine-grained video-text modeling is a long-standing and intractable problem. The authors could provide a more nuanced description of the challenges.\n\n2. The author should conduct a more detailed exploration of the temporal encoder. Dicosa[1] found that using the text-side [cls] token to perform cosine similarity with each video frame feature, and then passing through softmax to convert into a probability distribution to guide the weighted summation of video features is a better compression method. Could this process bring performance improvements in composed video retrieval?\n\n3. The author lacks citations of relevant papers. Specifically, DPC-KNN has also been used in other video retrieval works [2], and relevant studies include Dicosa [2] and FreestyleRet [3]. FreestyleRet contains the exploration of composed image retrieval.\n\n[1]. Text-video retrieval with Disentangled Conceptualization and Set-to-Set Alignment\n[2]. Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning\n[3]. FreestyleRet: Retrieving Images from Style-Diversified Queries", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aqgDXQnqoh", "reviewer": "Reviewer_GX4t", "summary": "The study introduces the FineCVR-1M dataset and proposes a Feature Disentanglement and Cross-modal Alignment (FDCA) framework, which enhances retrieval by disentangling and aligning text and video features at both the sentence and token levels, achieving superior results in composed video retrieval. I hold a positive attitude toward this research; however, there are a few minor issues in the paper that need correction.", "questions": "1、In lines 253 and 256, the image encoder and text encoder use CLIP; the authors should provide a reference for CLIP.\n2、In line 405, please add references after mentioning CLIP and BLIP.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The study introduces the FineCVR-1M dataset and proposes a Feature Disentanglement and Cross-modal Alignment (FDCA) framework, which enhances retrieval by disentangling and aligning text and video features at both the sentence and token levels, achieving superior results in composed video retrieval. I hold a positive attitude toward this research.", "weaknesses": "1、In line 269, the structure of the Transformer encoder could be explained in the supplementary materials, or a reference could be added.\n2、In the Methods section, there is no explanation of how $\\mathcal{L}^{T}$ and $\\mathcal{L}^{S}$ are derived; please provide this information in the Methods section.\n3、In line 432, the title above Table 4 is incorrect; it is labeled as Figure 4. (Table 5; Table 7; Table 9). Please carefully review the manuscript for existing minor errors.\nNote: Please revise the Method section carefully, as it is crucial to the paper. Some implementation details that cannot be included in the main text can be added to the supplementary materials, which will help readers understand the underlying principles. I hold a positive attitude toward your work, but the manuscript requires thorough revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TYlwiBporW", "reviewer": "Reviewer_JTbd", "summary": "The authors introduce FineCVR-1M, a large-scale dataset comprising 1,010,071 video-text triplets generated through an automated process that identifies key concept changes between video pairs. This dataset includes textual descriptions of both static and dynamic concepts, facilitating nuanced video retrieval. The authors further propose the Textual Feature Disentanglement and Cross-modal Alignment (FDCA) framework, which disentangles features at both the sentence and token levels.", "questions": "1. Do the weights in Eq. 8 have different effects on the results?\n2. The specific meaning of L^T and L^S need to be explained.\n2. Please refer to the Weaknesses for the other questions.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Originality\nThe introduction of the FineCVR-1M dataset represents a significant advancement in the field of composed video retrieval (CVR). By providing a large-scale dataset with 1,010,071 video-text triplets, the authors address a critical gap in fine-grained video retrieval research. \n2. Clarity\nThe paper is well-structured and clearly communicates its objectives, methodologies, and findings. \n3. Significance\nThe work has impressive implications for the field of video retrieval, particularly in enabling fine-grained retrieval tasks that meet user-specific requirements. Experimental results indicate that the framework effectively extracts and aligns features, achieving superior performance compared to existing methods.", "weaknesses": "1. The comparative analysis with other methods is insufficient. In Section 5.2, Table 2 is presented but not mentioned or discussed in the text. This oversight limits the effectiveness of the comparison and fails to guide readers through the results.\n2. Are all the experiments conducted solely on the introduced FineCVR-1M dataset? If so, this raises concerns about the generalizability and robustness of the proposed method. Relying on a single dataset may limit the findings and their applicability to other contexts.\n3. Although the methods proposed by the authors demonstrate superiority over WebVid and EgoCVR in many aspects, it is essential to conduct experiments on existing datasets to further validate the effectiveness of FDCA. Testing on established benchmarks is crucial for establishing credibility and ensuring that the proposed method generalizes well to different scenarios.\n4. Additionally, the results presented in Table A4 indicate that the authors' proposed method performs worse than CoVR in certain metrics. This raises questions about the robustness of the proposed approach.\n5. To facilitate the authors' reading experience, it is recommended that all images be provided as vector graphics whenever possible. This approach will help prevent distortion when magnifying figures, ensuring that details remain clear and crisp. Specific figures that would benefit from this adjustment include Figures A10, A12, A14, and A15. Additionally, it has been noted that some of the fonts in Figures 2 and 3 are vector-based while others are scalar. This inconsistency can lead to variations in clarity and readability.\n6. Based on the proposed Auxiliary Loss Construction method, there appears to be some degree of redundancy among the four losses defined in Eq. (8), particularly between L^T, L^S, and L^R.\n7. In addition to retrieval performance, inference efficiency and model parameters are critical factors in the field of retrieval. The ability to deploy models effectively in real-world applications often hinges on these aspects. For example, Pic2word, which contains only one MLP as trainable parameters, highlights how simplicity can lead to enhanced efficiency without compromising performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)