Title: CoVR: Learning Composed Video Retrieval from Web Video Captions
Venue: NeurIPS 2023
Accepted: no
Keywords: composed image retrieval

deep learning

vision and language

computer vision
Abstract: Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, containing image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs. To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. We automatically construct our WebVid-CoVR dataset by applying this procedure to the large WebVid2M collection, resulting in 1.6M triplets. Moreover, we introduce a new benchmark for composed video retrieval (CoVR) and contribute a manually annotated evaluation set, along with baseline results. We further show that training a CoVR model on our dataset transfers well to CoIR, improving the state of the art in the zero-shot setup on both the CIRR and FashionIQ benchmarks. Our code, datasets, and models will be made publicly available

1

Abstract

Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, containing image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs . To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. We automatically construct our WebVid-CoVR dataset by applying this procedure to the large WebVid2M collection, resulting in 1.6M triplets. Moreover, we introduce a new benchmark for composed video retrieval (CoVR) and contribute a manually annotated evaluation set, along with baseline results. We further show that training a CoVR model on our dataset transfers well to CoIR, improving the state of the art in the zero-shot setup on both the CIRR and FashionIQ benchmarks. Our code, datasets, and models will be made publicly available.

'at night'

'with people'

'during show'

1 Introduction 17

Consider the scenario where a traveller takes a picture of a landmark or scenic spot and wants to 18

discover videos that capture the essence of that location, by specifying certain conditions via text. For 19

example, the query image in Figure 1 (of a fountain in Barcelona), along with the text 'during show' 20

should bring the video showcasing the fountain show. Further refining the text query such as 'during 21

show at night', would allow the traveller to decide whether to wait for the show until the night time. 22

In this work, our goal is composed video retrieval (CoVR), where the user performs such multi-modal 23

search, by querying an image of a particular visual concept and a modification text, to find videos 24

that exhibit the similar visual characteristics with the desired modification, in a dynamic context. 25

Young man working

with

tablet

and

blueprints at his new home

Young man working

with

smartphone

and

blueprints at his new home

2.5M video-caption pairs

MTG-LLM

Trained with 715 text

triplet annotations

q

t

v

change tablet for

smartphone

CoVR has many use cases, including but not limited to searching online videos for finding reviews of 26 a specific product, how-to videos of a tool for specific usages, live events in specific locations, sports 27 matches of specific players. Similar to composed image retrieval (CoIR), CoVR is also particularly 28 useful when conveying a concept with a visual is easier and/or more accurate than only using words 29 (e.g., unknown location/object, a specific camera view, a specific color). 30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

Given the increased momentum in vision and language research in the recent years [31, 45], CoIR has emerged as a new task [57], and since then witnessed improvements of both models and benchmarks [6, 7, 21, 28, 37, 58]. However, to the best of our knowledge, CoVR was not studied before. A key challenge in building CoVR models is the difficulty of gathering suitable training data of image-textvideo triplets. We overcome this limitation by developing an automatic approach to generate triplets from existing video-caption collections. Specifically, we mine video pairs whose corresponding captions slightly differ in text space. We automatically describe this difference with a language model, which we train for a modification-text generation task. In particular, we use manually annotated triplets, each containing: (a) source caption, (b) target caption, (c) the modification text. We then finetune a large language model (LLM) [54] by inputting (a-b), and outputting (c). We assume the resulting modification to describe the difference between the corresponding videos, thus obtaining video-text-video triplets (see Figure 2 for an overview). When training our CoVR/CoIR models, we can select one or more frames from the videos, enabling multiple settings (i.e., retrieving images or videos).

45

46

47

48

49

50

51

52

53

54

55

56

We apply our triplet generation approach to the WebVid2M dataset [4] which contains 2.5M Webscraped video-caption pairs. This results in the WebVid-CoVR training dataset with 1.6M CoVR triplets. By virtue of its automatic generation procedure, WebVid-CoVR is inherently noisy. To efficiently train on such large-scale and noisy training data, we use a contrastive loss [55] and additionally sample hard negatives that have the same source caption but different target captions. We design a CoVR model based on the cross-modal BLIP [31] and use query scoring [5] to exploit information from multiple video frames. Training this model on WebVid-CoVR transfers well to the CoIR task, in both zero-shot and finetuning settings, and achieves state-of-the-art results on the CIRR and FashionIQ benchmarks in the zero-shot setup. Finally, to foster research in CoVR, we repeat our generation procedure on a separate subset of the WebVid10M dataset [4] and manually select correctly generated samples to constitute WebVid-CoVR m , a test set of 2,435 CoVR triplets. We find that our model achieves promising results on WebVid-CoVR m compared to standard baselines.

To summarize, our contributions are: (i) We propose a scalable approach to automatically generate 57 composed visual retrieval training data. We apply this pipeline to the WebVid2M dataset and generate 58 the WebVid-CoVR training dataset with 1.6M CoVR triplets. (ii) We show that training a CoVR 59 model on WebVid-CoVR transfers well to the CoIR task, and achieves state-of-the-art results on the 60 CIRR and FashionIQ benchmarks in the zero-shot setup. (iii) We evaluate our model on WebVid61

CoVR m , a new CoVR benchmark that we manually annotate. Our code and dataset are provided in 62

the Supplementary Material, and will be publicly released together with our models. 63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

2 Related Work

Composed image retrieval (CoIR). CoIR [57] has been an active area of research in recent years [7, 14, 25]. Most methods designed for this problem use manually annotated data for training. Some recent works, such as Pic2Word [47] and SEARLE [6], explore zero-shot CoIR setups where no manually annotated CoIR triplet is used. These approaches build on CLIP [45] and train directly on unlabeled image(-text) data. In contrast, we use unlabeled video-text pairs to automatically generate composed video retrieval (CoVR) triplets, train a CoVR model on the generated data, and study zero-shot and finetuning transfer of the resulting model on both CoIR and CoVR.

Datasets for composed image retrieval. CIRR [37] and Fashion-IQ [58] are the two most widely used CoIR benchmarks. Both are manually annotated, hence small scale (about 30K triplets, see Table 1) due to the high cost implied in collecting CoIR triplets. To scale up, two concurrent works proposed larger, automatically generated CoIR datasets: LaSCo [28] and SynthTriplets18M [21]. However, these two datasets are currently not publicly available. The LaSCo dataset [28] is generated using the visual question answering annotations and the pairing between images and counterfactual images in the VQAv2 dataset [3]. In detail, this dataset provides for each (image, question, answer) triplet a counterfactual triplet with the same question and different image and answer. In contrast, we do not rely on such expensive annotation schemes. SynthTriplets18M [21] uses the text-conditioned image editing framework InstructPix2Pix [8] to automatically generate CoIR data. Their edit text generation process is similar to ours, but our generation process differs in that we automatically mine similar videos from a dataset of unlabeled video-text pairs to construct CoVR triplets instead of generating visual data. In experiments, we show the superiority of our generation procedure as we achieve much higher CoIR results (e.g., 38% vs 19% zero-shot R@1 on CIRR while generating fewer data). Lastly, our WebVid-CoVR dataset is composed of videos, and not limited to still images.

Vision-language pretraining. Many strong multi-modal models have been pretrained on large datasets of image-caption pairs [2, 13, 24, 27, 30, 32, 34, 38, 45, 48, 51, 67, 71] or video-caption pairs [1, 29, 33, 41, 42, 53, 59, 60, 68, 69, 70]. In contrast, we generate CoVR training data from video-caption pairs instead of directly training on them. Our data generation approach is also related to other generation approaches used for other tasks, e.g., action recognition [43], visual question answering [62, 63] and visual dialog [35]. However, unlike all these tasks, the CoVR task requires retrieving visual data.

Video retrieval. Text-to-video retrieval has received great attention over the last few years [17, 18, 94 19, 36, 39, 40, 46, 59, 61, 64, 65]. We also make use of multiple video frames with query scoring 95 similar to [5]. However, different from these methods, we focus on composed video retrieval, where 96 the query consists of both text and visual data. 97

3 Automatic Triplet Generation and CoVR Training 98

99

100

101

102

103

104

The goal of the composed video retrieval (CoVR) task is, given an input video or image q and a modification text t , to retrieve a modified video v in a large database of videos. We wish to avoid the manual annotation of ( q, t, v ) triplets for training. Hence we automatically generate such triplets from Web-scraped video-caption pairs, as explained in Section 3.1 and illustrated in Figure 2. The resulting WebVid-CoVR dataset, together with its manually curated evaluation set, is presented in Section 3.2. Finally, we present how we train a CoVR model using WebVid-CoVR in Section 3.3.

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

3.1 Generating composed video retrieval triplets

Given a large (Web-scraped) dataset of video-caption pairs ( v, c ) , we wish to automatically generate video-text-video CoVR triplets ( q, t, v ) where the text t describes a modification to the visual query q . However, the dataset of video-caption pairs neither contains annotations of paired videos, nor modification text that describes their difference. Hence we propose a methodology to automatically mine paired videos and describe their difference, as described below. Note that for illustration, we take as an example the WebVid2M dataset [4] with 2.5M video-caption pairs, but this methodology could be applied to other large datasets of video-text (or image-text) pairs.

Mining paired videos by pairing captions. In order to obtain paired videos, we leverage their captions. The core idea is that videos with similar captions are likely to have similar visual content. Specifically, we consider captions that differ by a single word, excluding punctuation marks. For instance, the caption "Young woman smiling" is paired with "Old woman smiling" and "Young couple smiling" . In the 2M distinct captions from WebVid2M, this process allows us to identify a vast pool of 1.2M distinct caption pairs with 177K distinct captions, resulting in 3.1M paired videos.

Filtering caption pairs. We wish to automatically generate the modification text between paired videos using their (paired) captions. However, caption pairs with the same meaning are likely to result in meaningless differences. On the contrary, caption pairs that differ too much are likely to result in large visual differences that cannot be easily described. To address these issues, we filter out caption pairs that are too similar and too dissimilar. Specifically, we exclude caption pairs with CLIP text embedding similarity ≥ 0.96 (e.g., "Fit and happy young couple playing in the park" and "Fit and happy young couple play in the park" ) and caption pairs with CLIP text embedding similarity ≤ 0.6 (e.g., "Zebra on a white background" and "Coins on a white background" ). We also exclude pairs where the captions differ by a digit (which mostly consist of date in practice), or by an out-of-vocabulary word. Finally, we remove templated captions such as "abstract of" , "concept of" , and "flag of" which are over-represented.

Generating a modification text from paired captions. In order to generate a modification text between paired videos, we apply a modification text generation large language model (MTG-LLM) to their corresponding paired captions. We describe the MTG-LLM inference process below and then explain its training details. The MTG-LLM takes as input two paired captions and generates a modification text that describes the difference between the two captions (see Fig. 2). In detail, the generation is auto-regressive, i.e., we recursively sample from the token likelihood distribution conditioned on the previously generated tokens until an end-of-sentence token is reached. To increase the diversity of the generated samples, we use top-k sampling instead of maximum-likelihood-based methods such as beam search and its variants [56]. Note that we only generate a single modification text per caption pair for computational efficiency, but the MTG-LLM could be used to generate multiple modification texts per caption pair which could serve as a data augmentation in future work.

We now describe the training details of the MTG-LLM. We start from a LLM pretrained with a next token prediction objective on a Web-scale text dataset [54]. We then finetune this LLM for the MTG task on a manually annotated text dataset. In particular, we repurpose the editing dataset from InstructPix2Pix [8], which provides a modification text and a target caption for 700 input captions. We augment this dataset with 15 additional annotations that are useful in our use case. These examples involve transformations such as changing singular nouns to plural ( tree to trees ), as well as addressing specific edge cases. More details can be found in the Supplementary Material.

Filtering video pairs. We wish to avoid some modification texts being over-represented in the dataset as it could harm training. Hence, if there are more than 10 video pairs associated with the same pair of captions (therefore leading to the same modification text), we only select 10 video pairs. As the CoVR task typically involves similar query-target video pairs, we choose pairs of videos with

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

Cookvegetables

Changetograzing

Cooking steak

Cooking vegetables

Horse eatinginfield

Horse grazing in field

makehimlookmorecompassionate

makea3dprinter

Angry younghandsome caucasian man

shows his emotion close-up

Compassionateyounghandsomecaucasian

man shows his emotion close-up

3d printing

3dprinter

the highest visual similarity, as measured by the CLIP visual embedding similarity computed at the middle frame of the videos.

3.2 Analysis of WebVid-CoVR

WebVid-CoVR: a large-scale CoVR training dataset. By applying the previously described pipeline to the WebVid2M dataset [4], we generate WebVid-CoVR, a dataset containing 1.6M CoVR triplets, which is significantly more than prior datasets (see Table 1). On average, a video lasts 16.8 seconds, a modification text contains 4.8 words, and one target video is associated with 12.7 triplets. WebVid-CoVR is highly diverse with 131K distinct videos and 467K distinct modification texts. Examples of CoVR triplets from the WebVid-CoVR dataset are illustrated in Figure 3. These examples show the diversity of the data in WebVid-CoVR, and its noise due to the automatic generation procedure. We provide further analysis of the WebVid-CoVR dataset in the supplementary material.

WebVid-CoVR m : a new CoVR evaluation benchmark. Due to the noise in WebVid-CoVR, we manually annotate a small test set, dubbed WebVid-CoVR m , for evaluation. For this, we first repeat the data generation procedure described in Section 3.1, but on a different corpus of video-caption pairs. Specifically, we consider video-caption pairs from the WebVid10M corpus [4] that are not included in the WebVid2M dataset, resulting in a pool of 8 million video-caption pairs. This ensures that other models using WebVid2M for pretraining have not been exposed to any of the test examples. In the video pairs filtering stage, for each pair of captions, we here only keep one pair of videos (the one with the highest visual similarity). This results in 163K candidate triplets that could be used for testing purposes. We randomly sample 7K triplets that we use for validation and randomly sample 3.1K other triplets that we manually annotate as described below.

We augment the 3.1K triplets by generating two additional modification texts with the MTG-LLM. The annotator reads the three generated modification texts, looks at three frames from the query and target videos, and either keeps the best modification text if at least one is valid or discards the sample. Through this meticulous annotation process, we ensure that the test set comprises high-quality and meaningful CoVR triplets. This results in a test set of 2.4K triplets, i.e., about 23% of the examples are considered as noisy and are discarded.

3.3 Training on WebVid-CoVR

Here, we describe our CoVR model architecture and how we train it on our WebVid-CoVR dataset. 181 CoVR-BLIP model architecture. Our model architecture builds upon a pretrained image-text model, 182 BLIP [31]. The BLIP model is pretrained on a large dataset of image-caption pairs with three vision183 language objectives: image-text contrastive learning, image-text matching, and image-conditioned 184 language modeling. However, BLIP is not pretrained for composed visual retrieval with both visual 185 and text inputs. Therefore we adapt BLIP to the CoIR/CoVR task as follows. 186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

We use the BLIP image encoder to encode the image query. The resulting visual features and the modification text are then forwarded to the BLIP image-grounded text encoder together, which outputs a multi-modal embedding f i ∈ R d where d is the embedding dimension. To retrieve a target video from a database of videos V , we compute embedding vectors for all possible videos as follows. We uniformly sample N frames from the video and compute a weighted mean of the BLIP image embeddings to obtain the video embedding vector ˆ v ∈ R d . The weights are obtained by computing the image-caption similarity for every video frame with BLIP image and text encoder, respectively, similar to [4] in the context of text-to-video retrieval. Finally, given a multi-modal embedding f i , the retrieved video is the one that maximizes the embedding similarity, i.e., arg max v ∈ V (ˆ v.f T i ) .

Training. In order to train on WebVid-CoVR, we use a contrastive learning approach [44, 55], as it has been shown to be effective to learn strong multi-modal representations from large-scale noisy data [41, 45]. We make several design choices to maximize its efficiency. First, we create a training batch by sampling distinct target videos and for each target video, we randomly sample an associated query image and modification text. This ensures that the same target video appears only once in a batch and maximizes the number of different target videos that can be used as negatives in contrastive learning.

Second, following HN-NCE [44], we use as negatives all target videos v j ∈B in the batch B and additionally increase the weight of most similar samples. In addition, we mine hard negative samples that we select based on the captions associated with the videos in WebVid2M. Specifically, for a given ( q i , t i , v i ) triplet, we consider as hard negatives all instances in the batch ( q j , t j , v j ) ∈ HN ( i ) where q i and q j have the same caption but v i and v j have different captions. In addition, to reduce the number of noisy negatives with the same semantic content as a given sample i , we exclude from the computation of the loss samples ( q j , t j , v j ) ∈ P ( i ) for which v i and v j have the same caption.

Formally, given a training batch B of triplets ( q i , t i , v i ) , we minimize the following loss:

where α and τ are learnable parameters, S i,j is the cosine similarity between the multi-modal embedding f i and the target video embedding ˆ v i , HN ( i ) is the set of hard negatives, P ( i ) is the set of noisy negatives and w i,j is set as in [44].

4 Experiments

In this Section, we first describe the experimental protocol including the datasets, evaluation metrics, and implementation details (Section 4.1). We then present the results of CoVR on our new video benchmark (Section 4.2), as well as transfer results of CoIR on standard image benchmarks (Section 4.3). Finally, we provide ablations on our key components (Section 4.4).

4.1 Experimental setup

Datasets. WebVid-CoVR is our proposed training CoVR dataset presented in Section 3.2, and WebVid-CoVR m is our new CoVR benchmark presented in Section 3.2.

CIRR [37] is a manually annotated CoIR dataset that contains open-domain natural images from NLVR2 [52]. It contains 36.5K queries annotated on 19K different images. CIRR includes two benchmarks: a standard one with the target search space as the entire validation corpus, and a fine-grained subset , where the search space is a subgroup of six images similar to the query image (based on pretrained ResNet15 feature distance). The dataset is divided into training, validation, and testing splits with 28,225/16,742, 4,181/2,265 and 4,148/2,178 queries/images, respectively.

FashionIQ [58] is a CoIR dataset that contains images of fashion products, divided into three categories of Shirts, Dresses, and Tops/Tees. The query and target images were automatically paired based on title similarities (crawled from the web), and modification texts were then manually annotated. This dataset consists of 30K queries annotated on 40.5K different images. It is divided into training and validation splits with 18,000/45,429 and 6,016/15,415 queries/images, respectively.

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

Evaluation metrics. Following standard evaluation protocols [37], we report the video retrieval recall at rank 1, 5, 10, and 50. Recall at rank k (R@k) quantifies the number of times the correct video is among the top k results. MeanR denotes the average of R@1, R@5, R@10, and R@50. Higher recall means better performance.

Implementation details. For our MTG-LLM, we use LLaMA 7B model [54] that we finetune for one epoch with an initial learning rate of 3 e -5 for MTG. For our CoVR model, we use the BLIP with ViT-L [16] at 384 pixels finetuned for text-image retrieval on COCO and freeze the ViT for computational efficiency. We train our CoVR model on WebVid-CoVR for 3 epochs with a batch size of 2048 and an initial learning rate of 1 e -5 . To finetune on CIRR/FashionIQ, we train for 6/3 epochs with a batch size of 2048/1024 and an initial learning rate of 5 e -5 / 1 e -4 . Experiments are conducted on 4 NVIDIA A100-SXM4-80GB GPUs. More details are included in the Supplementary Material.

4.2 Composed video retrieval results

We report CoVR results on our WebVid-CoVR m test set in Table 2. For models trained on WebVidCoVR, we find that using both modalities is crucial for performance, as the model with visual and text inputs outperforms both the text-only and the visual-only models. Furthermore, using multiple target video frames is beneficial, as the model with 15 frames improves over the model with 1 frame.

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

We also evaluate baselines that are not trained on WebVid-CoVR and that directly apply the pretrained BLIP model [31] to the CoVR task. These baselines outperform the random baseline but underperform compared to models trained on WebVid-CoVR, showing the benefit of our automatically generated training dataset. Note that BLIP [31] is pretrained for image-text retrieval but not for image-textimage retrieval, hence the drop in performance when applied directly to CoVR with both input modalities compared to only using visual information.

4.3 Transfer learning to composed image retrieval

While our focus is video retrieval, we also experiment with transferring our CoVR models to image retrieval tasks on standard CoIR benchmarks. We define zero-shot CoIR as not using any manually annotated CoIR triplet for training. We perform zero-shot CoIR by directly applying our model trained on our automatically generated WebVid-CoVR dataset to CoIR tasks and also explore finetuning our model on the training set of the downstream benchmark.

Tables 3 and 4 report results on CIRR and Fashion-IQ datasets, respectively. These results show that our model highly benefits from training on WebVid-CoVR, especially in the zero-shot setting, on both datasets. In addition, our model achieves state-of-the-art zero-shot performance on both CIRR and FashionIQ, and performs competitively in the finetuning setting on both benchmarks.

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

4.4 Ablation studies

In this Section, we ablate the importance of several key aspects of our method by evaluating the downstream performance of the model trained only on WebVid-CoVR.

Importance of data scale. In Table 5, we evaluate the importance of the scale of the dataset of video-captions used in our generation pipeline. We construct subsets of videos such that larger ones include smaller ones, and only keep triplets that contain the sampled videos for training. We find that results steadily increase when using more videos, demonstrating that our method largely benefits from scaling the size of the seed dataset of video-captions. We also observe the importance of the filtering techniques described in Section 3.1, as the model trained on unfiltered generated data underperforms.

Modification text generation. We use a large language model finetuned for modification text generation as explained in Section 3.1. We here compare this solution to a rule-based baseline that uses several templates to generate the modification text given the two captions that differ by one word. Specifically, the modification text is based on the two different words from the captions. We generate templates that use these words and chose one at random during training. These templates include variations such as "Remove txt _ diff 1 " and "Change txt _ diff 1 for txt _ diff 2 " . A full list of all the templates can be seen in the Supplementary Material. In Table 6, we show that our large language model generates better modification texts than the rule-based baseline, by evaluating the results of the model trained on the generated data. Qualitative examples comparing the two approaches are provided in the Supplementary Material.

Training strategies. In Table 7, we first show the benefit on WebVid-CoVR of training by iterating on target videos instead of CoVR triplets. This is to avoid having the same target video appearing multiple times in a training batch, hence increasing the number of correct negatives that are used in the contrastive loss. Furthermore, sampling hard negatives, as described in Section 3.3, also slightly benefits the downstream performance.

5 Conclusions, Limitations, and Societal Impacts

In this work, we studied the new task of CoVR by proposing a simple yet effective methodology to create automatic training data. Our results on several benchmarks (including our manually curated video benchmark, as well as existing image benchmarks) suggest that, while noisy, such an automated and scalable approach can provide effective CoVR model training. One potential limitation of our method is that our dataset may not depict some visible changes due to the way we generate triplets. Moroever, our modification text generation model is suboptimal due to only inputting text (i.e., without looking at images). Future work can incorporate visually grounded modification generation.

Societal impact. Our model constitutes a generic multi-modal search tool, but is not intended for a specific application. While there are helpful use cases such as online shopping, traveling, and personal development (i.e., how-to), there may be potential privacy risks associated to surveillance applications, searching for a specific person in videos.

301

302

303

304

305

306

307

References

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS , 2021. 3

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022. 3

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence 308 Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV , 2015. 3 309

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video 310 and image encoder for end-to-end retrieval. In ICCV , 2021. 2, 4, 5, 6 311

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A CLIP-hitchhiker's guide to 312 long video retrieval. arXiv:2205.08508 , 2022. 2, 3 313

Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot 314 composed image retrieval with textual inversion. arXiv:2303.15247 , 2023. 2, 3 315

316

317

Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining CLIP-based features. In CVPR , 2022. 2, 3, 7, 8

Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow 318 image editing instructions. arXiv:2211.09800 , 2022. 2, 3, 4 319

320

321

322

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. In CVPR , 2021. 8

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, 323 and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. 324 arXiv:1504.00325 , 2015. 7 325

Yanbei Chen and Loris Bazzani. Learning joint visual semantic matching embeddings for 326 language-guided retrieval. In ECCV , 2020. 8 327

Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolin328 guistic attention learning. In CVPR , 2020. 8 329

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, 330 and Jingjing Liu. UNITER: Universal image-text representation learning. In ECCV , 2020. 3 331

Ginger Delmas, Rafael Sampaio de Rezende, Gabriela Csurka, and Diane Larlus. ARTEMIS: 332 Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity. In ICLR , 2022. 333 3, 7, 8 334

Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye. Modality-agnostic 335 attention fusion for visual search with text feedback. CoRR , abs/2007.00145, 2020. 7, 8 336

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, 337 Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, 338 Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image 339 recognition at scale. In ICLR , 2021. 7 340

Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. CLIP2Video: Mastering video-text retrieval 341 via image clip. arXiv:2106.11097 , 2021. 3 342

Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. CLIP2TV: 343 an empirical study on transformer-based methods for video-text retrieval. arXiv:2111.05610 , 344 2021. 3 345

Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridgeformer: 346 Bridging video-text retrieval with multiple choice questions. In CVPR , 2022. 3 347

Sonam Goenka, Zhaoheng Zheng, Ayush Jaiswal, Rakesh Chada, Yue Wu, Varsha Hedau, 348 and Pradeep Natarajan. FashionVLP: Vision language transformer for fashion retrieval with 349 feedback. In CVPR , 2022. 8 350

351

352

353

Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun. CompoDiff: Versatile composed image retrieval with latent diffusion. arXiv:2303.11916 , 2023. 2, 3, 7

354

355

356

357

358

359

360

361

362

363

364

365

366

Surgan Jandial, Pinkesh Badjatiya, Pranit Chawla, Ayush Chopra, Mausoom Sarkar, and Balaji Krishnamurthy. SAC: Semantic attention composition for text-conditioned image retrieval. In WACV , 2022. 8

Surgan Jandial, Ayush Chopra, Pinkesh Badjatiya, Pranit Chawla, Mausoom Sarkar, and Balaji Krishnamurthy. TRACE: Transform aggregate and compose visiolinguistic representations for image search with text feedback. CoRR , abs/2009.01485, 2020. 8

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021. 3

Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim. Dual compositional learning in interactive image retrieval. AAAI , 2021. 3, 8

Seungmin Lee, Dongwan Kim, and Bohyung Han. CoSMo: Content-style modulation for image retrieval with text feedback. In CVPR , 2021. 8

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less 367 is more: ClipBERT for video-and-language learning via sparse sampling. In CVPR , 2021. 3 368

Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data roaming and early fusion 369 for composed image retrieval. arXiv:2303.09429 , 2023. 2, 3, 7, 8 370

Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and 371 prompt: Video-and-language pre-training with entity prompts. In CVPR , 2022. 3 372

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image 373 pre-training with frozen image encoders and large language models. In ICML , 2023. 3 374

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping language375 image pre-training for unified vision-language understanding and generation. In ICML , 2022. 2, 376 5, 8 377

378

379

380

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , 2021. 3

381

382

383

384

385

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for video+language omni-representation pre-training. In EMNLP , 2020. 3

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV , 2020. 3

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 386 arXiv:2304.08485 , 2023. 3 387

Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. TS2-Net: Token shift and 388 selection transformer for text-video retrieval. In ECCV , 2022. 3 389

Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval 390 on real-life images with pre-trained vision-and-language models. In ICCV , 2021. 2, 3, 6, 7, 8 391

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic 392 visiolinguistic representations for vision-and-language tasks. In NeurIPS , 2019. 3 393

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: 394 An empirical study of CLIP for end to end video clip retrieval. arXiv:2104.08860 , 2021. 3 395

Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-CLIP: 396 End-to-end multi-grained contrastive learning for video-text retrieval. In ACMMM , 2022. 3 397

398

399

400

Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR , 2020. 3, 6

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV , 2019. 3

Arsha Nagrani, Chen Sun, David Ross, Rahul Sukthankar, Cordelia Schmid, and Andrew Zisserman. Speech2action: Cross-modal supervision for action recognition. In CVPR , 2020. 3

Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. In arXiv , 2023. 6

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021. 2, 3, 6

Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In CVPR , 2023. 3

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. CVPR , 2023. 3, 7, 8

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION5B: An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks Track , 2022. 3

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , 2018. 7

Minchul Shin, Yoonjae Cho, ByungSoo Ko, and Geonmo Gu. RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network. arXiv:2104.03015 , 2021. 8

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of generic visual-linguistic representations. In ICLR , 2019. 3

Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In ACL , 2019. 6

Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. In NeurIPS , 2022. 3

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv:2302.13971 , 2023. 2, 4, 7

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv:1807.03748 , 2018. 2, 6

Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, 441 David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural 442 sequence models. arXiv:1610.02424 , 2016. 4 443

Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing 444 text and image for image retrieval - an empirical odyssey. In CVPR , 2019. 2, 3, 7 445

446

447

448

Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogério Feris. Fashion IQ: A new dataset towards retrieving images by natural language feedback. In CVPR , 2021. 2, 3, 6

449

450

451

Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In EMNLP , 2021. 3

Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, 452 and Baining Guo. Advancing high-resolution video-language representation with large-scale 453 video transcriptions. In CVPR , 2022. 3 454

Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. 455 CLIP-ViP: Adapting pre-trained image-text model to video-language representation alignment. 456 arXiv , 2022. 3 457

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: 458 Learning to answer questions from millions of narrated videos. In ICCV , 2021. 3 459

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Learning to 460 answer visual questions from web videos. IEEE TPAMI , 2022. 3 461

Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. TACo: Token-aware cascade contrastive 462 learning for video-text alignment. arXiv , 2021. 3 463

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, 464 Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image 465 pre-training. In ICLR , 2022. 3 466

Youngjae Yu, Seunghwan Lee, Yuncheol Choi, and Gunhee Kim. CurlingNet: Compositional 467 learning between images and text for fashionIQ data. arXiv:2003.1229 , 2020. 8 468

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong 469 Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for 470 computer vision. arXiv:2111.11432 , 2021. 3 471

Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, 472 Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERLOT Reserve: Neural script 473 knowledge through vision and language and sound. In CVPR , 2022. 3 474

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, 475 and Yejin Choi. MERLOT: Multimodal neural script knowledge models. In NeurIPS , 2021. 3 476

Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations 477 from large language models. In CVPR , 2023. 3 478

Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. 479 Unified vision-language pre-training for image captioning and VQA. In AAAI , 2020. 3 480

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the task of Compositional Video Retrieval (CoVR) in which given a video and a text which modifies the video aims to rank and retrieve the modified video. In this work, a new large-scale dataset for pre-training (named WebVid-CoVR) is developed with generated modifications from a Large Language Model (LLM). Additionally, a smaller dataset for evaluation is manually annotated. Alongside this, a method is proposed that follows HH-NCE to learn the video compositions. Experiments show that the proposed method works well in both the supervised and zero-shot cases, and even translates well to Compositional Image Retrieval (CoIR) datasets CIRR and FashionIQ.\n\nSTRENGTHS:\nThe created dataset(s) looks like it will be a useful addition to vision-language pre-training and especially for the Composed Video Retrieval task which currently doesn't have a dataset to train on.\n\nThe methodology of creating the dataset is well-done and all the steps make sense, especially for the manually annotated portion of the test set.\n\nThe paper is generally well-written and easy to follow.\n\nWEAKNESSES:\nLine 137: Is the Top K sampling referring to the tokens within a modification text? If not it's not clear what this means as only a single modification is generated as specified on Line 138.\n\nRegarding the rule-based ablation in Table 6, could the generated rules be paraphrased by a LLM? In this case, the MTG-LLM wouldn't necessarily need to be fine-tuned and a standard LLM could be used instead.\n\nLine 164: The paper mentions that WebVid-CoVR is noisy which is why a subset is manually annotated for the test set. What is the main source of noise within the main dataset? Is this from the generated modifications? The captions not matching the videos or something else? Has a human study shown how much of the dataset can be considered noisy from looking at a small number of samples?\n\n\\alpha and \\tau are learnable parameters? Normally, \\tau at least, is a set hyperparameter. How these are learnt aren't mentioned within the text.\n\nHas any analysis been performed into the types of modifications within the dataset? I.e. noun/adjective/verb changes and others? It is mentioned within the limitations that certain modifications may not have been generated and it would be good to see what coverage there could be - even if this is from a human annotated subset.\n\nQUESTIONS:\nFor more context regarding the questions, please see the weaknesses section above.\n1. Has any analysis been performed into the types of modifications within the dataset?\n2. What is the main source of noise within the main dataset?\n3. How are \\alpha and \\tau learnt?\n4. How does learning \\tau compare to keeping it a fixed value as is normally done in practice? How does \\tau change during training?\n5. Could the generated rules be paraphrased by a LLM? Would this change the performance within Table 6?\n6. Is the Top K sampling referring to the tokens within a modification text?\n\nLIMITATIONS:\nThere is a short section on limitations which mentions the scope of modifications within the dataset, I think this could be improved by performing some analysis on the generated prompts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of composed video retrieval, which involves querying a video and a modification text to find videos that exhibit similar visual characteristics with the desired modification. The main challenge is the lack of data for composed video retrieval. To overcome this, the paper proposes mining paired videos with similar captions from a large database and generating the corresponding modification text using a large language model. The paper explains the BLIP-based video and text encoder and the training process for the model using the collected data. Experimental results demonstrate that the model trained on the compiled dataset can generalize to both zero-shot and fine-tuning settings.\n\nSTRENGTHS:\n- The paper introduces a novel task of video retrieval and establishes a benchmark for future research in this area.\n- Overall, the paper is well-written, clear, and easy to follow.\n\nWEAKNESSES:\n- Examples in Figure 3 suggest that most samples do not require handling dynamic content, implying that there may not be a significant difference between the proposed CoVR task and existing CoIR tasks.\n- The MTG-LLM method requires manually created data for fine-tuning, which can be resource-intensive.\n- The training method is standard and not particularly innovative, although it is reasonable for the task.\n- The training data is limited to modifications that can be represented with single-word differences, potentially excluding other types of modifications. This point is mentioned as a limitation in the paper. By providing concrete examples of scenarios not addressed by this work, readers will understand the challenge clearly.\n\nQUESTIONS:\n- What annotations are added to the dataset? (line 145)\n\nLIMITATIONS:\nThe paper mentions two limitations.\n- The data creation pipeline may not adequately capture some visible changes due to its design.\n- The generation of modification text may not be optimal as the text generation depends only on input captions.\n\nI appreciate the authors for pointing out the meaningful limitations and suggesting ideas for future research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a scalable approach to automatically generate composed visual retrieval training data. Specifically, based on the WebVid2M dataset, the authors generates a WebVid-CoVR training dataset with 1.6M CoVR triplets.

STRENGTHS:
The data augmentation strategy is scalable.

WEAKNESSES:
The overhead for the dataset augmentation should be detailed.

QUESTIONS:
The overhead for the dataset augmentation should be detailed.

LIMITATIONS:
The overhead for the dataset augmentation should be detailed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

comments: SUMMARY:
This paper automatically constructs a new dataset called WebVid-CoVR by applying a scalable automatic dataset creation procedure that generates triplets from video-caption pairs to a large-scale WebVid2M collection, resulting in 1.6M triplets. Moreover, this paper introduces a new benchmark for composed video retrieval(CoVR) and contribute a manually annotated evaluation set, along with baseline results. The results demonstrate that training a CoVR model on WebVid-CoVR transfers well to CoIR with competitive performance.

STRENGTHS:
- The automatic triplet generation pipeline is carefully designed with many phases.
- Provides strong baseline results on CoVR and shows transferability to CoIR.

WEAKNESSES:
- The automatic triplet generation pipeline seems to rely only on caption similarity, as well as MTG-LLM, which may introduce noise and ignore visual similarity. Visual similarity between videos should also be taken into account.
- More dataset analysis, especially about the visual part, should be provided to gain more understanding of the characteristics of WebVid-CoVR. Besides, the human check may be necessary.
- Only one model (CoVR-BLIP) is evaluated on the proposed CoVR task. More baselines like frozen/finetuned multimodal transformers could be added.

QUESTIONS:
Please refer to Weaknesses.

LIMITATIONS:
Please refer to Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-16