Title: Data Curation for Image Captioning with Text-to-Image Generative Models
Venue: NeurIPS 2023
Accepted: no
Keywords: Vision-language learning

Image captioning

Data curation

Text-to-image generation

Stable Diffusion
Abstract: 
    Recent advances in image captioning are driven by increasingly larger-scale vision--language pretraining, relying on massive computational resources and increasingly large datasets. Instead of solely focusing on scaling pretraining, we ask whether it is possible to improve performance by improving the quality of the samples in existing datasets. We pursue this question through two approaches to data curation: one that assumes that some examples should be avoided due to mismatches between the image and caption, and one that assumes that the mismatch can be addressed by replacing the image, for which we use the state-of-the-art Stable Diffusion model. These approaches are evaluated using the BLIP model on the COCO and Flickr30K datasets. Models trained with our data curation approaches consistently outperform their baselines, indicating that better image captioning models can be trained by curating existing resources. Finally, we conduct a human study to understand the errors made by the Stable Diffusion model and highlight directions for future work in text-to-image generation.

Abstract

Recent advances in image captioning are driven by increasingly larger-scale visionlanguage pretraining, relying on massive computational resources and increasingly large datasets. Instead of solely focusing on scaling pretraining, we ask whether it is possible to improve performance by improving the quality of the samples in existing datasets. We pursue this question through two approaches to data curation: one that assumes that some examples should be avoided due to mismatches between the image and caption, and one that assumes that the mismatch can be addressed by replacing the image, for which we use the state-of-the-art Stable Diffusion model. These approaches are evaluated using the BLIP model on the COCO and Flickr30K datasets. Models trained with our data curation approaches consistently outperform their baselines, indicating that better image captioning models can be trained by curating existing resources. Finally, we conduct a human study to understand the errors made by the Stable Diffusion model and highlight directions for future work in text-to-image generation.

1 Introduction

Large-scale vision-language pretraining has been the driving force behind recent advances in image captioning [14]. The amount of image-text data needed to pretrain recent generative language models [28, 23, 53] has made it necessary to train on 'noisy' samples harvested from the web [46, 45], as opposed to crowdsourced captions [32]. This emerging reliance on harvested data has made it important to perform additional filtering steps to remove low-quality data [28], in addition to more resource-intensive pretraining. Given that computing resources are not equally distributed [21], there is a need to also pursue less resource-intensive research directions.

We show how to improve image captioning by improving the quality of the downstream task data through data curation : the process of dynamically updating the samples during training. We devise three techniques for data curation that are designed to prevent the total size of the dataset from increasing: the complete removal of an image-caption sample from a dataset; replacing a caption with another caption; and replacing images using a text-to-image generation model [41]. These curation techniques are used to update image-caption samples that have outlier losses, with respect to the rest of a training dataset, under the current model parameters. In other words, the samples that are proving difficult to model. Also, the synthesis of completely new images is radically different from standard data augmentation techniques, such as random cropping or color manipulation [47], or swapping and mask words in text [12].

We conduct experiments using BLIP [28], a strong image captioning model, on the Flickr30K [56] 33 and MS COCO datasets [32]. The results show that the sample removal and image replacement 34 techniques lead to consistent improvements of 1-3 CIDEr points compared to not curating the 35 dataset. Our analyses show that Flickr30K benefits from more curation than COCO due to differences 36

BLIP

Original data

BLIP

Curated data

D

0

D

1

A group of dancers

getting ready to

perform.

Low Loss

High Loss

Reenactment of old

fashion waltzing with

costumes - 3 couples

The Olde English

exhibits to park …..

A group of dancers

getting ready to

perform.

Remove

Replace

Caption

Dynamic removal or replacement

Method

SD-based Image Replacement

Method

Two people in helmets

walking through bushes.

Two people in helmets

walking through bushes.

Data curation

Data curation

BLIP

Curated data

D

2

…

…

Keep

T

Original Image

SD-synthesized Image

in the distribution of long captions in each dataset. Finally, we find that it is better to curate the 37 data dynamically while training instead of replacing images before starting to train the model. 38 Taken together, these findings show the promise of model-in-the-loop text-to-image generation for 39 multimodal learning, while highlighting that improvements in text-to-image generation are likely to 40 further enhance the effectiveness of data curation. 41

2 Related work 42

Image Captioning Image Captioning is the task of describing images with syntactically and 43 semantically sentences. Current deep learning-based image captioning models have evolved as 44 the encode-decoder frameworks with multi-modal connection [8, 9], attentive [24, 16] and fusion 45 strategies [58]. Standard captioning datasets contain Flickr30K [56] and the commonly used MS 46 COCO [32], which consisting of images with events, objects and scenes. Each image is paired with 47 five captions. Some works have demonstrated the benefits of training on synthetic captions [29, 3] or 48 datasets collected from other vision-and-language learning tasks [38, 7]. 49

50

51

52

53

54

55

56

57

58

Data Augmentation Data augmentation [13] has achieved increasing attention in both natural language processing [33] and vision-and-language learning [27]. Early methods generate augmented examples in the model's feature space [54] or interpolate the inputs and labels of few examples [57]. For downstream tasks in the text domain, Yang et al. [55] and Anaby-Tavor et al. [1] generate synthetic text examples through state-of-the-art pretrained language models and show improved performance on common-sense reasoning and text-classification. For image captioning, BERT [11] has been used to generate additional captions to improve the diversity of the captioning datasets [3]. Hossain et al. [22] used GAN-synthesized images as additional augmentation training set to improve image captioning models.

Diffusion Models and Application Diffusion models [49, 35] have grown rapidly and become 59 the powerful deep generative models. They have shown potential in a variety of applications, 60 including text-to-image generation [36, 15], image-to-image translation [42], as well as semantic 61 segmentation [26, 5] and video generation [20, 48, 52]. While recent large scale latent diffusion 62 models have shown strong capability in generating both artistic and photo-realistic high-resolution 63 images [41, 34, 39, 43], applying large-scale stable diffusion models in vision-language downstream 64 tasks remains under-explored. Concurrently, Azizi et al. [4] and Jain et al. [25] show that image 65 classifiers can be improved by learning from augmentation images generated by finetuned stable66 diffusion models. To the best of our knowledge, we are the first to explore how image captioning 67 models can benefit from simple data curation without scaling up existing datasets, and how stable68 diffusion text-to-image models can be applied and contribute in the process. 69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

3 Data Curation for Captioning

Our goal is to improve image captioning models by preventing the model from training on difficult samples. There are many reasons for the possible existence of these difficult samples, including mismatches or inconsistencies between the image and caption [3]. More formally, given an image captioning training dataset D with K images, let I k be the k -th image. Each image is paired with J captions; let C j k be j th caption of image k , and thus, let (I k , C j k ) be an image-caption sample in the dataset. Assume the existence of model M , which is being trained on dataset D , from which we can calculate the loss of each sample at each epoch t : L t M (I k , C j k ), which can be used to track the difficult samples. At the end of each epoch, the difficult samples are candidates for our data curation techniques, resulting in dynamic updates to the training dataset D → D 1 →··· → D T .

3.1 Identifying the difficult samples

Difficult training samples may contain mismatches or inconsistencies between the image and the caption [3]. We propose to use the captioning model that is being trained to automatically identify such samples. After each epoch, we compute the loss of each sample in the current training dataset, given the current model parameters. The highest loss samples are targets for our data curation methods; more specifically, we focus on samples with losses that are either two standard deviations from the mean, or a fixed X% away e.g. 10%, 20%, etc. In this way, the training dataset is dynamically updated at the end of each epoch according to the model's captioning capability. The adjacent figure shows the empirical distribution of losses in the training samples of the Flickr30K dataset. It is clear that, without data curation, the high-loss samples remain high-loss during five epochs of training.

3.2 Sample Removal / Caption Replacement

The simplest approach to data curation is to remove or replace the high-loss samples. In REMOVE, the high-loss samples are completely removed from the remainder of the training process, reducing the total number of image-caption training samples. In REPLACECAP, we simply replace the caption in the image-caption sample with a different caption taken from the other captions that describe the image, effectively creating a duplicate. With the caption replacement method, the total number of samples used to train the model remains the same, as well as the total number of the unique images. This creates a clean control condition for the subsequent experiments.

3.3 Image Generation-based Replacement

An alternative to removing difficult samples or replacing captions is to pair an existing caption with a new image. This has the benefit of training the model on the same total number of samples while exposing it to more unique images. The new image could be found by humans, in a long-running human-in-the-loop cycle. Instead, we use a text-to-image generation model, in a rapid model-inthe-loop step, to synthesize images based on the other sentences that describe the image. Some representative examples of images generated using this technique can be seen in Figure 10.

Our methodology is based on the open source Stable Diffusion model [41], which can generate images given a textual prompt. 1 We integrate this into training as follows: Given an image I k in the training data and its captions { ( I k , C 1 k ) , . . . , ( I k , C J k ) } , we synthesize a new image ˆ I k without increasing the total number of samples in the original dataset. Instead, we replace the original image in the sample with the generated image. Specifically, for image I k , we replace a high-loss sample ( I k , C j k ) with the synthesized image-text pair ( ˆ I k , C j k ) .

0

50

100

150

200

Loss value

0

250

500

750

1000

1250

1500

1750

2000

Count

0

1

2

3

4

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

Round-trip captioning evaluation

In order to effectively use a text-to-image generation model for data curation, we need an objective measure that can estimate the expected quality of a generated image. Most previous work uses image-oriented measures like FID [19] or CLIPScore [17] but these measures are claimed to lack alignment with perceptual quality [44]. We also found they were not suitable for our purpose, and that CLIPScore cannot distinguish between low- and high-loss samples in the captioning model (Figure 9). Here, we propose an alternative that is directly related to our task: given the generated image, measure the quality of the caption that can be generated by a fixed model.

Our assumption is that if the generated images are of a similar quality to the original images, the resulting captions should be similar to each other. We call this a round-trip captioning evaluation, which comprises three steps illustrated in Figure 3. In Step (1), we use the captions in the validation set to generate images using a text-toimage generation model. In Step (2), we use an existing image-captioning model to predict captions for the generated images. Specifically, we use BLIP fine-tuned on the COCO dataset but any other strong captioning model could be used instead. Finally, in Step (3), we compare the predicted captions against the original captions. We now discuss the the factors that we found make a difference when generating images.

Prompt engineering matters

Recall that text-to-image generation models produce images based on a textual prompts. Given a set of five captions that describe an image, there are several options for how to prompt the image generation model. We experiment with three options:

Single caption: Each caption is used in isolation to generate a new image.

Sentence-BERT selection: There is a lot of variety in how different captions describe the same image. Instead of using all captions, we can use a representative caption from the set. This is achieved using the Sentence-BERT [40] model to find the caption that is closest to the average embedding of all captions.

Concatenation: All five captions are concatenated as the text prompt for generation.

For all three approaches mentioned above, we can append an additional string to the prompt as a styler to force a specific style in the generated image (+Styler). The styler used here is: "national geographic, high quality photography, Canon EOS R3, Flickr". 2

Finetuning improves image relevance

Table 1 shows the results of the round-trip captioning evaluation on the Flickr30K dataset using different textual prompts and whether or not to fine-tune the diffusion model. When we fine-tune StableDiffusion, we use the MS COCO [32] dataset with a prompt consisting of a concatenation of all 5 captions, for 15,000 steps with a constant learning rate of 1 e -5 and a batch size of 32. The best performance is clearly found by fine-tuning Stable Diffusion 1.5 and using a prompt with a concatenation of the captions and the styler. We use this configuration in the remainder of the paper.

Stable Diffusion

Text2Img

Original Validation

SD Validation

Predicted Captions

Evaluation

Image

Captioning

a person in a blue jacket is

sitting against a wall covered

in graffiti

a person in a blue jacket is

sitting against a wall covered

in graffiti

A person in a blue jacket

is sitting against a wall

covered in graffiti

A person in a jacket and

wearing jeans kneels

down to take a picture of

a graffiti-laden wall.

Original Captions

Finetuned

BLIP

model

Image Synthesize

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

a soldier is taking a picture of a road

a soldier is looking through a scope

(a) Incorrect activity

a man is driving a tractor through a muddy field

a man is driving a jeep through a mud puddle

(b) Incorrect object

a woman in a black jacket is sitting on the ground

a woman is sitting on a stone bench

(c) Incorrect location

a woman standing in a kitchen preparing food

a woman washing a baby in a yellow tub

(d) Incorrect activity and object

4 Experiments

We evaluate our data curation methods on the MS COCO and Flickr30K datasets when finetuning the pretrained BLIP [28] model. We evaluate the captions using B LEU [37], M ETEOR [10], R OUGE [31], C IDEr [51], S PICE [2], C LIP S core, and R ef C LIP S core [18].

We use the ViT-based BLIP model [28] as our captioning model. We note that BLIP has a captioning and filtering (CapFilt) data augmentation process during its pretraining, where both components were finetuned on the COCO dataset. Therefore we use pretrained checkpoint BLIP CapFilt for Flickr30k and BLIP base for COCO in our experiment, removing the effects from the CapFilt process. We finetune BLIP using a batch size of 128 for 5 epochs on 4 × A100 GPUs.

4.1 Results

Removal/Caption Replacement As shown in Table 2, dynamically removing mismatched imagetext pairs or replacing captions can effectively improve performance on both datasets over baselines on all metrics. For Flickr30K, the dynamic updates work best when apply to the top 1% of high-loss samples for REPLACECAP, and to samples whose loss are two standard deviations higher than the mean for REMOVE. For COCO, both REPLACECAP and REMOVE works best when curating the top 1% of high-loss samples. We repeat that during the curation process, no additional data samples or computation cost is introduced. We further study the effect of the amount of curation in Section 5.

Image Generation-based Replacement We evaluate Image Generation-based Replacement on both the Flickr30K and COCO dataset. During finetuning, we replace images in the original textimage pairs with Stable Diffusion-synthesized images (ReplaceImg in Table 2). The results show improvements compared to the baseline in every evaluation measure with best performance obtained at replacement ratio of 40% for Flickr30K and at 10% for COCO. We show qualitative examples in Figure 4, where models finetuned with our proposed curation method can generate better captions for some scenes that may confuse the standard finetuned model. In Section 5.1, we analyze the effects of varying the amount of synthetic images replaced, and in Section 5.2, we conduct a human study of the types of errors found in the generated images.

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

1%

5%

2 std

10%

20%

40%

50%

0.0

0.5

1.0

1.5

2.0

2.5

3.0

CIDEr

Flickr30K

ReplaceCap

Remove

ReplaceImg

1%

5%

2 std

10%

20%

40%

1.5

1.0

0.5

0.0

0.5

1.0

CIDEr

COCO

ReplaceCap

Remove

ReplaceImg

5 Analysis and Discussion

5.1 Data Curation: how much and when?

We analyze how the amount of curation affects image captioning performance. We examine different ratios of training samples that are removed, replaced with an alternative caption, or replaced with a synthesized image. For REMOVE and REPLACECAP, we consider curation ratio of 1%, 5% and 10% of high-loss samples. For REPLACEIMG, we consider 10%-80% curation ratio. In addition to fixed X% ratios, we also intereven on samples that have losses two standard deviations worse than the mean.

Flickr30K needs more curation than COCO. The results of this analysis are shown in Figure 5. The best improvement in performance for Flickr30K is achieved either through removing high loss samples that are two standard deviations away, or replacing images for 40% of the high loss samples.

In the COCO dataset, replacing images for 10% of the high loss samples gives the best improvement compared to no data curation. The second best performing method for COCO is removing or replacing captions of only 1% of the high loss samples. This indicates that Flickr30K may contain more noisy samples than the MS COCO dataset. Compared to MS COCO, Flickr30K contains more samples with long captions (Figure 6), which may include overly-specific details that are inconsistent with other captions and are hard for the model to learn. See more examples in our supplemental materials. Through our curation-based finetuning, these samples can be effectively identified, removed or replaced, which indicates that

our method is efficient when training with noisy datasets. We note that curating more than 50% of the data does not benefit training and actually harms performance.

Static image replacement versus dynamic replacement In REPLACEIMG (Section 3.3), we 219 dynamically replace images for the difficult training samples. Another static approach is to replace 220 the identical images, i.e. I k in { ( I k , C 1 k ) , . . . , ( I k , C J k ) } , with unique SD-synthesized images before 221 training, instead of updating the training samples while training. With static image replacement, for 222 each of the reference captions, we replace their original image with a SD-synthesized image. Static 223 replacement with 20%-80% curation ratio corresponds to replacing images for one-four captions of 224

225

226

227

228

229

230

231

232

233

0

50

100

150

200

0

500

1000

1500

2000

2500

Count

Remove (2 std)

Epoch

0

1

2

3

4

0

50

100

150

200

ReplaceImg (40%)

Epoch

0

1

2

3

4

Loss value

2 std

20%

40%

50%

60%

80%

80

85

90

95

CIDEr

static

dynamic

no curation

0.6

0.7

0.8

0.9

1.0

CLIPScore

25

50

75

100

125

150

175

Loss Values

Avg num errors

0

1

2

3

4

5

(a) Distribution of text-to-image generation errors.

(b) Human evaluation versus CLIPScore.

Figure 9: Results of the human study of the errors made by the Stable Diffusion model in 100 images. The images used in the study were chosen to represent either low or high model loss. (a) Histogram of the number of errors annotated in each category. The most frequently occurring annotations concern weird deformations in the expected objects or humans. (b) Relationship between average number of identified errors by human annotations for each synthesized image and its captioning loss with regard to original captions. More errors are identified in images of higher loss. However, CLIPScore appears to fail in validating qualities of the synthesized images, as the score ranges are almost identical for samples that contain more errors.

the original five. The 50% replacement ratio mimics a fair coin-flip, where for each of the text-image samples, there is 50% probability for the image to be replaced by a synthesized image.

We compare the efficacy of these two approaches in Figure 7. When evaluating on the original 1k validation set, we see that for both approaches, incorporating synthesized images of 20% or 40% can assist finetuning and achieves higher BLEU4 and CIDEr scores. Nevertheless, dynamic image replacement consistently performs better than the static method, showing focusing on the hard samples is effective. For both replacement methods, performance starts to decrease when the curation ratio is too high. This may indicate that when incorporating too many images from the synthetic distribution, the gap increases between the training and evaluation sets.

Figure 8 shows the effect of the curation techniques in the training loss distributions across epochs. 234 For the REMOVE approach, training samples with loss that are two standard deviations worse than the 235 mean are dynamically removed during training, leading to the shrinking tail of the loss distribution. 236 SD-based image replacement gradually reduces losses through learning from a mixture of Gaussian 237 distribution from original image-text pairs and the ones contain synthesized images. 238

239

240

241

242

5.2 Human Study: Errors made by SD models

Finally, we conduct a human study of the errors present in the SD-synthesized images. This will serve to better understand any shortcomings with this approach that is not captured by automatic evaluation measures.

We first ranked SD-synthesized images by model loss from the 1K images in the validation set. This 243 validation set of synthesized images was generated using the best performing configuration of the 244 Stable Diffusion model (see Section 3.3). We then sampled a subset for human annotation using the 245 top and bottom 50 images based on their loss using our fine-tuned captioning model. These images 246 are uniformly divided into 5 sets, each containing 20 images with equal number of the high loss 247 ones and the low loss ones. The data was annotated by 12 people, members of a university research 248 lab with a basic understanding of Stable Diffusion but no knowledge of the bi-modal distribution 249 of images. The annotators were asked to categorize the errors they observed in the synthesized 250 images, given both the image and the reference sentences that were used to generate the images. Each 251 participant annotated one set of 20 images. 252

Starting from the categories defined by van Miltenburg and Elliott [50], we predefined 25 categories 253 including general errors such as color, or number mismatches, and errors related to people and 254 objects in the images. Please see the user interface in supplemental materials. We analyze the human 255 judgements for the images that have at least three annotations, yielding 74 unique images. 256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

As shown in Figure 9a, the most common problem of SD-synthesized images are that they often generate weird face or body parts, which makes the images less natural or pleasant. The Stable Diffusion model is also weak at generating the correct number of people or objects. From Figure 9b we confirm the quality of our collected annotations that high loss figures often contain more errors on average. Furthermore, we note that CLIPScore does not appear to align with human judgements, indicating its weak capability of evaluating quality of generated images. Please see more concrete examples in Figure 10.

6 Conclusion

In this paper, we have shown a simple, yet effective, data curation framework that can improve the performance of image captioning models. We investigated three approaches to data curation that dynamically update the training dataset based on high-loss image-caption samples. The methods involved either removing a sample, replacing the caption in a sample, or generating a new image from existing captions. Experimental results on the Flickr30K and MS COCO datasets show the effectiveness of these approaches to data curation without increasing the total size of the training dataset. A deeper analysis of the images synthesized by Stable Diffusion shows frequent errors on generating objects of a certain amount or color, and struggles with human body features. A human evaluation of the errors in those images shows a clear difference in images with high or low losses.

In the future, we expect that better text-to-image generation models will lead to further improvements from using synthesized images for difficult captions in existing training datasets. We plan on verifying whether these findings extend to other image captioning models, which was not possible here due to computational issues. Finally, we are interested in applying the same framework to other multimodal tasks, especially those with undercomplete datasets that cannot comprehensively cover the distributional space due to the cost of crowdsourcing enough data, e.g. visual question answering, or visually-grounded dialog.

Limitations

While our curation methods being effective on image-captioning in the finetuning and fewshotlearning settings, it is not clear if the same strategy can be scaled and adapted also to vision-language pretraining. Currently our data curation methods also rely on state-of-the art pretrained models for both image understanding and text-to-image generation. In pretraining, models will often be trained from scratch and pretraining data are often collected from multiple datasets and resources.

Moreover, while we take an online approach to data curation, our current approach is upper bounded in speed and performance of the text-to-image generation model. This might be a large bottle neck for adapting the strategy for more complicated vision-and-language tasks.

Ethics Statement

Text-to-image generation with Stable Diffusion is controversial in the broader AI and ethics community[6]. For example, it can generate images according to gender or racial stereotypes, which may prove harmful to members of those communities [30]. In this paper, we use Stable Diffusion to improve the quality of an image captioning model, given a specific set of crowdsourced captions. Those captions may themselves contain harmful stereotypes that would become more prevalent in our dynamically updated training datasets. As we dynamically update the model with new images based on loss values, we remove the water-marker in our generated images to prevent information leak to the model. Use of the synthesized images will strictly follow community guidelines.

References

Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In AAAI , pages 7383-7390. AAAI Press, 2020. 2

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic 304 propositional image caption evaluation. In European conference on computer vision , pages 305 382-398. Springer, 2016. 5 306

Viktar Atliha and Dmitrij Šešok. Text augmentation using bert for image captioning. Applied 307 Sciences , 2020. 2, 3 308

309

310

Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification, 2023. 2

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 311 Label-efficient semantic segmentation with diffusion models. In International Conference on 312 Learning Representations , 2022. URL https://openreview.net/forum?id=SlxSY2UZQT . 313 2 314

315

316

317

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 , 2023. 9

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for VQA are image captions. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz, editors, NAACL , pages 1947-1963. Association for Computational Linguistics, 2022. 2

Fuhai Chen, Rongrong Ji, Jinsong Su, Yongjian Wu, and Yunsheng Wu. Structcap: Structured semantic embedding for image captioning. In Qiong Liu, Rainer Lienhart, Haohong Wang, Sheng-Wei "Kuan-Ta" Chen, Susanne Boll, Yi-Ping Phoebe Chen, Gerald Friedland, Jia Li, and Shuicheng Yan, editors, MM , pages 46-54. ACM, 2017. 2

Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, and Jinsong Su. Groupcap: Groupbased image captioning with structured relevance and diversity constraints. In CVPR , pages 1345-1353. Computer Vision Foundation / IEEE Computer Society, 2018. 2

Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation , pages 376-380, 2014. 5

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 . 2

Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 968-988, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.84. URL https://aclanthology.org/2021.findings-acl.84 . 1

Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard H. Hovy. A survey of data augmentation approaches for NLP. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL , volume ACL/IJCNLP 2021 of Findings of ACL , pages 968-988. Association for Computational Linguistics, 2021. 2

Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. Vision-language 347 pre-training: Basics, recent advances, and future trends, 2022. 1 348

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and 349 Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR , pages 350 10686-10696. IEEE, 2022. 2 351

Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. Normalized and 352 geometry-aware self-attention network for image captioning. In CVPR , pages 10324-10333. 353 Computer Vision Foundation / IEEE, 2020. 2 354

355

356

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP , 2021. 4

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021.emnlp-main.595. URL https://aclanthology.org/2021.emnlp-main. 595 . 5

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS , 2017. 4

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03458 , 2022. 2

Sara Hooker. The hardware lottery, 2020. 1

Md. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, Hamid Laga, and Mohammed Bennamoun. Text to image synthesis for improved image captioning. IEEE Access , 9:6491864928, 2021. doi: 10.1109/ACCESS.2021.3075579. 2

Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17980-17989, 2022.

1

Lun Huang, Wenmin Wang, Jie Chen, and Xiaoyong Wei. Attention on attention for image captioning. In ICCV , pages 4633-4642. IEEE, 2019. 2

Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=99RpBVpLiX . 2

Peng Jiang, Fanglin Gu, Yunhai Wang, Changhe Tu, and Baoquan Chen. Difnet: Semantic segmentation by diffusion networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/ paper/2018/file/c2626d850c80ea07e7511bbae4c76f4b-Paper.pdf . 2

Guodun Li, Yuchen Zhai, Zehao Lin, and Yin Zhang. Similar scenes arouse similar emotions: Parallel data augmentation for stylized image captioning. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran, editors, MM , pages 5363-5372. ACM, 2021. 2

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022. 1, 5

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping languageimage pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, ICML , volume 162 of Proceedings of Machine Learning Research , pages 12888-12900. PMLR, 2022. 2

Minghui Li, Yan Wan, and Jinping Gao. What drives the ethical acceptance of deep synthesis applications? a fuzzy set qualitative comparative analysis. Computers in Human Behavior , 133: 107286, 2022. ISSN 0747-5632. doi: https://doi.org/10.1016/j.chb.2022.107286. 9

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza400 tion Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational 401 Linguistics. URL https://aclanthology.org/W04-1013 . 5 402

403

404

405

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR , abs/1405.0312, 2014. 1, 2, 4

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP , pages 9031-9041. Association for Computational Linguistics, 2020. 2

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022. 2

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors, ICML , volume 139, pages 8162-8171. PMLR, 2021. 2

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, ICML , volume 162, pages 16784-16804, 2022. 2

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002. 5

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, ICML , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 2021. 2

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. 2

Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410 . 4

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2022. 1, 2, 3

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In Munkhtsetseg Nandigjav, Niloy J. Mitra, and Aaron Hertzmann, editors, SIGGRAPH , pages 15:1-15:10. ACM, 2022. 2

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022. 2

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint , 2022. URL https://doi.org/10.48550/arXiv.2205.11487 . 4

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint , 2021. URL https: //doi.org/10.48550/arXiv.2111.02114 . 1

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1238. URL https://aclanthology.org/P18-1238 . 1

Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data , 6(1):1-48, 2019. 1

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2022. 2

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021. 2

Emiel van Miltenburg and Desmond Elliott. Room for improvement in automatic image description: an error analysis. CoRR , abs/1704.04198, 2017. 8

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015. 5

Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, 2022. 2

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. Learning , 2021. 1

Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, EMNLP-IJCNLP , pages 6381-6387. Association for Computational Linguistics, 2019. 2

Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. G-daug: Generative data augmentation for commonsense reasoning. In Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP , volume EMNLP 2020 of Findings of ACL , pages 1008-1025. Association for Computational Linguistics, 2020. 2

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL , 2:67-78, 2014. 1, 2

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR . OpenReview.net, 2018. 2

Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. In AAAI , pages 1304113049. AAAI Press, 2020. 2

Peer reviews:
comments: SUMMARY:
This paper focuses on data curation for image captioning. This paper shows that mismatched image-caption pairs do harm to the captioning model. To address this problem, generative models are used. In detail, the BLIP model is used to generate captions based on images, and the Stable Diffusion model is used to create images based on captions.

STRENGTHS:
1. The data curation is an important and effective topic, which could benefit many tasks including visual synthesis, image captioning, language and visual representation, etc.
2. This paper discovers the weak point of captioning datasets, especially for the Flicker30K.
3. It is interesting to use the BLIP model and the Stable Diffusion model to create data for training.

WEAKNESSES:
1. There are many methods to augment text dates, e.g., adding or editing some words, using synonyms, and changing sentence structure. I think these methods are also worth evaluating.
2. From Table 2, we can see that the BLIP's performance is not significantly affected by the methods proposed in this paper (i.e., Remove, ReplaceCap, and ReplaceImg). For example, the CIDEr of COCO only slightly raises from 132.0 to 133.1.
3. Figure 5 shows that the proposed methods might make the performance worse, especially in the COCO dataset. So I am concerned about the generalization of the proposed methods.

QUESTIONS:
Both the BLIP model and the Stable Diffusion model use a large size of datasets (e.g., LAION-5B). Would the performance of Image Captioning be improved by using part of the data in LAION-5B?

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies data curation strategies for training image captioning models. Firstly, it identifies the “difficult samples” based on the captioning loss dynamically at the end of each epoch. Subsequently, it introduces three data curation strategies to modify the difficult samples: (1) removal of an image-text pair, (2) replacement of the caption and (3) replacement of the image using text-to-image generative models. The main technical innovation is the third strategy, which is carefully designed in terms of prompt engineering and fine-tuning on the image captioning datasets.\n\nThe empirical studies show that the proposed data curation strategies can enhance the performance of the baseline BLIP captioning model. The authors also conduct analysis on the data curation ratio, dynamic versus static curation strategy and the errors of images generated by the stable diffusion model.\n\nSTRENGTHS:\n* The idea of employing text-to-image generative models to curate training data for image captioning is novel and well-motivated.\n\nWEAKNESSES:\n### Effectiveness of the proposal\n* According to Table 2, the performance of the third data curation strategy, which is the main technical innovation of this work, is not advantageous compared to the heuristic removal and caption replacement strategies.\n* According to Figure 5, all three proposed strategies are sensitive to data curation ratio. Consequently, training the captioning model multiple times is necessary to achieve satisfactory performance, which is less efficient compared to the baseline BLIP model.\n\n### Design of the method\n* Identifying the samples to modify based on training loss is questionable. A higher loss does not necessarily imply that the sample is harmful to training. Although Section 5.2 has shown that more errors are identified in images of higher loss, the experimental setup has two issues: (1) The loss is computed over the generated images rather than the real images in the original dataset. (2) The errors are categorized as targeting image generation, rather than image captioning. In other words, an image that possesses imperfect visual quality but aligns well with the caption may not necessarily be considered a noisy training sample for image captioning.\n\n### Missing reference\n* An idea similar to the “round-trip captioning evaluation” is already proposed by [1], which generates a caption from the synthesized image and measures the similarity between input text and predicted caption.\n\n### Clarity\n* In line 243, it is unclear whether the “model loss” refers to captioning loss or image generation loss.\n\n[1] Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe authors have discussed the limitations of this work from three aspects: (1) Lack of adaptation of the proposal to the pre-training stage. (2) Reliance on pre-trained image understanding and text-to-video generative models. (3) Increase in training time due to the usage of text-to-image generative model. Moreover, a significant limitation of this study is the absence of evidence demonstrating the superior effectiveness of the generated images compared to heuristic data curation strategies.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a data curation model for image captioning. If the loss of a particular image caption pair is high, then either remove the image-caption pair from the training set or replace the caption with a more similar caption or they generate a new image for the difficult caption. The authors demonstrate these strategies help to improve the performance of BLIP caption generation model.

STRENGTHS:
The idea is interesting.

Paper shows some positive gains on COCO and FLickr30K.

WEAKNESSES:
The details of the method are not clear. How to select a replacement caption?

Why one should pick only the high-loss image-caption pairs? Loss may be high due to many other reasons.

Compared to other data augmentation methods in the literature that is also discussed in the related work, what is the novelty?

Why this is a significant finding? I am not sure if this is a significant finding. 

The method is also evaluated using a single model. 

Obtained results are not state-of-the-art.

It is not clear whether such a mechanism will contribute to any state-of-the-art methods in captioning.

QUESTIONS:
What are the conceptual differences w.r.t. [3]? W.r.t [3] is this paper novel?

I am not able to understand Figure 2 and the message behind this figure.

No guarantee synthesized image has a smaller loss than the original image.

It is not clear to me what is F in Table 1.

LIMITATIONS:
Limitations and societal impact are discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-06

comments: SUMMARY:
This paper focuses on improving image captioning by improving the quality of the existing dataset. To this end, this paper proposes three data curation methods: the removal of an image–caption sample; replacing a caption with another caption; and replacing images using a text-to-image generation model. Experimental results demonstrate that models trained with the proposed methods consistently outperform baselines.

STRENGTHS:
1.The proposed method is well-motivated, that is to improve the quality of the existing dataset. This paper explores the problem of making better use of existing datasets, which is a very interesting research direction.

2.The authors conduct extensive experiments over these two datasets, where the models trained with the proposed methods outperform baseline methods consistently.

3.The paper is well-written and easy to follow.

WEAKNESSES:
1.From my understanding, it is risky to judge the quality of the sample based on the loss value. A sample with a large loss value may be a hard sample or a mislabeled sample, so it is risky to judge the sample quality only based on the loss value.

2.In addition to the performance of the model on the test set, the generalization ability of the model is also important. It is not clear whether the proposed method reduces the gap between the training set and the test set or improves the quality of the training set.

3.Lack of necessary theoretical analysis.

QUESTIONS:
Can the proposed method improve the generalization ability of the model? It would be better to carry out cross-domain testing to verify the generalization ability of the model, that is, the Flickr30K dataset is used as the training set, and the test set of the COCO dataset is used as the test set.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an iterative training approach to improve image captioning models.\nThis approach _refreshes_ the training dataset every epoch with _higher quality_ image-text pairs (authors call it \"data curation\").\nDataset samples with very high training loss are updated -- the real image is replaced with one generated by the Stable Diffusion model.\nThe authors compare their approach with two baselines: one which removes high-loss samples, and one where the image is replaced by another from the training dataset itself.\nExperiments are performed with the BLIP model and two captioning datasets -- COCO and Flickr30K.\nAuthors also perform an accompanying human study to provide directions for future work.\n\nSTRENGTHS:\nThis paper has numerous technical strengths:\n\n- The proposed method is conceptually simple and easy to implement.\n- The strategy of updating the training dataset with \"better\" samples is very general: it is agnostic to the model architecture and the multi-modal task at hand.\n- The writing and presentation quality of the paper is excellent. It contains adequate implementation details to make this work reproducible.\n- The experimental setup and ablation study is very meticulous. Tables of results contain experiments that begin with a BLIP baseline, and subsequent rows introduce one change at a time.\n- The authors have conducted a human study with sensibly defined failure categories to understand how failure modes of Stable Diffusion can impact captioning performance.\n\nWEAKNESSES:\nLike its technical strengths, this paper also has some shortcomings.\nBelow I list a few salient concerns with the paper.\nI look forward to hearing the authors' response, and I am happy to update my final assessment.\n\n1. **Results do not match with the presented story:**\nThe main results (`Table 2`) indicate that all considered dataset curation approaches are beneficial over a BLIP baseline that doesn't train on curated data.\nHowever, the main pitch of this paper is to use generative models like Stable Diffusion to replace images (last row),\nwhich in fact performs marginally better or even worse than other curation techniques.\nThe biggest improvements are generally yielded by \"Remove\" strategy.\nI recommend the authors rethink the positioning of motivation and frame it as an exploratory study --\nit seems obvious to use generative models for iterative training/distillation and some works already do it for other applications,\nbut for this task, a practitioner is better off by simply filtering noisy samples altogether.\n\n2. **Captioning metrics appear saturated, maybe overkill for COCO/Flickr:**\nThe captioning metrics on COCO and Flickr are already saturated,\ne.g. decimal improvements are less meaningful for COCO in the range of 130+ CIDEr and 20+ SPICE score.\nSince BLIP is already rained with large amounts of data and diverse tasks,\nthe proposed approach may be an overkill for the tasks considered in this paper.\nI suggest the authors rethink other applications where the benefits of this strategy are more prominently observed (see Weakness 5 below).\n\n3. **What if the caption is noisy and can't generate meaningful images?**\nAn image-text pair may be unaligned if the caption is uninformative,\nas frequently encountered in larger web datasets like\n[Conceptual Captions](https://arxiv.org/abs/2102.08981),\n[YFCC](https://arxiv.org/abs/1503.01817),\n[RedCaps](https://arxiv.org/abs/2111.11431), etc.\nFor instance, captions coming from alt-text may not have any semantic content whatsoever\n(e.g. see Figure 2 in [ALIGN paper](https://arxiv.org/abs/2102.05918)) to generate meaningful images.\nThe proposed approach forces the generative model to create an arbitrary image and ends up adding noise to the training data.\nSome selective mechanisms to replace either image or caption may be needed to scale this approach to general image captioning beyond COCO and Flickr30K.\n\n4. **Related work needs more coverage:**\nThe main focus of this paper is image captioning, hence a broad coverage of prior works on image captioning is necessary.\nHowever, this section only cites a handful of very recent modeling papers.\nI suggest the authors begin the discussion with some early image captioning papers like:\n\n  - (Vinyals et al, CVPR 2015) Show and tell: A neural image caption generator\n  - (Karpathy and Li, CVPR 2015) Deep visual-semantic alignments for generating image descriptions\n  - (Donahue et al, CVPR 2015) Long-term recurrent convolutional networks for visual recognition and description\n\n5. **[Related to 1, 2] Have the authors considered applications others than image captioning?**\nWhat if this curation strategy is used to train general visual representations?\nI suggest a CLIP-style contrastive model and/or BLIP/VirTex-style generative model.\nThe contribution can be strengthened by broadening the scope to various downstream tasks.\n\nQUESTIONS:\nSome comments and suggestions:\n\n- Imagen is cited twice (44 and 45). Please remove the duplicate.\n- Related work section has phrases like \"large-scale stable diffusion models\" (`Line 64`) and \"stable diffusion text-to-image models\" (`Line 69`).\n  \"Stable Diffusion\" is a set of models developed by Stability AI startup, and these phrases seem to appropriate a brand as a mathematical/conceptual term.\n  I suggest the authors remove \"stable\" from these phrases and call them \"text-to-image diffusion/generative models\" or something.\n\nLIMITATIONS:\nThe limitations section should be updated if any of the above-mentioned open questions are not within the scope of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}