Title: Attribute Based Interpretable Evaluation Metrics for Generative Models
Venue: NeurIPS 2023
Accepted: no
Keywords: evaluation metric for Generative models
Abstract: While generative models continue to evolve, the field of evaluation metrics has largely remained stagnant. Despite the annual publication of metric papers, the majority of these metrics share a common characteristic: they measure distributional distance using pre-trained embeddings without considering the interpretability of the underlying information. This limits their usefulness and makes it difficult to gain a comprehensive understanding of the data.
To address this issue, we propose using a new type of interpretable embedding. We demonstrate how we can transform deeply encoded embeddings into interpretable embeddings by measuring their correspondence with text attributes. With this new type of embedding, we introduce two novel metrics that measure and explain the diversity of the generator: the first metric compares the frequency of appearance of the training set and the attribute, and the second metric evaluates whether the relationships between attributes in the training set are preserved. By introducing these new metrics, we hope to enhance the interpretability and usefulness of evaluation metrics in the field of generative models.

Abstract

While generative models continue to evolve, the field of evaluation metrics has largely remained stagnant. Despite the annual publication of metric papers, the majority of these metrics share a common characteristic: they measure distributional distance using pre-trained embeddings without considering the interpretability of the underlying information. This limits their usefulness and makes it difficult to gain a comprehensive understanding of the data. To address this issue, we propose using a new type of interpretable embedding. We demonstrate how we can transform deeply encoded embeddings into interpretable embeddings by measuring their correspondence with text attributes. With this new type of embedding, we introduce two novel metrics that measure and explain the diversity of the generator: the first metric compares the frequency of appearance of the training set and the attribute, and the second metric evaluates whether the relationships between attributes in the training set are preserved. By introducing these new metrics, we hope to enhance the interpretability and usefulness of evaluation metrics in the field of generative models.

1 Introduction

Significant advancements have been achieved in the image generation field, from the pioneering introduction of generative adversarial networks (GANs) to the more recent emergence of diffusion models (DMs). [5, 10, 27] In recent years, generated images are hardly distinguishable from real images. In this context, evaluating the generated images for a given training dataset has played a critical role in the development.

Envision an evaluation scenario where the outputs of two generative models are compared against 22 a common training dataset. What would be the underlying factors for judging a set as superior to 23 another set? As the goal of generative models is mimicking the real data distribution, various metrics 24 have been designed to assess the similarity between the generated images and the training dataset, e.g., 25 Fréchet Inception Distance (FID)[9], Precision and Recall[25][17], and Density and Coverage[22]. 26

Most of these evaluation metrics capture the disparity between the training data distribution and the 27 distribution of generated images by examining the differences in feature representations within the 28 embedding space of a pre-trained network[26, 28]. FID is a widely used metric that quantifies the 29 dissimilarity in visual features to assess the quality and diversity of the generated images. Specifically, 30 it measures the distance between the real and fake distributions in the embedding space of Inception31 V3[28]. 32

An important question arises regarding the suitability of the embedding space employed for evaluating 33

generated images. The embedding space of the pre-trained model may vary depending on the dataset 34

and task it was trained on. For instance, Inception V3 was trained for image classification on 35

Training dataset

Model 1

Model 2

(b) proposed metric

(a) existing metrics

Model 1

Model 2

FID↓

P/R/D/C↑

“makeup"

"inakeup"

1.0

“long hair"

"young"

long hair"

"young"

0.75

Model 1

Model/2

0.50

2

0.25

"smiling'

"mustache”“smilinig"

"mustache"

FID Precision Recall Density Coverage

0

"black hair"

"black hair"

ImageNet[3], suggesting that its embedding space is designed to compress image information and 36 discern essential patterns for classification. Consequently, the appropriateness of employing this 37 embedding space for evaluating generated images remains an open question. 38

Returning to the fundamental question at hand, Figure 1 makes an evaluation scenario a little bit more 39 specific. Suppose there are realistic generated images from two distinct models. As shown in the 40 example images, it is evident that Model 2 generates biased images, i.e., there are only women, while 41 Model 1 successfully generates various images that are close to training data. Fortunately, although 42 there remains an open question about embedding space, the values of various metrics in Figure 1 (b) 43 align reasonably well with our interpretation; Model 1 is perceived as superior. 44

However, what are the underlying factors that contribute to such judgment? Although the results 45 are consistent with a person's conclusion, it far fails to provide a comprehensive explanation. The 46 interpretation of distances within the embedding space from a pre-trained classification model remains 47 elusive, posing challenges in evaluation. On the contrary, humans readily discern certain factors for 48 judgment; individuals easily recognize the bias of Model 2. These factors suggest more information 49 and a direction beyond simple ranking. In this paper, we propose an evaluation metric that aims to 50 interpret the underlying factors behind such judgments. 51

52

53

54

55

56

57

58

59

To address this objective, we begin by examining attribute comparison methods in human judgment. When evaluating two generated image distributions, humans compare the attributes present in the training dataset with those exhibited by the generated images. Key attributes under consideration include gender, facial representation, and age distribution. Ideally, with well-defined training data, we anticipate the attributes in the generated images to align with those in the training data. If the model lacks essential attributes (e.g., gender, age, glasses, or hats), it is insufficient to generate visually realistic images. Incorporating these attributes into the evaluation process may enable a more explicit and comprehensive assessment.

60

61

62

63

64

65

66

67

68

This paper presents a novel approach for evaluating generative models by leveraging a newly proposed embedding space that incorporates attribute-specific information. Similar to human visual judgment, our metrics evaluate images in terms of various characteristic attributes. Figure 1 (b) illustrates the concept of our metric; it captures the distribution differences of attributes. We use pre-trained CLIP [24], a language-image model trained on a huge dataset, to define a new embedding space that can quantify images for multiple attributes.

To facilitate our embedding space, we introduce the "Directional CLIPScore" (

DCS

), a method for quantifying each attribute based on the training data. Within our proposed embedding space, each

channel comprises DCS values that explicitly indicate the relevance of an image to specific attributes.

We introduce two novel evaluation metrics to use the newly proposed embedding space. Firstly, 70 the "Single attribute KL Divergence ( SaKLD )" compares attribute distributions between training 71 data and the generated images, providing a quantitative measure of the similarity between attribute 72 distributions. It quantifies how closely the attributes of generated images align with the attribute 73 distribution in training data. Secondly, we introduce the "Paired attribute KL divergence ( PaKLD )" 74 that considers correlations among multiple attributes. This metric accounts for the relationship 75 between attributes, such as the presence of a beard in an image of a woman. PaKLD successfully 76 evaluates the generated images while taking into consideration attribute relationships. 77

78

79

80

81

82

83

We validate our metrics through a series of carefully designed experiments, demonstrating their effectiveness and interpretability. By employing our metric, we conduct a comprehensive analysis of prominent generation models currently considered state-of-the-art [11, 13, 12, 14, 23]. Interestingly, our findings reveal variations in performance across different datasets. For instance, diffusion models exhibit superior performance on datasets with a large number of samples, such as FFHQ. In contrast, GANs outperform diffusion models on datasets with relatively smaller sample size, such as MetFaces.

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

In summary, this paper presents a novel approach for evaluating generative models using a new embedding space that incorporates attribute-specific information. Our proposed method, along with the introduced evaluation metrics, allows for a comprehensive assessment of generated images by considering attribute distributions and correlations. Our findings contribute to the research field by advancing the understanding and evaluation of generative models, offering insights into their strengths and limitations. Moreover, our work opens avenues for future research and potential improvements in the field of generative image synthesis by comprehensive evaluation metrics.

2 Related Work

Fréchet Inception Distance Fréchet Inception Distance (FID) [9] measures the distance between the estimated Gaussian distributions of two datasets by passing them through a pre-trained Inceptionv3[28] model. However, Kynkäänniemi et al. [18] revealed that when generated images are far from training data, the embeddings may incorrectly highlight irrelevant parts of images. To address this issue, the researchers proposed using the CLIP [24] image encoder instead of Inception-v3 to calculate the 2-Wasserstein distance, which provides reliable results regardless of the dataset being measured.

Fidelity and diversity Sajjadi et al. [25] introduced precision and recall for evaluating generative model, and subsequent studies by Kynkäänniemi et al. [17] and Naeem et al. [22] have further refined this approach. Most of these methods use a pre-trained network to examine whether the embedding of generated images falls within the boundary of real image embedding (precision) and whether the embedding of real images falls within the boundary of generated image embedding (recall) for assessing fidelity and diversity.

Rarity score Han et al. [6] proposed a metric for measuring the rarity of generated images. They quantified how rare the generated images are within a k-NN sphere to assess their rarity. The key difference between the rarity score and diversity in precision and recall is that the rarity score considers only the generated samples that fall within the manifold of real samples. In other words, it focuses on how well the generated images fit within the distribution of real images in terms of rarity, rather than capturing the overall diversity of generated samples.

However, we note that the concept of using raw embeddings from a pre-trained classifier remains consistent among all these metrics.

Acall for explainable evaluation Existing evaluation metrics in the field of generative models lack the ability to provide detailed insights into the diversity of generated images. As shown in Figure 1, even though metrics like FID, Precision and Recall indicate poor performance for a biased generator towards specific attributes (e.g., "makeup" and "long hair"), they do not provide an explanation for judgment factors. Therefore, researchers manually identified the underlying factors by visual inspection but it becomes increasingly challenging with larger sample sizes. To address this issue, we propose novel explainable evaluation metrics that provide in-depth analysis and insights into the diverse generation abilities of models.

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

CS

(a) CLIPScore

(b) Directional CLIPScore

DCS

25

15

20

10

5

0

5

-5

0

-10

Arched

Wong hair

red lip

makeup

mustache

Arched

longhair

red lip

makeup

mustache

eyebrows

eyebrows

3 Attribute-Driven Embedding

Existing metrics for evaluating generated images commonly utilize embeddings before FCN, from Inception-V3 or CLIP image encoder[7, 4]. However, these approaches lack interpretability as the meaning of each channel in the embedding. Additionally, Kynkäänniemi et al. [18] have shown the FID scores improve significantly when the classification distribution matches that of the training set, irrespective of the quality, highlighting another limitation of the existing embedding. To address these issues and develop an explainable evaluation metric, we design each embedding of images to possess an 'interpretation'. Section 3.1 presents the process of generating explainable embedding for individual images using the CLIP encoder, and Section 3.2 introduces the Directional CLIPScore, a novel embedding approach that enhances interpretability and accuracy.

3.1 Attribute-driven embeddings for better representations

To achieve an interpretable embedding, we utilized each channel of the embedding as a measure of the attribute's prominence in the image. A straightforward approach to quantify attribute strength is by employing CLIPScore;

where x is a single image, a is a given text of attribute, sim ( ∗ , ∗ ) is cosine similarity, and E I and E T are CLIP image encoder and text encoder respectively. We selected multiple attributes that effectively represent image characteristics as textual descriptions and measured CLIPScore with individual images and selected attributes. The way to select attributes will refer to Section 3.3. By assigning these CLIPScores as the values for each channel in the embedding, we obtained an interpretable representation. However, relying solely on CLIPScore has challenges as the cosine similarity values tend to be similar, making it difficult to discern the relative differences between attribute strengths. Intuitively, selected human-related attributes tend to cluster closely in the CLIP embedding, resulting in smaller variations in cosine similarity. To address this limitation, subsequent subsections introduce the Directional CLIPScore, which offers a more precise scoring approach.

3.2 Directional CLIPScore

As discussed, CLIPScore exhibits a narrow distribution of values, which can be attributed to measuring similarity between human-related attributes, resulting in their dense clustering on the CLIP embedding. Figure 3 (a) visualizes it. To address this issue, we propose Directional CLIPScore (DCS), which leverages the centers of training images and predefined attribute texts on the CLIP embedding.

Given training data, denoted as { x 1 , x 2 , x 3 , ... } ∈ X , we define C X as the center of images and C T as another center of images for text attributes on the CLIP embedding, respectively. By using the image captioning model, BLIP[19], we define C T as the center of images in text respect;

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

(a) CLIPScore

(b) Directional CLIPScore

ܵܥ

̶̶

ൌ ͳͲͲ

כ



ߠ

ଵ

ൌ ʹͻǤ͵

ܵܥ

̶̶

ൌ ͳͲͲ

כ



ߠ

ଶ

ൌ ʹͳǤͳ

ܵܥܦ

̶̶

ൌ ͳͲͲ

כ



ߠ

ଵ

ܵܥܦ

̶̶

ൌ ͳͲͲ

כ



ߠ

ଵ

𝐷𝐶𝑆 "makeup"

= 100 ∗ cos 𝜃

!

= 8.2

𝐷𝐶𝑆 "mustache"

= 100 ∗ cos 𝜃

"

= -7.6

𝐶𝑆 "makeup"

= 100 ∗ cos 𝜃

!

= 29.3

𝐶𝑆 "mustache"

= 100 ∗ cos 𝜃

"

= 21.1

(b) Directional CLIPScore

(a) CLIPScore

t1 =“makeup’

t =“makeup”

Cx

02

t2=“mustache"

t2=“mustache"

ൌ ͺǤʹ

ൌ െ͹Ǥ͸

These centers serve as reference points in the embedding space and aid more accurate attribute 152 scores. We define DCS as the measure of similarity between two directions, V x and V a where a set 153 of attributes defined as { a 1 , a 2 , a 3 , ... } ∈ A . The first direction spans from the center of the image 154 to the image itself, and the second direction extends from the center of the attributes to the desired 155 attribute. 156

where sim ( ∗ , ∗ ) is cosine similarity. For extending DCS from a single sample to data we denote the probability density function (PDF) of DCS ( x i , a i ) for all x i ∈ X as DCS X ( a i ) for brevity.

Figure 3 visually illustrates the distinction between DCS (Directional CLIPScore) and CS (CLIPScore). Unlike CS, which lacks a clear reference point, DCS is based on the center, enabling the determination of attribute magnitudes relative to a zero point. Furthermore, DCS exhibits superior accuracy compared to CS, as demonstrated in Table 1. The table presents the accuracy results of CS and DCS for annotated attributes in CelebA[20]. By evaluating how well positive samples with the highest score align with positive samples for a given attribute, DCS consistently outperforms CS in accuracy. Notably, this trend remains consistent across refined attributes, which are removed for subjective attributes such as "Attractive" or "Blurry".

3.3 attribute selection methodologies

Our evaluation metric for measuring the performance of the generator is dependent on the attributes we choose to measure. To explore how to choose attributes that accurately reflect generator performance, we introduce three methods for attribute selection.

BLIP extracted attribute We aim to identify and quantify the attributes present in the training data from image descriptions. We can determine which attributes are most commonly occurring in the training data by counting attributes that appear in the training data. We use the image captioning model, BLIP[19], to extract attribute-related words from training data. We use N attributes that appear frequently in the training data as a set of attributes A for our proposed metric.

User annotation Another option for attribute selection is to use a set of human-annotated attributes. 177 By explicitly assigning attributes for evaluating generative models, users can fairly compare the 178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

impact of each attribute score or focus on specific attributes. Especially, the CelebA dataset provides 40 binary attributes about the human face domain, which can be used to evaluate a wide range of (generated) human image sets.

GPT attributes We leveraged the power of GPT-3[1] to extract attributes. Through repetitive questioning, such as 'Give me 50 words of useful visual attributes for distinguishing faces in a photo' and 'Give me 50 words of useful visual attributes for discerning variations in facial features to identify people in images,' we obtained a set of attributes, which frequently appeared in the responses across different datasets. The list of questions posed to GPT-3 can be found in the Appendix, and we followed the questioning methodology outlined in [21].

4 Evaluation Metric with Interpretable Attribute-Driven Embedding

In this section, by leveraging the knowledge of attribute intensities, we have developed two understandable metrics. In Section 4.1, we present Single attribute KL Divergence (SaKLD), which measures the distance of attribute distributions between training data and generated images. In Section 4.2, we introduce Paired attribute KL divergence (PaKLD), a metric that assesses the relationship of attributes.

4.1 Single attribute KL Divergence (SaKLD)

We design SaKLD to distinguish a good generative model which produces the same quantity of each attribute present in the training data. For example, if 50,000 training data contains 3,000 images with eyeglasses, the model should generate exactly 3,000 images with eyeglasses. Any deviation from this ideal distribution is considered undesirable. We introduce a new metric that quantifies density of each attribute in dataset by utilizing interpretable embedding. Our metric, SaKLD, quantifies the difference in density for each attribute between the training dataset ( X ) and the set of generated images ( Y ).

We define SaKLD as

where i denotes an index for each attribute, N is the number of attributes, KL(*) is Kullback-Leibler Divergence, and note that we denote the PDF of DCS ( x i , a i ) for all x i ∈ X as DCS X ( a i ) .

We compare the PDFs of Directional CLIPScore for each attribute in X and Y . The DCS PDF for each attribute in X and Y represent the distribution of the amount of that attribute in the respective sets. If the distribution of the amount of a specific attribute in X and Y is similar, the DCS distribution will also be similar, and the PDFs of the two sets will be close. We used Kullback-Leibler Divergence(KLD) to compare the each Directional CLIPScore PDFs for their attribute in X and Y, to quantify the extent to which the generator has created too few or too many instances of a specific attribute. We then calculate the average KLD value between the PDFs of each attribute in X and Y to obtain the final value of SaKLD.

4.2 Paired attribute KL Divergence (PaKLD)

We design another metric, PaKLD for examining that generated images preserve the attribute relationships present in training data. The model should generate images that adhere to the attribute relationships observed in the training data. For instance, if all 50,000 male images in the training data wear glasses, then all generated male images should also wear glasses. To evaluate the preservation of attribute relationships, we compare the difference in the joint probability density distribution of attribute pairs between training data. Our proposed metric, Pairwise Attribute KL Divergence (PaKLD), is defined with joint probability density functions as follows:

where M = nP 2 , ( i, j ) denotes an index pair of attributes, and the pair of attributes' joint PDF is denoted as DCS X ( a i,j ) .

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

PaKLD analyzes the performance of the model more comprehensively. For example, if the generator's probability density function for the attribute pair ("makeup", "long hair") significantly differs from that of the training data, we can infer that the generator does not preserve the ("makeup", "long hair") relationship. PaKLD allows to quantify the degree of preservation of attribute relationships and measure quantitative entanglements between attributes that have not been considered in previous researches.

5 Experiments

Experimental details To estimate the probability density function (PDF) of Directional CLIPScore (DCS) in the training data and generated images, we use Gaussian kernel density estimation. We sample 10,000 points from each PDF to obtain a discretized distribution and use it to calculate SaKLD and PaKLD. In all experiments, we use a set of N = 20 attributes.

5.1 Correlated Image Injection Experiment: Validating the Effectiveness of Our Metric

In this subsection, we provide a carefully designed experiment to compare the proposed metrics with FID; we first create two non-overlapping subsets of 30,000 images from FFHQ and consider them as training data X and generated images Y , respectively. We then compare the scores for all metrics after including the edited images in set Y . Specifically, we use DiffuseIT[16] to prepare two sets of edited images: 'man' with 'makeup' and 'man' with 'bangs'. We use CelebA attributes for user annotation method (denoted by USER in Table 2).

As shown in Table 2, our metrics and FID show consistent tendency: score increases when more edited images are included in imageset Y . Furthermore, thanks to the nature of focusing on the attributes of the image domain, our metrics show more obvious numerical differences compared to FID. These results demonstrate that SaKLD successfully captures the attribute distribution difference and PaKLD captures the joint distribution difference between attribute pairs. Basically, our three attribute selection scenarios have similar tendencies across the two proposed metrics, but there are several differences. See supplement material for more details.

5.2 Necessity of PaKLD

Weconducted another toy experiment, a scenario in which the SaKLD metric fails to detect a particular attribute relationship, while PaKLD metric successfully identified it. We define the curated subsets of CelebA-HQ as training data and generated images with discrepancies in attribute relationship. Specifically, for training data, we collect 20,000 'smiling men' images and 20,000 'non-smiling women' images using ground truth labels of CelebA-HQ. Conversely, the generated images consist of 20,000 'non-smiling men' and 20,000 'non-smiling women'. In this scenario, the PDFs of the 'man', 'woman', and 'smile' attributes would not differ significantly between the two sets, and thus the SaKLD score would not capture it well. However, Paired attribute KL divergence would exhibit significant differences because the relationships between attributes within each set are completely different.

Figure 4 clearly illustrates the disparities in the evaluation results. While SaKLD score remained relatively unchanged for noteworthy attributes such as 'man', 'woman', and 'smile', the Paired

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

…

(a) SaKLD

(b) PaKLD

PaKLD

SaKLD

0.00018

0.0012

Otherattributes

smiling&woman

smiling

0.00016

smiling & man

0.0010

makeup

woman

0.00014

other attributes

smiling

man

0.00012

wavyhair

0.00010

woman

0.00008

0.00006

(woman，smiling

（man，smiling

man

(smiling,beard

0.00004

smiling,makeup

0.0002

grayhair

0.00002

0.0000

0.00000

makeup

smiling

wavyhair

woman

man

redlip

beard

attribute KL divergence score showed significant variations. This can be attributed to the distinct probability density functions (PDFs) of the 'woman ∩ smiling'. Note that we can easily understand the judgment factors; top attributes such as 'woman ∩ smiling' and 'man ∩ smiling' increase the score. These findings demonstrate the superior sensitivity and discernment of our proposed metrics, allowing for a more comprehensive evaluation of the generator's generation ability.

5.3 Comparing generative models including GANs and diffusion models with our methods

Leveraging the superior sensitivity and discernment of our proposed metrics, we compare the performance of GANs and Diffusion Models (DMs) in Tables 3. Interestingly, there are two attractions; 1) StyleGAN2-ADA shows the worst performance and 2) despite the respectable generative capability of DMs, iDDPM showed worse performance than StyleGAN models in all datasets.

The score of StyleGAN2-ADA implies that data augmentation for generative models may ruin attribute distribution in spite of FID's superiority. Please refer to Appendix for an analysis. And we suppose that although there are many advantages of DMs, it is inferior to GANs in attribute-based analysis.

To investigate the reason for the inferiority of DMs, we leverage the flexibility of constructing attributes to analyze the score changes according to the characteristics of attributes. We constructed attributes that focus only on color (e.g., 'yellow fur', 'black fur') and attributes that focus on shape (e.g., 'pointy ears', 'long tail') for LSUN Cat.

Table 4 shows that iDDPM's performance was particularly poor for color attributes. This is consistent with the assumption by Khrulkov et al. [15] that the encoder map of DMs coincides with the optimal transport map for common distributions; which means the pixel-based Euclidean distance corresponds to high-level texture and color-level similarity regardless of dataset and model. Therefore, the color

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

color attributes

shape attrbutes

SaKLD

↓

PaKLD

↓

SaKLD

↓

PaKLD

↓

StyleGAN1[11]

36.614

75.884

33.214

72.454

StyleGAN2[13]

36.621

67.518

34.642

68.954

iDDPM [23]

111.302

121.877

72.181

80.511

45

35

25

15

20            30            40

(b) number of attributes

PaKLD

20            30            40

SaKLD

(a) number of samples

10k  20k  30k  40k  50k

SaKLD

PaKLD

10k  20k  30k  40k  50k

6.0

6.5

7.5

7.0

8.0

12.0

13.5

16.5

15.0

18.0

8

12

20

16

24

16

24

40

32

48

SaKLD_stylegan1

PaKLD_stylegan1

SaKLD_seed2

PaKLD_seed2

SaKLD_stylegan2

PaKLD_stylegan2

SaKLD_seed3

PaKLD_seed3

PaKLD_stylegan2ada

SaKLD_seed4

SaKLD_seed5

PaKLD_seed4

PaKLD_seed5

SaKLD_stylegan3

PaKLD_stylegan3

of the output images only depends on the initial latent noise x T , and the Monge optimal transport map between training data and the standard normal distribution. We conclude that the distribution of color-related attributes is the inferiority of DMs.

5.4 Impact of Sample Size and Attribute Count on Proposed Metric

We provide ablation experiments to investigate the effect of a number of samples and attributes in Figure 5. We obtain generated images by StyleGAN3 from FFHQ with various random seeds. When the number of samples increases, SaKLD and PaKLD converge, especially more than 50,000 samples (Figure 5 (a)). We argue that the scores started to stabilize when using more than 50,000 images and note that we use 50,000 images for Tables 3 and 4. As for the number of attributes, we observe that the rank of the models mostly remained consistent regardless of the number of attributes. However, scores of DMs, purple line of Figure 5 (b), is increased as the number of attributes is increased because of color-related attributes. We argue that 20 attributes are sufficient, but more information can be obtained by using more diverse cases. Please see Appendix for an analysis of each score.

6 Discussion and Conclusion

In this paper, we introduce a novel metric that not only assesses the performance of the generator but also provides explicit explanations. Our proposed method, Directional CLIPScore, quantifies the attributes captured in an image and aligns them close to human judgment. Leveraging the interpretability of DCS, we propose two novel metrics, namely the SaKLD and PaKLD, which allow us to compare attribute appearance frequencies and examine attribute relationships, respectively.

While our metrics offer comprehensive explanations, unreliable results may arise when the attributes present in the images are ambiguous. For instance, in complex modern artworks with intricate color patterns, extracting appropriate attributes becomes challenging or even impossible, rendering our metric ineffective. Additionally, if the generative model's ability is significantly poor, the same limitation arises: measuring DCS from generated images becomes challenging.

Despite these limitations, our research establishes a solid foundation for the development of explain306 able evaluation metrics for generative models and contributes to the advancement of the field. 307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11472-11481, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung-Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness of synthesized images. arXiv preprint arXiv:2206.08549 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems , 33:12104-12114, 2020.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems , 34:852-863, 2021.

Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets. Understanding ddpm latent codes through optimal transport. arXiv preprint arXiv:2202.07477 , 2022.

Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. arXiv preprint arXiv:2209.15264 , 2022.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019.

Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role 357 of imagenet classes in frechet inception distance. arXiv preprint arXiv:2203.06026 , 2022. 358

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre359 training for unified vision-language understanding and generation. In International Conference 360 on Machine Learning , pages 12888-12900. PMLR, 2022. 361

362

363

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 , 2022.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning , pages 7176-7185. PMLR, 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems , 31, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes an evaluation metric for generative models which compare the distributions of real and generated images using a predefined set of attributes, or pairwise occurrences of attributes. The advantage of these metrics over the previous work is that they provide explicit visibility of which aspects contribute to the final metric value.\n\nSTRENGTHS:\nThe idea for the proposed metric is novel – it uses language-image models to measure alignment of distributions of real and generated images given a set of text attributes. Additionally, the metric is highly customizable for downstream tasks because users may define their own set of attributes that are of interest to the specific task and drop irrelevant attributes.\n\nWEAKNESSES:\nFailure modes of the metric are not discussed (or limitations of language-image models and how they affect the metric). \n\nConsider a scenario where there are two models, A and B, with the same SaKLD. Model A produces essentially perfect alignment on all attributes other than one which fails dramatically, causing a large spike in SaKLD histogram, and this attribute is the only contributing to the final score. Model B on the other hand performs poorly across all attributes but averaging over attributes yields the same SaKLD score as the model A. In this scenario SaKLD would potentially not agree with human judgment, since failing in a single attribute might not be visible when inspecting large image grids. Can this kind of scenario occur in practice, and if it can, what would be your recommendation for the user of the metric in that case?\n\nFig. 4 shows that SaKLD and PaKLD are dominated by few attributes of attribute pairs. Is this usually the case in practice? Fig. 5 (b) also indicates that adding new attributes contribute to the metric with diminishing strength. This might be misleading for the user of the metric. Intuitively, adding a large set of attributes should correspond to more thorough evaluation of the model, however, this might not be the case if few attributes are dominating the final value of the metric.\n\nThe empirical effectiveness of the attribute based metric is not fully demonstrated. The authors advocated for an interpretable metric but unfortunately end up comparing modern generative models using single scalar numbers (as the existing metrics do), instead of taking advantage of the interpretability of the metric and showing a more fine-grained analysis of the models.\n\nQUESTIONS:\n1. Fig. 2 shows CS and DCS values for distinct attributes, how would these scores look like for pairs of “opposite attributes”, e.g., “long hair - short hair” or “makeup - no makeup”? Would DCS values be symmetric around zero or something else?\n\n2. The description of Tab. 1 was slightly unclear, how is Tab. 1 exactly calculated? The numerical values seem to be relatively close to each other, is the difference here significant? Tab. 1 shows mean accuracy but what does the accuracy distribution looks like?\n\n3. Automatic extractions of text attributes is an interesting idea, how robust BLIP is with various datasets? What are the most frequently extracted attributes?\n\nMinor notes:\n\n4. Sec. 3.3 a capital letter is missing\n5. Line 154: Needs to be “the center of the images”\n6. Line 220: $n$ and $P_2$ are not defined\n7. Tab. 3 would benefit from also having a column for FID to see how SaKLD and PaKLD differ from it\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new metric to evaluate the quality and diversity of generated images based on\ninterpretable embeddings. To obtain the interpretable embeddings for selected attributes, the cluster\ncenters of the encodings of two separate encoders, one for images and the other for the text attributes\nare calculated and the interpretable embeddings are the difference between the encodings of each\nimage and attribute and their respective cluster centers. The interpretable embeddings represent the\ndirection in which a particular embedding lies with respect to it’s cluster center. The CLIPScore\nbetween the interpretable embeddings of an image and an attribute is calculated and is named as\n’Directional CLIPScore’, since the interpretable embeddings represent the ’direction’.\nThe authors have proposed two metrics : 1. SaKLD - to quatify how closely the attribute distri-\nbution in generated images matches with that of training images. 2. PaKLD - to quantify correlations\nbetween different attributes. The KL divergence between the probability density functions of the\nDirectional CLIPScores for all the images in the train data and generated data.\nUsing SaKLD, the KL divergence between the attribute distribution in training and generated\nimages is calculated. PaKLD calculates the KL divergence similar to SaKLD, except that the presence\nof a pair of attributes is required in the training and generated images.\n\nSTRENGTHS:\nThe motivation for the idea is good and has a huge potential impact for improving evaluation of generative models.\n\nWEAKNESSES:\n- Although, the motivation for developing this metric is valid, the overall methodology and the\nexperimental results are not convincing for the use of this metric. Also, the experiments performed\nare inadequate and do not sufficiently justify how well the metric performs compared to the previously\nproposed metrics. Additionally, previous metrics can directly evaluate quality of generation based\non the generated images alone, but this metric heavily relies on the attributes in the form of text\ndescriptions. Thus, it limits the applicability and generalizability of this metric.\n- The proposed metrics use CLIPScore (which already exists) for interpretable embeddings and then\napplies KL Divergence for the PDFs of the ClipScores of images and attributes, thus, showing limited\nnovelty.\n- Most of the paper is easy to follow but some important parts like, how is the center of text attributes\ncalculated, results from table 1 (what does accuracy stand for) etc. are a bit ambiguous. There is a\nlot of scope to improve the technical soundness of the paper. Although some of the popular metrics\n1are mentioned, there has been a lot of work in the generative modelling domain which the literature\nsurvey must cover. The proposed methodology is not very sound and is not well supported with the\nexperimental setup. The results are also not sufficiently explained. Diversity is the main motivation\nfor the paper as it is mentioned in the abstract but any theoretical or empirical work to support it is\ncompletely missing.\n- Motivation and methodology including the steps to evaluate generated samples is understandable but\nsome parts are ambiguous (please refer above comments).\n- Based on the current state of the experiments, the contributions don’t seem to be significant as there\nis not enough validation to support the claims.\n\n typos:\n- Line 43 : Instead of ”Figure 1 (b)”, it should be Figure 1 (a)\n- Line 168 : Section 3.3 First letters of all the words in the heading must be capital\nOther remarks :\n- Line 56 : Instead of ”If the model lacks essential attributes” the following sounds technically\ncorrect ”if model lacks ”representation” of essential attributes”.\n- Line 56-58 : This claim does not seem to be correct.\n- Line 147 : Link to the specified figure is missing\n- Figure 2 is never refered to in the text, is it an unnecessary figure?\n- Line 253 : Generated set has non-smiling men and non-smiling women. But Figure 4 caption\nsays otherwise\n\nQUESTIONS:\n1. If I understood correctly, the text and the image encoders are different? If yes, then\nthe embedding space of both models is going to be very different and will be influenced by the\ndownstream tasks that they are trained on. Would this metric always work for embeddings from\ndifferent models? Does it make sense to calculate similarity between embeddings from different\nmodels?\n2. Table 1 represents accuracy results for CS vs DCS. What is meant by evaluating ”positive\nsamples” in line 164 and 165? And what is meant by ”how well they are aligned”? Also, what\nexactly is the accuracy calculated for?\n3. In Section 3.1, there is a mention of ”each channel of the embedding being utilized”, how exactly\ndoes this happen?\n4. Line 196-197 claims that the generated distribution is desirable only if it has exactly same number\nof samples as in the training set that has a particular attribute. Why should it be? Afaik, the\nunderlying distribution of latents represent semantic features of the data and the generative\nmodel is responsible to generate novel samples that could be realistic but not necessarily show\nexactly same features same number of times in the training set. So, is the assumption in the\nmentioned lines appropriate? Can you prove this or maybe refer to some papers that provide\nsuch guarantees?\n\nLIMITATIONS:\nlimitations have been addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe embedding space used for calculating FID is computed with Inception V3 which is trained for image classification as the target task. This means it is more likely to capture discriminative features, raising doubts about its ability to effectively evaluate generative models. There is also a need to devise a new evaluation metric that can interpret underlying factors. This paper introduces a method called Directional CLIP Score (DCS) to properly evaluate this. The pre-trained CLIP is used as the embedding space. In particular, the paper proposes \"Single attribute KL div (SaKLD)\" to measure single attribute alignment and \"Paired attribute KL div (PaKLD)\" to measure multiple attribute alignment as new metrics. This paper provides some insightful measurement result using the proposed metrics.\n\nSTRENGTHS:\nThe effort to evaluate generative models from various perspectives through the introduction of such metrics can be considered novel and a contribution. Especially considering the current issues such as the bias in stable diffusion [1], the proposal of such metrics can bring benefits to the field from the perspective of trustworthy AI.\n\nWEAKNESSES:\nIt appears to be an intrinsic limitation that a significant number of samples (50k) are still required to obtain stable results. Nonetheless, thanks to the reported findings, we can gain more insights, and I appreciate that. \nRemaining concerns are written in [Questions] section.\n\nQUESTIONS:\n1) In lines 96-97, it seems that the authors of the referenced paper already propose using the CLIP embedding space for evaluating generative model? If not, it would be helpful to clarify the differences compared to previously proposed methods. Regarding the for explainable evaluation in the related work section, it is necessary to determine whether this paper is the first to come up with it. If there are any additional previous works, it would be helpful to share the, \n\n2) The major concern raised is the reliance on the embedding space. What if there are biases in the CLIP space itself? For example, if being female and wearing makeup are highly correlated, even a woman who does not actually wear makeup may show high similarity to makeup attribute. In such cases, it becomes difficult to consider CLIP score or CLIP direction score as accurate measures of similarity. While a more disentangled multimodal latent space may help alleviate this problem, I'm curious about the authors' perspective on this issue.\n\n3) Directional CLIP Score adopts a method of computing the center of training images and normalizing them. Does this mean that anyone who wants to evaluate a generative model needs to share all these training images? Also, considering the introduction of the auxiliary image captioning model called BLIP, the improvement in performance seems minimal. Do the authors believe it is worth the trade-off?\n\n4) Question about Table 5: cI may have missed it, but was there an explanation for the last column of FID_clip in Table 5? What does it represent? Also, in Table 5, it seems that SaKLD is calculated for a single attribute. How is it computed?\n\n5) Can PaKLD be proposed for more than two attributes? It would be great if judgments could be made for a wide range of attributes as shown in Figure 1 (b).\n\n6) How should we interpret numerical values itself of the scores?  Although it may not be as interpretable as accuracy I'm still curious about the authors' insights based on their experience.\n\nLIMITATIONS:\nIt seems that there are no particular specific limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a very interesting research work. The main contribution of this paper is to consider the attribute information in the original training data when evaluating the quality of images generated by the model. There are two benefits to this approach: 1) determining whether the model can correctly imitate the distribution of the training data; 2) explaining which attributes the model does not perform well on. In implementing this idea, the authors found that directly calculating the CLIPScore between the image representation and the text representation of the attribute does not yield distinctive results. Therefore, they proposed Directional CLIPScore (DSC). The main idea of this approach is to move the reference point for calculating vector similarity to a more reasonable point. At the same time, they proposed two methods to apply DSC, one is Single attribute KL Divergence, and the other is Paired attribute KL Divergence (PaKLD) considering the combination of attributes.\n\nSTRENGTHS:\n1. I think this research question is very interesting. It is very meaningful to use the attribute information of the dataset to help evaluate the quality of the model's generation.\n2. The Directional CLIPScore (DSC) proposed by the authors is very concise and appears to be quite effective in the case study.\n\nWEAKNESSES:\n1. In Section 3.3, I agree with the extraction of attributes using the BLIP and manual annotation methods. However, one method of extracting attributes is to generate an attribute list with GPT and then filter it with training data. I think the attribute list generated by GPT in advance may bring biases. The core of this paper is mainly to study the correlation between the model and the training data. However, if the list generated by GPT is not the most representative attribute, the results may be biased.\n2. The experimental part in Sections 5.1-5.3 does not seem very convincing. The authors mainly verify that the proposed method can indeed be consistent with some expected experimental designs, but there is a lack of more convincing quantitative indicators to show that their proposed evaluation metrics are better than those proposed by others previous research works. I would prefer to see the authors analyze the correlation coefficient between their proposed evaluation metrics and human evaluation, as well as whether their evaluation metrics have improved in terms of correlation coefficient compared to previous evaluation indicators. This is my biggest concern for this work.\n\nQUESTIONS:\n1. In Section 3.1, can the problem exhibited by CLIPScore be solved by directly calculating the Euclidean distance between E(x) and E(a) without relying on Directional CLIPScore?\n2. Notation overload. The meanings of N used in Eq. (5) and Eq. (2) are different.\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two new metrics allowing to measure and explain the diversity of a generated set of images w.r.t a training set. Instead of the usual distributional distances relying upon an embedding space from a pre-trained model, these metrics rely on a set of textual attributes. The similarity between an image and an attribute is computed using their representation in a common semantic space, via the CLIP model - vectors are shifted using a centre of training images/attributes to make similarity scores more meaningful. Several ways to obtain attributes (Captioning, User-based or GPT-based) are investigated. The usefulness of the metrics is tested with an experiment injecting images correlated with target attributes in data, an experiment aiming to detect a specific attribute relationship in a curated dataset, and in a comparison of several generative models.\n\nSTRENGTHS:\n- This paper proposes an interesting approach, aiming at using the common semantic space between images and text proposed by CLIP to measure attribute and image relatedness, in order to provide interpretable representations of generated images and measure attribute-based metrics between a reference and generated dataset; such an application seems relatively original to me. \n- The two first experiments demonstrate well the usefulness of the approach, able to detect a shift when the data has been curated by human, based on which attributes.\n\nWEAKNESSES:\n- The presentation of the paper could be improved upon. This include the writing, and the readability of the figures, as well as the quantity of information provided. This last points concerns mainly the presentation and framing of the problem of interpretability of representations, as well as the presentation and motivation of the experimental settings. \n- In particular, the paper lacks related work on concept-based representation for interpretability of images. While building metrics dedicated to generative model seems new to me, there are based on an idea which has been explored extensively before. See for example \"Concept Whitening for Interpretable Image Recognition, Chen et al, 2020\". \n- The paper focuses on a narrow choice of methods to generate attributes, which, to me, should be one of the key experimental investigation of the paper. Notably, the previous literature explores using different kind of attributes, coming from existing data (for example, \"Interpretable Basis Decomposition for Visual Explanation, Zhou et al, 2018\") or to be learnt (\"A Framework to Learn with Interpretation\", Parekh et al, 2021). The authors only (very shortly) argue about the number of needed attributes.\n- The toy experiments seem relevant but are very fastly presented and should be expanded upon. The remaining experiments are too short to be convincing and only focus on a handful of models.\n\nQUESTIONS:\n- Could you better formalize the notation $DCS_{\\mathcal{X}}(a_{i,j})$ for joint distributions ?\n- Is it necessary to center the text and images differently ? They lie in the same semantic space. Also, would it not be reasonable to be using the attributes themselves, rather than captions ?\n\nLIMITATIONS:\n- Previous distributional metrics are several times referred as relying on external models in your paper. However, the attributes that you use also rely on external models (except the USER one, of course - but in this case, the computing of DCS still relies on a captioning model). How would you address this issue ?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}