On the Evaluation of Conditional GANs
Authors: Terrance DeVries
Adriana Romero
Luis Pineda
Graham W. Taylor
Michal Drozdzal
Venue:  (2020)
Keywords: FJD
Frechet Joint Distance
GAN
cGAN
generative adversarial network
conditional
evaluation
metric
FID
Frechet Inception Distance

Abstract: Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains. Despite outstanding progress, quantitative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity. In this setting, model benchmarking becomes a challenge, as each metric may indicate a different "best" model. In this paper, we propose the Frechet Joint Distance (FJD), which is defined as the Frechet distance between joint distributions of images and conditioning, allowing it to implicitly capture the aforementioned properties in a single metric. We conduct proof-of-concept experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics. Moreover, we use the newly introduced metric to compare existing cGAN-based models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions). We show that FJD can be used as a promising single metric for model benchmarking.

Under review as a conference paper at ICLR 2020
ON THE EVALUATION OF CONDITIONAL GANs
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains. Despite outstanding progress, quan- titative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity. In this setting, model benchmarking becomes a chal- lenge, each metric may indicate different "best" model. In this paper, we propose the Fréchet Joint Distance (FJD), which is defined as the Fréchet distance between joint distributions of images and conditioning, allowing to implicitly cap- ture the aforementioned properties in a single metric. We conduct proof-of- coonccet experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics. Moreover, we use the newly introduced metric to compare existing cGAN-basee models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions). We show that FJD can be used as a promising single metric for cGAN benchmarking and model selection.
INTRODUCTION
The use of generative models growing across many domains (van den Oord et al. 2016c Vondrick ct al. 2016 Serban et al. 2017 Karras et al. [018 Brock et al. 2019)). Among the most promising approaches. Variational Encodess (VAEs) (Kingma & Welling 2014), auto-regressive models (van den Oord et al. 2016a b). and Generative Adversarial Networks (GANs) (Goodfellow et al.
2014) have been driving significant progress, with the latter at the forefront of a made-range of applications (Mirza & Osindero, [0144, Reed et al. 2016 Zhang et al. [2118a] Vondrick et al. 2016]] Almahairi et | 2018 Subramanian et al.) 2018 Salvador et al. 2019). In particular, significant research has emerged from practical applications, which require generation to be based on existing context. For example, tasks such as image inpainting, super-resolution, or eext-to-image synthesis have been successfully addressed within the framework of conditional generation, with conditional GANs (cGANs) among the most competitive approaches. Despite these outstanding advances, quantitative evaluation of GANs remains challenge (Theis al. 2016 Borji 2018).
In the last few years, a significant number of evaluation metrics for GANs have been introduced in the literature (Salimans al. 2016: Heusel et al. 2017 Binkkwski et al. 2018 Shmelkov al.)
2018 Zhou et al.] 2019 Kynkäänniemi et al. [0199] Ravuri & Vinyals 2019). Although there is no clear consensus on which quantitative metric is most appropriate benchmark GAN-based models. Inception Score (IS) (Salimans et al. 2016) and Fréchet Inception Distance (FID) (Heusel et al. 2017) have been extensively used. However, both IS and FID were introduced in the context of unconditional image generation and, hence, focus on capturing certain desirable properties such as visual quality and sample diversity, which do not fully encapsulate all the different phenomena that arise during conditional image generation.
In conditional generation, we care about visual quality, conditional consistency - i.e., verifying that the generation respects its conditioning, and conditionni diversity i.c., sample diversity per conditioning. Although visual quality is captured by both metrics, IS is agnostic to conditioon diversity and FID only captures it indirectly Moreover, neither of them can capture conditional con-
FID compares image distributions and, as such, should be able to roughly capture the intra- cottaaconditioning diversity. Since it cares about the image marginal distribution exclusively, it fails to capture intra-conditioning diversity when changes only affect the image-conditioning joint distribution. See Appendix A

---

Under review as a conference paper at ICLR 2020
sistency. In order to overcome these shortcomings, researchers have resorted to reporting conditional consistency and diversity metrics in conjunction with FID (Zhao et al. 2019 Park et al.. 22019).
Consistency metrics often use some form of concept detector to ensure that the requested conditioning appears in the generated image as expected. Although intuitive to use, these metrics require pre- trained models that cover the same target concepts in the same format the conditioning (i.e., classifiers for image-level class conditioning, semantic segmentation for mask conditioning, etc.), which may or may not be available off-the-shelf. Moreover, using different metrics to evaluate different desirable properties may hinder the process of model selection, as there may not be a single model that surpasses the rest in all measures. In fact, it has recently been demonstrated that there is a natural trade-off between image quality and sample diversity ( (Yang et al. 2019), which calls into question how we might select the correct balance of these properties.
In this paper we introduce new metric called Fréchet Joint Distance (FJD), which is able to implicitly assess image quality, conditional consistency, and conditoonii diversity. FJD computes the Fréchet distance on an embedding of the joint image-conditioning distribution, and introduces only small computational overhead over FID compared to alternative methods. We evaluate the properties of FJD on variant of the synthetic dSprite dataset (Matthey et al. 2017) and verify that it successfully captures the desired properties. We provide an analysis on the behavior of both FID and FJD under different types of conditioning such class labels, bounding boxes, and object masks, and evaluate a variety of existing cGAN models for real-world datasets with the newly introduced metric. Our experiments show that (1) FJD captures the three highlighted properties of conditional generation; (2) it can be applied to any kind of conditioning (e.g., class, bounding box, mask, image, text, etc.); and (3) when applied existing cGAN-based models, FJD demonstrates its potential to be used as promising unified metric for hyper-paramete selection and cGAN benchmarking. To our knowledge, there are no existing metries for conditional generation that capture all of these key properties.
2 RELATED WORK
Conditional GANs have witnessed outstanding progress in recent years. Training stability has been improved through the introduction of techniques such as progressive growing, Karras et al. (0018). spectral normalization (Miyato et al. 2018) and the two time-scale update rule (Heusel et al. 2017).
Architecturally. conditional generation has been improved through the use of auxiliary classifiers
(Odena et al.) 2017) and the introduction of projection- based conditioning for the discriminator (Myyato & Koyama 2018). Image quality has also benefited from the incorporation of self aelf-antention (Zhang et al. 2018a). as well as increases in model capacity and batch size (Brock et al. 2019).
All of this progress has led to impressive results, paving the road towards the challenging task of generating more complex scenes. To this end, a flurry of works have tackled different forms of
conditional image generation, including class-based (Mirza & Osindero 2014 Heusel et al. 2017 7
Miyato et al.
2018. 2018/Odena et al. 2017: Miyato & Koyamal 2018 Brock et al. 2019), image- based
(Isola et al.
2017 [hu et al. 2017a] Wang et al.] 2018 Zhu et 20176b 2 Almahair et al. 2018
Huang et al.
[0188] Mao et al. 2019).
mask- and bounding box-d (Hong et al. 2018 Hinz et al.
2019: Park et al.
2019
Zhao et al.
2019). , as well as text- (Reed et al. [016]] Zhang et al. 2017 7 conditionings (Sharma et al. 2018
2018a Xu et al.
2018 Hong et al. 2018) and dialogue-
2019). This intensified research has lead to the development of variety of metrics to assess the three factors of conditional image generation process quality, namely: visual quality, conditional consistency, and intra-conditioning diversity.
El-Nouby et al.
Visual quality. A number of GAN evaluation metrics have emerged in the literature to assess visual quality of generated images in the case of unconditional image generation. Most of these metrics either focus on the separability between generated images and real images (Lchmann & Romano
2005 Radford et al.) 2016; Yang et af. 2017 Isola et al.) 2017 Zhou et 2019)) compute the
distance between distributions (Gretton et al. 2012 Heusel et al., 2017] Arjovsky et al.) 2017).
assess sample quality and diversity from conditional or marginal distributions (Salimans et al. 2016
Gurumurthy et al. 22017 Zhou et al. 2018)) measure the similarity between generated and real images
(Wang et al. 20044] Xiang & Li, 2017 Snell et al. 2017 Juefei-Xu Xu et al. 2017) or are log- likelihood
based (Thii et al. 2016 Among these, the most accepted automated visual quality metries are
We refer the reader
(Borji 2018 for a detailed overview and insightful discussion of existing metrics.
2

---

Under review as a conference paper at ICLR 2020
Inception Score (IS) (Salimans et al. 2016) and Fréchet Inception Distance (FID) (Heusel et al. 2017).
Conditional consistency. To assess the consistency of the generated images with respect to model conditioning, researchers have reverted to available, pre-trained geed-forward models. The structure of these models depends on the modality of the conditioning (e.g. segmentation models used for mask conditioning or image captioning models are applied to evaluate text conditioning). Moreover, the metric used evaluate the forward model the generated distribution depends on the conditioning modality and includes: accuracy in the case of class-conditioned generation, Intersection over Union when using bounding box- and moask-coniiionings. BLEU (Papineni al., 2002), METEOR (Banerjee & Lavie 2005) or CIDEr (Vedantam et al.) 2015) in the case of text-basee conditionings, and Structural Similarity (SSIM) or peak signal-to-nois ratio (PSNR) for image-conditioning.
Intra-conditioning diversity. The most common metric for evaluating sample diversity is Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., (018b), which measures the distance between samples in learned feature space. Alternatively, (Miyato & Koyama, 2018) proposed Intra-FID, which calculates FID score separately for each conditioning and reports the average score over all conditionings. This method should in principle capture the desirable properties of image quality, conditional consistency, and intra-class diversity. However, it scales poorly with the number of unique conditions, the computationally intensive FID calculation must be repeated for each case, and because FID behaves poorly when the sample size is small (Binkowski et al. (2018). Furthermore, in cases where the conditioning cannot be broken down into set of discrete classes (e.g.. pixel-based conditioning), Intra-FID is intractable. As a result, it has not been applied beyond class-conditioning.
REVIEW OF FRÉCHET INCEPTION DISTANCE (FID)
FID aims to compare the statistics of generated samples to samples from real dataset. Given two multivariate Gaussian distributions M(,, Σ) and (f² Σ), Fréchet Distance (FD) is defined as:
dP (1v, Σ), (a. 2) || AllB Tr Σ - 122
When evaluating generative model, .V Σ)) represents the data (reference) distribution, obtained by fitting a Gaussian to samples from reference dataset, and (f, Σ) represents the learned (generated) distribution, result of fitting samples from generative model.
In FID, both the real images and model samples are embedded in a learned feature space using a pre-trained Inception v3 model (Szegedy et al., (2016). Thus, the Gaussian distributions are defined in the embedded space. More precisely, given dataset of images (x(1) Jx=o set of model samples ( k= and an Inception embedding function f, we estimate the Gaussian parameters μe, E, ft and E as:
N
1
(1(( -1) (f(x(0)) ( -11) - ")"
1
(x)
E
(2)
N
N
i=0
k=0
M
M
A
1
((x()))
(S(Z)0) - 11) (1(2(0) -1)
(3)
M 1=0
M
1=
FRÉCHET JOINT DISTANCE (FJD)
In conditional image generation, dataset is composed of image-condition pairs (xx(), (y)() == where the conditioning can take variable forms, such as image-level classes, segmentation masks, or text. The goal of conditional image generation is produce realistic looking, diverse images & that are consistent with the conditioning y. Thus, set of model samples with corresponding conditioning can be defined as: ()((), {
As discussed in Section B. the Fréchet distance (FD) compares any two Gaussians defined over arbitrary spaces. In FJD, we propose to compute the FD between two Gaussians defined over the joint inage-conditioning embedding space.
3

---

Under review as a conference paper at ICLR 2020
More precisely, given an image embedding function f. a conditioning embedding function h, condi- tioning embedding scaling factor a, and a merging function g that combines the image embedding with the conditioning embedding into joint one, we can estimate the respective Gaussian parameters pe, Σ, f and Σ as:
M Σ M
(1(x()),
(1((()),
1
(4)
N
N
Σ
Σ
(((x()), -w) (1(x")), -w)
(5)
N
i=0
M
Σ
9 (f(x()), ah(311)) (f(x()).ah(((0)) a) (1(f()))) (a -m)²
(6)
M
-
Note that by computing the FD over the joint image-conditioning distribution, we are able to simultancously assess image quality, conditional consistency, and intra-conditioning diversity, all of which are important factors in evaluating the quality of conditional image generation models.
To ensure reproducibility, when reporting FJD scores it is important to include details such which conditioning embedding function was used, which dataset is used for the reference distribution, and the a value. We report these values for all of our experiments in Appendix B
4.1 CONDITIONING EMBEDDING FUNCTION: h
The purpose of the embedding function h is to reduce the dimensionality and extract useful feature representation of the conditioning. As such, the choice of h will vary depending on the modal- ity of conditioning. In most cases, off-the-shelf, pretrained embedding can be used for the purposes of extracting a useful representation. In the absence of preexisting, pretrained conditioning embedding functions, a new one should be learned. For example, for bounding box and mask condi- tionings the embedding function could be learned with autoencoder. For suggested assignments of conditioning modalities to embedding functions please refer to Table 1
Table 1: Suggested embedding functions for different con- ditioning modalities.
Conditioning Modality
Embedding Function
Class / Attribute labels Bounding boxes Masks
One hot encoding
Regularized AE Ghoch et al.
2019
Images Captions/ Dialogue
Inceptionv (zegedy of al
2016
Sentence- BERT Remmers & Gureyych
2019
4.2 CONDITIONING EMBEDDING SCALING FACTOR:
In order to control the relative contribution of the image component and the conditioning component to the final FID value, we scale the conditioning embedding by constant a. essence, a indicates how much we care about the conditioning component compared to the image component. When a = O, the conditioning component is ignored and FJD is equivalent to FID. As the value of a increases, the perceived importance of the conditioning component is also increased and reflected accordingly in the resulting measure. To equally weight the image component and the conditioning component, recommend setting be equal to the ratio between the average L2 norm of the image embedding and the conditioning embedding. This weighting ensures that FJD retains consistent behaviour across conditioning embeddings, even with varying dimensionality magnitude. We note that a should be calculated on data from the reference distribution (real data distribution), and then applied to all conditioning embeddings thereafter. See Appendix F for an example of the effect of the hyperparameter.
*In the initial stages of this project, we also explored methods to bypass this additional training step by projecting a visual representation of bounding box or mask conditioning into an Inceptionv3 embedding space. However, the Inceptionv embedding may not properly capture object positions as it is trained to classify, discard- ing precise spatial information. Therefore, consider autoencoders (AF) to be hetter suited to our setup since they are trained recover both object appearance and spatial information from the embedded representation.
4

---

Under review as a conference paper at ICLR 2020
4.3 MERGING FUNCTION: 9
The purpose of the merging function is to combine the image embedding and conditioning embed- ding into a single joint embedding. We compared several candidate merging functions and found concatenation of the image embedding and conditioning embedding vectors to be most effective, both in terms of simplicity and performance. As such, concatenation is used as the merging function in all following experiments.
EVALUATION OF THE PROPERTIES OF FRÉCHET JOINT DISTANCE
In this section, we demonstrate that FJD captures the three desiderata of conditional image generation, namely image quality, conditional consistency and intra-conditioning diversity.
5.1 DATASET
dSprite-textures. The dSprite dataset (Matthey et al. 2017) is synthetic dataset where each image depicts simple 2D shape a black background. Each image can be fully described by a of factors, including shape, scale, rotation, position, and y position. We augment the dSprite dataset create dSprite- by adding three texture patterns for each sample. Additionally, we include class labels indicating shape, as well as bounding boxes and mask labels for each sample (see Figure D total, the dataset contains 2,211,8 1,84 unique images. This synthetic dataset allows us to exactly control our sample distribution and, thereby, simulate generator with condeccondiiioning inconsistencies or reduced sample diversity. embed the conditioning for calculating FJD in the following experiments, use onc-ho hot encoding for the class labels, and autoencoder representations for the bounding box and mask labels We are releasing the code to generate ASprite-texturee
Figure 1: Left: dSprite-textures images. Center: Bounding box labels. Right: Mask labels.
5.2 IMAGE QUALITY
In this subsection, aim to test the sensitivity of FJD to image quality perturbations. To do so, we draw 10k random samples from the dSprite-textures dataset form
FID
175
a reference dataset. The generated dataset simulated by duplicating the reference dataset and adding Gaussian noise
150
FJD. bbox
Distance
FJD, must
drawn from [00 or) to the images, where E [0, 0.25)
100
and pixel values are normalized (and clipped after noise addition) to the range [0, 1]. The addition of noise mimics generative model that produces low quality images. We repeat this experiment for all three conditioning types in dSprite-textures: class, bounding box, and mask.
75
0.00
0.05
0.20
0.25
0.20
0.25
Noise Magnitude
Results are shown in Figure 2 where we plot both FID and FJD as a function of the added Gaussian noise (a is indicated the z-axis as Noise Magnitude). We find that, all cases, FJD has very similar trend to FID, indicating that it successfully captures image quality. Additional image quality experiments on the large scale COCO-Stuff dataset be found in Appendix
Figure Image quality: FID and FJD exhibit similar trends for class, bound- ing box, and mask conditioning under varying noise levels added to images.
Architectural details for autoencoders used in this paper can be found in Appendix G
5

---

Under review as a conference paper at ICLR 2020
Scale
Orientation
Position
Position
3.-
3.0
Sar
Diotanco
mask
05
0.
Offiet (scale level
Offset (degreis)
Offset(pisels)
Offsct (pixols)
Figure 3: Conditional consistency: Change in FJD with respect to offset on Dsprite-textures dataset
for class, bounding box and mask conditionings.
5.3 CONDITIONAL CONSISTENCY
In this subsection, we aim to highlight the sensitivity of FJD to conditional consistency. In particular, we target specific types of inconsistencies, such as incorrect scale, orientation, or position. We draw a set of 10k samples from the dSprite-textures dataset and duplicate it to represent the reference dataset and the generated dataset, each with identical image and conditioning marginal distributions. For 30% of the generated dataset samples we swap conditionings of pairs of samples that are identical in all but one of the attributes (scale, orientation, x position or y position). For example, if one generated sample has attribute 2 position 4 and a second generated sample has attribute position 7, swapping their conditionings leads to generated samples that are offset by 3 pixels w.r.t. their ground truth æ position. Swapping conditionings in this manner allows us to control for specific attributes' conditional consistency, while keeping the image and conditioning marginal distributions unchanged. As result, all changes in FJD can be attributed solely to conditional inconsistencies.
Figure 3 depicts the results of this experiment for four different types of alterations: scale, orientation, and x and y positions. We observe that the FID between image distributions (solid blue line) remains constant even as the degree of conditional inconsistency increases. For class conditioning (dotted orange line), FJD also remains constant, as changes to scale, orientation, and position are independent of the object class. Bounding box and mask conditionings, as they contain spatial information, produce variations in FJD that are proportional to the offset. Interestingly, for the orientation offsets, FJD with mask conditioning fluctuates rather than increasing monotonically. This behaviour is due to the orientation masks partially re-aligning with the ground truth around 90º and 180" Each of these cases emphasize the effective sensitivity of FJD with respect to conditional consistency. Additional conditional consistency experiments with text conditioning can be found in Appendix [
5.4 INTRA-CONDITIONING DIVERSITY
In this subsection, we aim to test the sensitivity of FJD to intra-conditioning diversity by alter- nating the per-conditioning image texture variability. More precisely, we vary the texture based on four different image attributes: shape that is captured in all tested conditionings, as well as scale, orientation and position that are captured by bounding box and mask conditionings only. To create attribute-tex -exture assignments, we stratify attributes based on their values. For example, one possible shape-based based stratification of dataset with three shapes might be: [squares, ellipses, hearts). To quantify the dataset intra-conditioning diversity, we introduce a diversity score. A diversity score of I means that the per-attribute texture distribution is uniform across stratas, while a diversity score of O means that each strata is assigned to a single texture. Middling diversity scores indicate that the textural distribution is skewed towards one texture type in each strata. We create our reference dataset by randomly drawing 10k samples. The generated distribution is created by duplicating the reference distribution and adjusting the per-attribute texture variability to achieve the desired diversity score.
The results of these experiments are shown in Figure 4 which plots the increase in FID and FJD, for different types of conditioning, as the diversity of textures within each subset decreases. For all tested scenarios, we observe that FJD is sensitive intra-conditioning diversity changes. Moreover, not surprisingly, since a change in the joint distribution of attributes and textures also implies a change to the image marginal distribution, we observe that FID increases with reduced diversity. This
'Note that for real datasets, intra-conditioning diversity is most often reduced as the strength of conditioning increases (e.g., mask conditionings usually present single image instantiation, presenting diversity).

---

Under review as a conference paper at ICLR 2020
Shape
Scale
Orientation
Position
175
13.0
32.3
bbox
Distance
Distance
16.0
23
7.5
13
intra-condtioning Diversity
conditionin Diversity
Intra sondstnnninnnnggggg Diversity
condionndioonnngggoomm Diversity
Figure 4: Intra-conditioning diversity: FJD and FID as intra-conditioning diversity decreases.
experiment suggests that FID is able to capture intra-conditioning diversity changes when the image conditional distribution is also affected. However, if the image marginal distribution were to stay constant, FID would be blind to intra-conditioning diversity changes (as is shown in Section 5.3).
6 EVALUATION OF EXISTING CONDITIONAL. GENERATION MODELS
In this section, we seek to demonstrate the application of FJD to evaluate models with several different conditioning modalities, in contrast FID and standard conditional consistency and diversity metrics. We focus on testing class-conditioned, image generation tasks, which have been the focus of numerous work Multi-label, bounding box, and mask conditioning are also explored in Appendix [ We note that FJD and FID yield similar rankings of models in this setting, which is to be expected since most models use similar conditioning mechanisms. Rankings therefore dominated by image quality, rather than conditional consistency. We refer the reader to Appendix and H for examples of cases where FJD ranks models differently than FID.
mmage-conditioned and text-conditioned
Class-conditioned cGANs. Table
2 compares three state-of-the-art class-conditioned generative models
Table 2: Comparison of class-conditioned models trained on ImageNet (resolution 128 x 128).
trained ImageNet at 128 x 128
resolution. Specifically, we evalu- SN-GAN (Miyato et al.) 20188) trained with and without a projection diseriminator (Miyato & Koyamal 2018), and BigGAN (Brock et al.
FJD 4
FID
Acc. 1
Diversity
SN-GAN (concat)
63.7
39.8
18.2
0.622
SN-GAN (proj)
41.7
27.4
35.7
0.612
2019). Accuracy is used to evaluate conditional consistency, and is com- puted as the Inception v3 accuracy of each model's generated samples, using their conditioning as used as conditioning generate
BigGAN
17.0
9.55
67.4
0.550
classification ground truth. Class labels from the validation 50k samples for each model, and the training set is used as the reference distribution. One-hot encoding is used to embed the class conditioning for the purposes of calculating FJD.
We find that FJD follows the same trend as FID for class-conditioned models, preserving their ranking and highlighting the FJD's ability capture image quality. Additionally, we note that the difference between FJD and FID correlates with each model's classification accuracy, with smaller gaps appearing to indicate better conditional consistency. Diversity scores, however, rank models in the opposite order compared to all other metrics.
This behaviour evokes the trade-off between realism and diversity highlighted by Yang et al. (2019). Ideally, we would like a model that produces diverse outputs, but this property is not as attractive if it also results in a decrease image quality. what point should diversity be prioritized over image quality, and vice versa? FJD is suitable metric for answering this question if the goal is to find model that best matches the target conditional data generating distribution.
Image-conditioned cGANs. Table 3 compares four state-of-the-art image translation models: Pix2pix (Isola et al.) (017), BicycleGAN (Zhu et al., 20175). MSGAN (Mao al., 2019), and MUNIT (Huang al. 2018). We evaluate four different image-to-imet -mage datasets: Facades
TTylečkk & Sára 2013). Maps (Isola et al. 2017). Edges2Shoes and Edges2Handbag (Zhu et al.
AA list of pre-trained models used in these evaluations can be found in Appendix

---

Under review as a conference paper at ICLR 2020
Table 3: Comparison of image-conditioned models. Results averaged over 5 runs.
Dataset
Facades
Maps
FJD +
FID +
Consistency 4
Diversity
FJD
FID
Consistency +
Diversity
Pix2pix
161.3
104.0
0.413
0.056
233.4
106.8
0.444
0.049
BicycleGAN
145.9
85.0
0.436
0.289
220.4
93.2
0.449
0.247
MSGAN
152.4
93.1
0.478
0.376
249.3
123.3
0.478
0.452
Edges2Shoes
Edges2Handbags
Pix2pix
115.4
74.2
0.215
0.040
162.3
95.6
0.314
0.042
BicycleGAN
88.2
47.3
0.239
0.191
142.1
76.0
0.324
0.252
MUNIT 98.1
56.2
0.270
0.229
147.9
79.1
0.382
0.339
2016). To assess conditional consistency utilize LPIPS measure the average distance between generated images and their corresponding ground truth images. Conditioning from the validation sets are used to generate images, while the training sets are used as reference distributions. An Inceptionv3 model is used to embed the image conditioning for the FJD calculation. Due to the small size of the validation sets, we report scores averaged over 5 evaluations of each model.
In this setting we encounter some ambiguity with regards to model selection, as for all datasets, each metric ranks the models differently, BicycleGAN appears to have the best image quality, Pix2pix produces images that are most visually similar to the ground truth, and MSGAN and MUNIT achieve best sample diversity scores. This scenario demonstrates the benefits of using a single unified metric for model selection, for which there is only single best model.
Text-conditioned eGANs. Table shows FJD and FID scores for three state-of-the-an text-conditioned mod- els trained the Caltech-UCSSD Birds 200 dataset (CUB-200) (Welin- der al. 2010) 256 256 resolu- tion: HDGan (Zhang et al. 2018c),
Table 4: Comparison of text-conditioned models trained on CUB-200 (resolution 256 256).
FJD /
FID )
VS sim.
Diversity 1
HDGan
26.1
23.3
0.340
0.687
StackGAN++
21.8
18.4
0.341
0.652
StackGAN++ (Zhang et 2018a),
and AttnGAN (Xu et al. 2018). Con-
AttnGAN 16.7
13.6
0.477
0.625
ditional consistency evaluated us-
ing visual-semantic similarity, as proposed by Zhang et al. (018c). Conditioning from the test set captions is used to generate 30k images, and the same test set is also used as the reference distribution. We use pre-computed Char-CNN-RNN sentence embeddings as the conditioning embedding for FJD, since they are commonly used with CUB-200 and are readily available.
In this case we find that AttnGAN dominates in terms of conditional consistency compared to HDGan and StackGAN++, while all models are comparable in terms of diversity. AttnGAN ranked best overall by FJD. In cases where the biggest differentiator between the models is image quality, FID and FJD will provide a consistent ranking as we see here. In cases where the trade-off more subtle believe practitioners will opt for metric that measurably captures intra-conditioning diversity.
7 CONCLUSIONS
In this paper we introduce Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric. We compare FJD to FID on the synthetic dSprite-textures dataset, validating its ability to capture the three properties of interest across different types of conditioning, and highlighting its potential be adopted a unified cGAN benchmarking metric. We also demonstrate how FJD can be used to address the potentially ambiguous off between image quality and sample diversity when performing model selection. Looking forward, FJD could serve as valuable metric to ground future research, as it has the potential to help elucidate the most promising contributions within the scope of conditional generation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Aug- mented CycleGAN: Learning many-to-man mappings from unpaired data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 195-204, Stockholmsmääsann Stock- holm Sweden, 10-15 Jul 2018. PMLR. URL
almahairi18a.hth
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214-223, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http: //proceedingg..m..ppe...... www.:pppwsss////pmm/om 2r100//rroova joooaaaryoosy..aht...
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL. https
//openreviie.ii eet wtw.:/nee/ffouummmmtmmm wwrrrmmmiirrr 100ZWCW
Ali Borji. Pros and cons of GAN evaluation measures. CoRR, ftbb//802.034466, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019. URL
Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pp. 1209-1218. IEEE Computer Society, 2018.
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, R. Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W. Taylor. Tell, draw, and repeat: Gen- erating and modifying images based continual linguistic instruction. CoRR, hts/1811.0984. 2019.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. CoRR, abs/1903.12436 2019. URL http:
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2672 2680. Curran Associates, Inc., 2014. URL
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Berhaard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13(1):723-773, March 2012. ISSN 1532-4435.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Computer Vision and Pattern Recognition, pp. 1-449. IEEE Computer Society, 2017. ISBN 978-1-5386-0457-1. URL http://
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems 30. pp. 6626 6637. Curran Associates, Inc., 2017. URL http: me sca iineecaceeeeeaaerreeeeeeeeeeeeer
ummppff

---

Under review as a conference paper at ICLR 2020
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. International Conference on Learning Representations, 2019. URL https:// Oppnreview.net// httpee/eeeoooommmiiieeeeeeeaaaaa
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Computer Vision and Paltern Recognition, pp. 7986-7994. IEEE Computer Society, 2018.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to to-e-ttogemmee translation. ECCV, 2018.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Emage-to-image translation with conditional adversarial networks. Computer Vision and Patler Recognition (CVPR), 2017.
Justin Johnson, Agrim Gupta, and Fei-Fei. Image generation from scene graphs. In CVPR, pp. 1219-1228. IEEE Computer Society, 2018.
Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Gang of gans: Generative adversarial networks with maximum margin ranking. CoRR, abs/1704.04865, 2017. URL
http://arxivoorg//as/11....
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. URLtttpss: wtwpeeerrriew.eeee/oruuuun
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. arXiv preprint arXiv: 1904.0691,, 2019.
Yann LeCun, Léón Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2324 1998.
E. L. Lehmann and Joseph Romano. Testing statistical hypotheses. Springer Texts in Statistics. Springer, third edition, 2005. ISBN 0-387-98864-5.
Qi Mao, Hsin Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma. and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://gggaabccoommmepemnnddddpp
2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/ 17.,1444 2014.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. International Con- ference on Learning Representations, 2018. /tll:/ttee.///ppeneeveeeeeeeeeee /ttp:///wwe.ee.eee/ooommmm.o
By51VpgRB
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
URL httpppss//eeeeeeeeeeeeeeeeteaaae B0-B
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with aux- iliary classifier GANs. In Proceedings of the 34th International Conference on Machine Learn- ing- Volume 70, ICML' pp. 2642-2651. JMLR.org. 2017. URL http:/ //dp:///ll... www.am...........mmmmm.m. om.org// citation.cfm..i... 3at.mm22ddd330033030553333
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th anmual meeting on association for computational linguistics, pp. 31 1-318.. Association for Computational Linguisties, 2002.
10

---

Under review as a conference paper at ICLR 2020
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, In International Conference on Learning Represen- tations, 2016.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. arXiv preprint arXiv:/903 10887, 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lec. Generative adversarial text image synthesis. In Proceedings of the 33rd International Confer- ence on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1060-1069. JMLR.org, 2016. htt.:p/pp......eooocc/aaaaaoo.o.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1otp/199880004,, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2234-2242, Curran Associates, Inc., 2016. URL http::/::aapee.aa.....pppppppppe Lmppooeed-teciiiquss-- for www.ettraiinnginnnee gans.pdf
Amaia Salvador, Michal Drozdzal, Xavier Giró Nieto, and Adriana Romero. Inverse cooking: Recipe generation from food images. In Computer Vision and Patter Recognition (CVPR). IEEE Computer Society, 2019.
Julian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI, pp. 3295-3301. AAAI Press, 2017.
Shikhar Sharma, Dendi Suhubdy, Vincent Michalski, Samira Ebrahimi Kahou, and Yoshua Bengio. Chatpainter: Improving text to image generation using dialogue. CoRR, abs/1802.08216, 2018.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? CoRR, (bs//107,,0949, 2018. URL
Jake Snell, Karl Ridgeway, Renjie Liao, Brett D. Roads, Michael C. Mozer, and Richard Zemel. Learning to generate images with perceptual similarity metrics. In 2017 IEEE Inter- national Conference on Image Processing, ICIP 2017, Beijing, China, September 17-20, 2017, doi: 10. D2.109//CIP.0017.22200
pp. 4277-4281 2017.
ICIP CCIP.2017.829705
Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C Courville, and Chris Pal. Towards text generation with adversarially learned neural outlines. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7551-7563, Curran Associates, Inc., towards text generation
2018. URL /ttp:pppp//ppppesspppccc////pp/p
wi w.D--awwerrrrraalyy-aaeeeeeeeeee
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, pp. 2818-2826. IEBE Computer Society, 2016.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations, 2016.
Radim Tyleček and Radim Šára. Spatial pattern templates for recognition of objects with regular structure. In Proc. GCPR, Saarbrucken, Germany, 2013.

---

Under review as a conference paper at ICLR 2020
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In D. D. Lee, M. Sugiyama. U. Luxburg, L. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems 29. pp. 4790-4798. Curran Associates, Inc., 20. URL 6527-condin ona www.mmaeeeeee...eee. www.:oot-t-i-peeeeoee-eeeeeeeeee
paper/
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICML'16, pp. 1747-1756. JMLR.org. 2016b. URL htttttttpd.......././
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Arxiv, 2016c. URL http:/tttt//xiiieee http:////////bb///9..03/43.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-baid based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575 2015.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynam- ics. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems 29, pp. 613-621. Curran Associates, Inc., 2016. -hteesseeettsseeeeee
http://ppppprssspp..cpppaapeeeee dyramics.pdff
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE TRANSACTIONS ON IMAGE PROCESSING, 13(4):600- 012:66000611,,00 2004.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technoloog. 2010.
Sitao Xiang and Hao Li. On the effects of batch and weight normalization in generative adversarial networks. arXiv preprint arXiv: 7704003971 2017.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attogan: Fine- aine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1316-1324, 2018.
Dingdong Yang, Seunghoon Hong. Yunseok Jang, Tianchen Zhao and Honglak Lec. Diversity- sensitive conditional generative adversarial networks. arXiv preprint arXiv: (001.0900000024 2019.
Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. LR-GAN: layered recursive gen-
erative adversarial networks for image generation. In International Conference Learning Representations. OpenReview.net, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stackgan++: Realistie image synthesis with stacked generative adversarial networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, July 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018b.
12

---

Under review as a conference paper at ICLR 2020
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with a hierarchically- nested adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6199-6208, 2018c.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In Computer Vision and Putlern Recognition. IEEE Computer Society, 2019.
Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Durim Morina, and Michael S Bernstein. Hype: Human eye perceptual evaluation of generative models. arXiv preprint 1rXiv:190..0112,, 2019.
Zhiming Zhou, Han Cai, Shu Rong, Yuxuan Song, Kan Ren, Weinan Zhang, Jun Wang, and Yong Yu. Activation maximization generative adversarial nets. In International Conference on Learning Representations, 2018. URLL/httpss:/ppeneeeeeenaa
Jun-Yan Zhu, Philipp Krähenbühl,, Eli Shechtman, and Alexei A. Efros. Generative visual manipula- tion on the natural image manifold. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part pp. 597-613, 2016. URL pttp::///ooo.occ///00000/77----3
doi: 10.
Jun-Ye Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017a.
Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in Neural Information Processing Systems, 2017b.
13

---

Under review as a conference paper at ICLR 2020
A ILLUSTRATION OF FID AND FJD ON TWO DIMENSIONAL GAUSSIAN DATA
In this section, we illustrate the claim made in Section 1 that FID cannot capture intra-conditioiin diversity when the joint distribution of two variables changes but the marginal distribution of one of them is not altered.
Consider two multivariate Gaussian distributions, Y)) (O E1) and (X2, Y2) NO E2). where
4 2
2 2 Σ
2.1 2
Σ
2
Figure (left) shows 10, 000 samples drawn from each of these distributions, labeled as Distl and Dist2, respectively. While the joint distributions of fx,,;, .Y, (X1, Y1) and fx,.y, (X2, Y2) are different from each other, the marginal distributions fyj (Y1) and fvg (Y2) are the same (Y ~ N 2) and
Y/2 N (0, 2)). Figure 5 (center) shows the histograms of the two marginal distributions computed from 10, 000 samples.
If we let X; take the role of the embedding of the conditioning variables (c.g., position) and Y; take the role of the embedding of the generated variables (i.e., images), then computing FID in this example would correspond to computing the FD between fy, and fy., which is zero. On the other hand, computing FJD would correspond the FD between fx,,y, and fx,,Y2, which equals 0.678. But note that Distl and Dist2 have different degrees coda-coondiioninn diversity, as illustrated by Figure 5 (right), where histograms of e are displayed, showing marked differences to each other (similar plots can be constructed for other values of Xt). Therefore, this example illustrates situation in which FID is unable to capture changes in intra-conditioning diversity, while FJD is able to do so.
Dict
Dud
2
2
Figure 5: Left: samples from two multivariate Gaussian distributions. Center: Histograms of marginal distributions for the Y variable. Right: Histogram of conditional distributions for Y conditioned on X (0.9, 1.1).
B EXPERIMENTAL SETTINGS FOR CALCULATING FJD
Important details pertaining to the computation of the FID and FJD metrics for different experiments included in this paper are reported Table For each dataset we report which conditioning modality was used, as well as the conditioning embedding function. Information about which split and image resolution are used for the reference and generated distributions also included, as well as how many samples were generated per conditioning. Values for reported here are calculated according to the balancing mechanism recommended in Section 4.2 Datasets splits marked by "_" indicate that the distribution is randomly sampled subset of the full dataset.
C IMAGE QUALITY EVALUATION ON COCO-STURE DATASET
We repeat the experiment initially conducted in Section 5.2 on real world dataset to see how well FJD tracks image quality. Specifically, we use the COCO-Stufi dataset (Caesar et al. 2018), which provides class labels, bounding box annotations, and segmentation masks. We follow the same experimental procedure as outlined in Section 5.2 Gaussian noise is drawn from A (0, or) and add to the images, where [0, 0.25) and pixel values are normalized (and clipped after noise addition) to
14

---

Under review as a conference paper at ICLR 2020
1. D.00 1000882062222
O. 0.00 100999662
O. 4.00003995001000
17. 17.465029
0.54181314
0.38 33800999
17. 7.810513
0. 0.9376455
1.1100407
0. 0.7325.5646
0.77434437
4. 1.26644555
8. 9.5398-15
1.00351441
0.0150718
OK
Chur- -an--CNN-NNN
Embedding
Encoder
Auto Encuder
Inception
Inception
Inception
Inception V3
AutoEncoder
Auto Encoder
Auto EncoEre
AutoEncoder
One-hu hot
One-hot
One-hot
One-hot
Table 5: Settings used to calculate FJD in experiments.
Img. res.
64
64
64
128
256
512
256
256
256
64
64
64
128
1.28
128
Generated distribution
Samp. per cond.
Valid. + Test
Split
Valid.
Test
Test
Test
Valid.
Valid.
Valid.
Valid.
Valid.
Valid.
Valid.
Reference distribution
Img. res.
64
64
64
128
256
512
256
256
256
64
64
64
128
128
128
Split
Train
Train
Train
Train
Train
Valid.
Valid.
Valid.
Valid.
Valid.
Valid.
Valid.
Modality
Class
BBox
Mask
Class
Image
Image
Image
Image
Text
Class
BBox
Mask
Class
BBox
Mask
Edges2Hamde Hanndbugs
COCO-Stuff
COCO-Stuff
COCO-Stuff
COCO-Stuff
COCO-Stuff
COCO-Stuff
Edges2Shocs
Dataset
dSprite
dSprite
aSprite
ImageNet
Facades
CUB-200
Maps
15

---

Under review as a conference paper at ICLR 2020
the range [0, 1]. The original dataset of clean images is used the reference distribution, while noisy images are used to simulate generated distribution with poor image quality. For the purposes of calculating FJD, use N-hot encoding embed the labels of the classes present in each image. and autoencoder representations for the bounding box and mask labels. As shown Figure ( FII) and FJDD both track image quality well, increasing distribution.
noise is added the generated image
140
FID
FJD,
FJD, bbox
100
FJD, mask
Distance
60
20
0.00
0.05
0.10
0.15
0.20
0.25
Noise Magnitude
Figure 6: Comparison between FID and FJD for class, bounding box, and mask conditioning under varying noise levels for COCO-Stuff dataset. Evaluated at 128x128 resolution.
D CONDITIONAL CONSISTENCY EVALUATION WITH TEXT CONDITIONING
In order to test the effectiveness of FJD at detecting conditional inconsistencies in the text domain, we use the Caltech-UCSS UCCD Birds 200 dataset (Welinder et al.) 2010). This dataset is a common benchmark for text conditioned image generation models, containing 200 one-grained bird categories, 11,788 images, and 10 descriptive captions per images. Also included of detailed binary annotations describing the attributes of the bird in each image. Each annotation indicates the presence or absence specific features, such as has billl billl__shaper curved or has \________ wiwing niggg __coloroo buue.
the dataset are vectors
Our goal in this experiment is to swap captions between images, and in this fashion introduce inconsistencies between images and their paired captions, while preserving the marginal distributions of images and labels. We compare attribute vectors belonging to each image using the Hamming distance to get an indication for how well the captions belonging one image might describe another. Small Hamming distances indicate good match between image and caption, while at larger values the captions appear describe a very different bird than what is pictured (as demonstrated in Figure
Hamming Distance
Caption
This is fully covered except for some parts of wing and has brown feet.
A red bird with short bill with a black cheek patch.
This small bird is bright red with black wings, black eyeing, and short black beak.
eyes. The body the bird ivory and the crown is bright red while the wing black and Ivory speckled.
bird yellow
51
Figure 7: A summer tanager, described by variety of captions (ground truth caption highlighted in blue). The Hamming distance between attribute vectors associated with each caption and the ground truth caption provides an indication of how well each caption describes the image.
To test FJD create datsets: which contains the original image-captions pairs from CUB- 200 to act as the reference distribution, and another in which captions have been swapped to act generated distribution that has poor conditional consistency. Thar-CNN-RNN embeddings
16

---

Under review as a conference paper at ICLR 2020
used encode the captions for the purposes of calculating FJD. In Figure & we observe that the average Hamming distance across captions increases (i.e., the captions become worse at describing their associated images), FJD also increases. FID, which is unable to detect these inconsistencies, remains constant throughout.
FID
FJD, caption
10
20
30
Average Hamming Distance
Figure 8: Change in FJD and FID with respect to the average Hamming distance between original captions and swapped captions. FJD increases as captions become worse at describing their associated image, while FID is insensitive.
LIST OF SOURCES OF PRE-TRAINED MODEL
Table 6 includes the hyperlinks to all of the pretrained conditional generation models used in our experiments in Section
Table 6: Source pre-trained models evaluated in Section 6
Model
Source
SN-GAN BigGAN
ottp://shtthuuooorr////s///01112 pix2pxx
http://tttluoo////a///////aodddr MSGAN
Pix2pix
BicyleGAN
MUNIT
HDGan
ht ht:pa///gichuucco////z2////aaa
StackGAN+
AttnGAN
EFFECT OF a PARAMETER
The a parameter in the FJD equation acts a weighting factor indicating the importance of the image component versus the conditional component. When = 0, then FJD is equal to FID, since we only care about the image component. As the value of increases, the magnitude of the conditional component's contribution to the value of FJD increases well. In our experiments, find neutral value for a that will balance the contribution from the conditional component and the image component. This balancing is done by finding the value of a that would result in equal magnitude between the image and conditioning embeddings (as measured by the average L2 norm of the embedding vectors).
attempt to
Instead of reporting FJD at a single a, an alternative approach to calculate and plot FJD for a range of values, shown in Figure Plotting versus FJD allows observe any change in rank of models as the importance weighting on the conditional component is increased. Here we use the truncation trick to evaluate BigGAN (Brock et al. 2019) at several different truncation values a. The
17

---

Under review as a conference paper at ICLR 2020
0.25
1.00
1.75
40
Suggested
30
20
10
Figure Alpha sweep for BigGAN at various truncation values a. FID equivalent to FJD O. The black dashed line indicates the value that is selected by calculating the ratio between the average Le norm of image and conditioning embeddings.
truncation trick is a technique wherein the noise vector used to condition GAN is scaled by a in order trade sample diversity for image quality and conditional consistency, without needing to retrain the model (as shown in Table 7
Table 7: Comparison of BigGAN model evaluated with different truncation values a. FJD is calculated at a 17.8. As inereases, classification accuracy decreases and diversity increases. Note that FID and FJD are consistent in their choice of the preferred model at a = 1.0, however, the relative ranking of 0.25 and o 2.0 different between the two metrics.
FJD J
FID |
Acc. 1
Diversity 1
0.25
50.5
22.6
81.7
0.247
0.5
33.8
16.5
80.8
0.380
1.0
17.1
9.6
67.5
0.550
1.5
26.2
16.3
45.9
0.644
1.75
37.3
23.9
35.8
0.674
2.0
49.1
31.9
27.0
0.696
We find that in several cases, the ranking of models changes when comparing them at a = 0 (equivalent FID), versus comparing them using FJD higher a values. Models with low truncation values initially achieve good performance when a is also low. However, as a increases, these models rapidly drop in rank due to lack of sample diversity, and instead models with higher values are favoured. This is most obvious when comparing = 0.25 and = 1.75 (blue and yellow lines Figure 9 respectively.
G AUTOENCODER ARCHITECTURE
To create embeddings for the bounding box and mask conditionings evaluated in this paper utilize a variant of the Regularized AutoEncoder with Spectral Normalization (RAE-SA SN) introduced by Ghosh et al. (219)) and enhance it with residual connections (Tables X and For better reconstruction quality, we substitute the strided convolution and transposed convolution for average pooling and nearest neighbour upsampling, respectively. Spectral normalization (Miyato al. (018) applied all linear and convolution layers in the decoder, and an Le penalty is applied to the latent representation z during training. Hyperparameters such as the weighting factor on the Le penalty and the number of dimensions in the latent space selected based on which combination produces the best reconstructions on a held-out validation set.
In Tables 10 and I
depict the architecture for autoencoder with 64 64 input resolution. but this can be scaled up or down by adding or removing residual blocks as required. ch represents a channel multiplier which is used to control the capacity of the model. M represents the number of
18

---

Under review as a conference paper at ICLR 2020
latent dimensions in the latent representation. C' indicates the number of classes in the bounding box or mask representation.
Table 8: ResBlock down
Table 9: ResBlock up
Input x
Input x
Σ + ConV3 x 3 - BN - ReLU - out
x + Conv3 x - BN - ReLU - out
out - Conv3 x BN - ReLU + out
out - Conv3 x 3 + BN - out
out + AvgPool2 x out
out Upsample2 x 2 - out
Conv
res
- Conv, x
res - AvgPool2 x > res
res + UUsaampe2 + res
oul + res + ReLU + oul
oul + res - ReLU + oul
Table 11: Decoder
Table 10: Encoder
2 E R
Input E R61x61xC
Linear M - ch x 8 x 8
ResBlock down C' - ch
BN + ReLU
ResBlock down ch. 2ch
ResBlock up 8ch - 4ch
ResBlock down 2ch + 4ch
ResBlock up 4ch -> 2ch
ResBlock down 4ch + 8ch
ResBlock up 2ch + ch
Linear 8ch x 4 x + M
Conv ch - C
Tanh
H FJD FOR MODEL SELECTION AND HYPERPARAMETER TUNING
In order to demonstrate the utility of FJD for the purposes of model selection and hyperparameter tuning, we consider the loss function of the generator from an auxiliary classifier GAN (ACGAN)
(Odena et al., 2017). as shown in Equation Here S indicates the data source, and C indicates the class label.
Ls Ellog P(S = real (real + Ellog P(S fakel (ake))
(7)
Lc E'log P(C c c/ere)))] + Ellog PCC = fake)
(8)
LG XLc Ls
(9)
The generator loss La is maximized during training, and consists of two components: an adversarial component Ls. which encourages generated samples to look like real samples, and a classification component Lc. which encourages samples to look more like their target class. In this experiment we add a weighting parameter X, which weights the importance of the conditional component of the generator loss. The original formulation of ACGAN equivalent to always setting unknown whether this is the most suitable setting as it is never formally tested. To this end, we train models on the MNIST dataset and perform a sweep over the À parameter in the range [0, 5]. training single model for each À value tested. Each model is evaluated using FID, FJD, and classification accuracy to indicate conditional consistency. For FID and FJD we use the training set as the reference distribution, and generate 50, 000 samples for the generated distribution. Classification accuracy is measured using a pretrained LeNet classifier (LeCun et al. 1998), where the conditioning label is used the groundtruth.
1, but
19

---

Under review as a conference paper at ICLR 2020
Scores from best performing models indicated by FID, FJD, and classification accuracy shown in Table [2 Sample sheets are provided in Figure where each column is conditioned on a different digit from 0 to 9. We find that FID is optimized when À 0.25 (Figure [Oa). This produces model with good image quality, but almost no conditional consistency. Accuracy is optimized when À = 5.0 (Figure [ yielding model with good conditional consistency, but limited image quality. Finally, FJD is optimized when À = 1.0 (Figure (Ob), producing model that demonstrates a balance between image quality and conditional consistency. These results demonstrate the importance of considering both image quality and conditional consistency simultaneously when performing hyperparameter tuning.
Table 12: Scores of ACGAN models trained with different values for conditioning weighting
FID
FJD 4
Accuracy
0.25
56.87
79.65
12.22
1.0
62.39
65.31
74.90
5.0
115.23
119.10
98.01
9
5
(
6
0
6
0
2
3
4
5
6
7
8
9
O
R
3
4
5
6
8
f
5
G
4
1
3
5
8 9
0
7
3
4
5
6
7
8
9
0
2
3
6
8
9
4
4
7 3 1
5
O
1
2
3
4
7 7 7 7 7 666888
5
6 6 6 6 6
8
9
0
2
3
4/
6
8
9
5
3
3
2
O
2
3
4/
5
9
O#
7
3
9
8
8
4
0
9 6 6 5 9
8
O
1
2
4
<
9 8 8
9
01 4
9
7
4
0
6 6 1 4 3
S
O
/
3
4
5
9
D
9
4
5
5
5
4
3
3
0
2
3
1
8
9
2
5 6
9
7
/
3
0
7
3
0
/
2
3
4
04
1
3
4
7
7
5
6
4
0
2
3
4
S
8 9
2/
3
5
5
7
7
3
Q
0
3
0
2
3
4
5
6
8
0 1
2
3
4
5
6
8
(a) À 0.25
(b) A
1.0
(c)
A
5.0
Figure 10: Sample sheets for ACGAN model trained with different conditioning weighting Each column is conditioned different digit, from 0 to 9. Low values of À produce models with very little conditional consistency, while overly large values of À yield models with reduced image quality and diversity.
I TRAINING AND EVALUATING WITH MULTI-LABEL, BOUNDING BOX, AND MASK CONDITIONING ON COCO-STUFF
To demonstrate FJD applied to multi-label, bounding box, and mask conditioning on a real world dataset, train GAN the COCO-Stuff dataset (Caesar et al.) 2018). To this end, train three generative models, one for each conditioning type. Following (Johnson et al.) 2018), we select only images containing between 3 and 8 objects, and also ignore any objects that occupy less than 2% of the total image area. Two image resolutions are considered: 64 x 64 and 128 x 128. We adopt BigGAN-sttye model (Brock al. 2019), but modify the design such that single fixed architecture can be trained with any of the three conditioning types. See Section [.] for architectural details. We train each model 5 times, with different random seeds, and report mean and standard deviation of both FID and FJD in Table 13, N-hot encoding is used as the embedding function for the multi-label conditioning, while autoencoder representations are used to calculate FJD for bounding box and mask conditioning,
In most cases we find that FID values are very close between conditioning types. A similar trend observed in FJD the 128 x 128 resolution. For models trained at 64 64 resolution however, we notice a more drastic change in FJD between conditioning types. Mask conditioning achieves the lowest FJD score, followed by multi-label conditioning and bounding box conditioning. This could indicate that the mask conditioning models are more conditionally consistent (or diverse) compared other conditioning types.
20

---

Under review as a conference paper at ICLR 2020
Table 13: FJD / FID results averaged over runs on COCO-stuff validation set with multi-label, bounding box (bbox) and mask conditionings for image resolutions 64 x 64 and 128 x 128.
class conditioning
bbox conditioning
mask conditioning
FJD L
FID
FJD +
FID
FJD 4
FID
64
57.35 + 1.60
40.75 1.38
67.97 + 1.70
41.81 + 1.50
49.44 2.46
41.27 + 2.36
128
68.49 + 2.72
50.74 + 2.31
71.58 + 1.77
51.78 + 1.55
68.12 + 1.33
46.02 + 1.22
I.1
COCO-STUFF GAN ARCHITECTURE
In order modify BigGAN Brock al. (2019) work with multiple types of conditioning make two major changes. The first change occurs in the generator, where we replace the conditional batch normalization layers with SPADE (Park al. 2019). This substitution allows the generator to receive spatial conditioning such as bounding boxes or masks. In the case of class conditioning with spatially tiled class vector, SPADE behaves similarly to conditional batch normalization. The second change we make is to the discriminator. The original BigGAN implementation utilizes single projection layer (Miyato Koyama 2018) in order to provide class-conditional information to the discriminator. To extend this functionality to bounding box and mask conditioning, we add additional projection layers after each ResBlock in the discriminator, The input to each projection layer is downsampled version of the conditioning that has been resized using nearest neighbour interpolation to match the spatial resolution of each layer. In this way we provide conditioning information range of resolutions, allowing the discriminator to use whichever is most useful for the type of conditioning it has received. Aside from these specified changes, and using smaller batch sizes, models trained with the same hyperparameters and training scheme as specified in (Brock ctt al. 2019).
1.2 SAMPLES OF GENERATED IMAGES
In this section, we present some random 128 x 128 samples of conditional generation for the models covered in Section I In particular, Figures II 1 show class, bounding box, and mask conditioning samples, respectively. Each row displays a depiction of conditioning, followed by different samples, and finally the real image corresponding to the conditioning. As shown in Figure II on classes leads to variable samples w.r.t. object positions, scales and textures. As we increase the
conditioning
conditioning strength, reduce the freedom of the generation and hence, in Figure 12 we observe how the variability starts appearing in more subtle regions. Similarly, in Figure [3] taking different samples per conditioning only changes the textures. Although the degrees of variability decrease as the conditioning strength increases, we obtain sharper, better looking images.
21

---

Under review as a conference paper at ICLR 2020
Figure 11: Clndsicond Conditioning, samples, and ground truth image for label-conditioned
GAN. Greyscale intensity indicates class label.
Figure 12: Bounding box conditioning: Conditioning, samples, and ground truth image for bound- ing box-conditioned GAN.
22

---

Under review as a conference paper at ICLR 2020
Figure 13: Mask conditioning: Conditioning, samples, and ground truth image for mask-conditioned GAN.
23


Reviews:
[{"review_id": "rJxA3B9S5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a variant of the use of Frechet Inception Distance (FID) for the evaluation and benchmarking of conditional GAN models. FID is a popular measure for comparing image distributions in the Inception v3 feature space, in terms of the means and variances of multivariate Gaussians fit to data samples from each distribution. The authors argue that FID is ill-suited for use with cGANs, in that they do not explicitly take into account conditional consistency or intra-conditioning diversity. The main contribution and basic idea of this paper is to create joint image-conditioning distributions from the image embedding and the conditioning embedding in the Inception embedding, and then to combine them (by default, through vector concatenation). These joint image-conditioning distributions are then fed to FID as per standard usage on image distributions alone. The authors refer to their approach as FJD (Frechet Joint Distance).\n\nAlthough the authors propose FJD as a new technique, it should more properly be regarded as the direct use of FID on joint distributions. The main practical contribution of the paper thus reduces to the notion of concatenating the learned conditioning representation with the image representation. As a research contribution, this is in itself not very substantial. However, in their experimentation the authors do take care to show through examples the effect of their joint image-conditioning approach in assessing image quality, conditional consistency, and intra-conditioning diversity, under a variety of conditionings.\n\nThere are some issues that are not adequately addressed:\n\n1) In all experimental cases FJD scores and FID scores correlate well, which undercuts the argument that FJD is superior to FID in assessing the performance of cGAN models. Can situations be experimentally demonstrated where that is not the case? In particular, what happens when the fit in image representation is dramatically better / worse than that of the conditioning representation? This situation is interesting, but not considered in this paper.\n\n2) The effect of the dimensionality of the learned representations is not addressed. When concatenating vectors to produce a joint image-conditioning representation, the dimensionality increases, which would tend to produce larger FJD values than their corresponding FID values. The experimental results of this paper seem to confirm this. However, is\nit really valid then to declare FJD as being somehow more sensitive to the conditioning simply by virtue of obtaining larger values and larger spreads than FID? It should be remembered that FID and FJD are *not* unitless measures.\n\nOverall, in its current state the paper appears to be below the acceptance threshold.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgKpMenKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper mention that there are some critical drawbacks existing in IS (Inception Score) and FID (Fréchet Inception Distance) which are two popular metrics to measure image generation quality. However, IS and FID scores are initially designed for measuring unconditional distribution, which fails to capture the conditional consistency of conditional distribution. Thus, the authors propose to concatenate conditioned embedding h(y) with image feature vector f(x) to extend the FID metric. The authors also implement the method on a toy dataset to show the sensitivity of FJD on conditional consistency and several popular cGAN models to show the efficiency of FJD on real data.\nPaper Strengths\n  1. The method is intuitive and easy to implement.\n\nPaper Weaknesses\n1. Although this paper shows the problem of FID for capturing the conditional consistency sprightly with the toy dataset, however, this problem does not obviously show up on real data. Basically, FID can also give a good comparison of the different model as FJD", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJxo_VsaOH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper extends the Fréchet Inception distance (FID) to the conditional distribution. To this end, the authors use an additional embedding for the condition variables (class, image, text), and concatenate to the data embedding. The proposed metric, name the Fréchet joint distance (FJD), captures three desired properties of conditional generative models: sample quality, conditional consistency, and sample diversity. The authors demonstrate that the proposed metric indeed captures the properties using a synthetic (dSprite) dataset, and shows reasonable values for real datasets.\n\nPros:\n- FJD is an intuitive extension of FID for conditional generative models.\n- FJD can be applied to various types of conditions (e.g., image and text), which cannot be done by prior work (e.g., [1]).\n- The paper is easy to read and experimental details are clearly stated.\n\nCons:\n\n1. FJD is a straightforward extension of FID.\n\nFJD simply follows the FID formula but concatenates the condition embedding to the original data embedding. It is a straightforward extension of FID and suffers from the design choice problems due to the concatenation, as stated below.\n\n2. FJD requires many design choices and not theoretically justified.\n\nAs FJD requires an additional embedding function h, balancing parameter \\alpha, and merging function g, it raises a burden of design choices. While the authors give some suggestions, they are not theoretically justified. Also, one may use the statistical distances [2] between data distribution p_data(x,c) and model distribution p_g(x,c) to evaluate conditional generative models in a principled way, e.g., measure the KL-divergence using the density ratios [3]. The advantage of FJD over such metrics is unclear, as stated below.\n\n3. The advantage over the prior work is not clear.\n\nFJD and FID show the same trend in all reported experiments (Table 2, 3, 4), hence the advantage of FJD is unclear. Also, one may measure the FID score on conditional distributions, i.e., \\sum_c FID( p_data(x|c), p_g(x|c) ). It also captures the desired three properties and would be a strong baseline for FJD. Besides, while the authors aim to design a single metric to stand the models in a line, identifying the trade-offs of models may also be useful. For example, Improved PRD [4] provides the precision-recall trade-offs of generative models, which provides some insights for the models.\n\n\n[1] Ravuri and Vinyals. Classification Accuracy Score for Conditional Generative Models. NeurIPS 2019.\n[2] https://en.wikipedia.org/wiki/Statistical_distance\n[3] Uehara et al. Generative Adversarial Nets from a Density Ratio Estimation Perspective. arXiv 2016.\n[4] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an extension of FID for conditional generation settings. While it's an important problem to address, the reviewers were concerned about the novelty and advantage of the proposed method over the existing methods. The evaluation is reported on toy datasets, and the significance is limited.
Decision: Reject