Reproducibility Study of "ITI-GEN: Inclusive Text-to-Image Generation"
Authors: Daniel Gallo Fernández
Răzvan-Andrei Matișan
Alejandro Monroy Muñoz
Janusz Partyka
Venue: Accepted by TMLR ()

Abstract: Text-to-image generative models often present issues regarding fairness with respect to certain sensitive attributes, such as gender or skin tone. This study aims to reproduce the results presented in "ITI-GEN: Inclusive Text-to-Image Generation" by Zhang et al. (2023), which introduces a model to improve inclusiveness in these kinds of models. We show that most of the claims made by the authors about ITI-GEN hold: it improves the diversity and quality of generated images, it is scalable to different domains, it has plug-and-play capabilities, and it is efficient from a computational point of view. However, ITI-GEN sometimes uses undesired attributes as proxy features and it is unable to disentangle some pairs of (correlated) attributes such as gender and baldness. In addition, when the number of considered attributes increases, the training time grows exponentially and ITI-GEN struggles to generate inclusive images for all elements in the joint distribution. To solve these issues, we propose using Hard Prompt Search with negative prompting, a method that does not require training and that handles negation better than vanilla Hard Prompt Search. Nonetheless, Hard Prompt Search (with or without negative prompting) cannot be used for continuous attributes that are hard to express in natural language, an area where ITI-GEN excels as it is guided by images during training. Finally, we propose combining ITI-GEN and Hard Prompt Search with negative prompting.

Published in Transactions on Machine Learning Research (05/2024)
Reproducibility Study of "ITI-GEN: Inclusive Text-to-lmage Generation"
Daniel Gallo Fernández University of Amsterdam
Rázvan-Andrei Matişan University of Amsteriam
Alejandro Monroy Muñoz University of Amsterdam
Janusz Partyka
University of Amsterdam
Reviewed on OpenReview: /ttps:// opwrrevee......ee.om net/ forum? id- d3V)360W42
Abstract
Text-to-image generative models often present issues regarding fairness with respect to cer- tain sensitive attributes, such as gender or skin tone. This study aims to reproduce the re- sults presented in "ITI-GEN: Inclusive Text-to-Imagge Generation" by Zhang et al. (2023a), which introduces a model to improve inclusiveness in these kinds models. We show that most of the claims made by the authors about ITI-GEN hold: improves the diversity and scalable to different domains, has plug-and-play capabil-
quality of generated images, ities, and it is efficient from computational point of view. However, ITI-GEN sometimes uses undesired attributes as proxy features and it is unable to disentangle some pairs of (correlated) attributes such as gender and baldness. In addition, when the number of con- sidered altributes increases, the Training lime grows exponentially and ITI-GEN struggles to generate inclusive images for all elements in the joint distribution. To solve these issues, we propose using Hard Prompt Search with negative prompting, method that does not require training and that handles negation better than vanilla Hard Prompt Search. Nonetheless, Hard Prompt Search (with or without negative prompting) cannot be used for continuous attributes that are hard to express in natural language, area where ITI-GEN excels as it is guided by images during training. Finally, we propose combining ITI-GEN and Hard Prompt Search with negative prompting.
1 Introduction
Generative AI models that solve text-to-image tasks pose a series societal risks related to fairness. Some of them come from training data biases, where certain categorics are unevenly distributed. As a consequence, the model may ignore some of these categories when it generates images, which leads to societal biases on minority groups.
In order to tackle this issue, Zhang et al. (023a introduce Inclusive Text-to-Image Generation (ITI-GEN), method that generates inclusive tokens that can be appended to the text prompts. By concatenating these fair tokens to the text. prompts, they are able to generate diverse images with respect to a predefined set of attributes (e.g. gender, race, age). For example, we can add a "woman" 7 token to the text prompt "a hcadshot of a person" to ensure that the person the generated image is woman.
In this work, aim to focus

---

Published in Transactions on Machine Learning Research (05/2024)
[Reproducibility study] Reproducing the results from the original paper. We verify the claims illustrated in Section by reproducing some of the experiments of Zhang et al. 2023a
[Extended Work] Proxy features. Motivated by attributes for which ITI-GEN does not perform well, we carry out experiments to study the influence of diversity and entanglement in the reference image datasets.
Extended Work] Generating images using negative prompts. Hard Prompt Search (HPS) (Ding et al. 2021 with Stable Diffusion (Rombach et al. 2022 is used as baseline in the paper. We study the effect of adding negative prompts to Stable Diffusion as an alternative way of handling negations in nat urtural language and compare the results with ITI-GEN. We also highlight the potential of combining negative prompting with ITI-GEN, using each method for different types of attributes.
[Extended Work] Modifications to the original code. We improve the performance at infer- ence by fixing bug that prevented the of large batch sizes, integrate ITI-GEN with ControlNet "Zhang et al. 2023b), well provide the code to run our proposed method that handles negations. At the same time, we include bash scripts to make it easy to reproduce our experiments.
In the next section, we introduce the main claims made the original paper. Then, we describe the methodology of our study, highlighting the models and datasets that we used, as well the experimental setup and computational requirements. An analysis of the results and discussion about them will follow.
2 Scope of reproducibility
In the original paper, the authors make the following main claims:
1) Inclusive and high-quality generation. ITI-GEN improves inclusiveness while preserving image quality using a small number of reference images during training. The authors support this claim by using KL divergence and FID score (Heusel et al. 2017) metrics.
2) Scalability to different domains. ITI-GEN can learn inclusive prompts in different scenarios such as human faces and landscapes.
3) Plug-and-play capabilitics. Trained fair tokens can be used with other similar text prompts in a plug-and-play manner. Also, the tokens can be used in other text-to-image generative models such as ControlNet (Zhang al. 2023b
4) Data and computational efficiency. Only few dozen images per category are required, and training and inference last only a couple of minutes.
5) Scalability multiple attributes. ITI-GEN obtains great results when used with different attributes the same time.
In this work, we run experiments to check the authors' statements above. Additionally, we study some failure cases of ITI-GEN and propose other methods that handle negations in natural language.
3 Methodology
The authors provide an open-source implementation on GitHut that we have used as the starting point. To make our experiments completely reproducible, we design series of bash scripts for launching them. Finally, since the authors did not provide any code for integrating ITI-GEN with ControlNet (Zhang et al. 2023b), we implement it ourselves to check the compatibility of ITI-GEN with other generative models. All of these are better detailed in Section 3.4
https://github. http:/hhww...cmmmmmmmmmnnnnnna//
2

---

Published in Transactions on Machine Learning Research (05/2024)
3.1 Model description
ITI-GEN is method that imppoves inclusiveness in texl-to-image generation. It outpuls set of fair tokens that are appended to text prompt in order to guide the model to generate images in certain way. It achieves this by using reference images for each attribute category. For example, if we use the prompt "a headshot of a person" and provide images of men and women, the model will learn two tokens, one for each gender.
Given a of M attributes where each abtribute has Km different attribute categories, ITI-GEN learns set of fair tokens that represent each attribute category. For each combination, the corresponding fair tokens are aggregated and concatenated to the original text prompt T to build a inclusive prompt P. We denote the set of all inclusive prompts with P, and the set of all inclusive prompts that correspond to category i of altribute m with P!
The training procedure involves two losses. The first one is the directional alignment loss, which relates the provided reference images to the inclusive prompts. In order to compare images and text, CLIP (Radford et al. 2021) used to map them to common embedding space using ils text encoder Exext and image encoder Eing- Following the original code, the ViT-L/ 14 pre-trained model is used. The directional alignment loss is defined by
Eair Σ Σ 1- (AT (A).
where (1. is the difference the average of image embeddings between categorics and j of attributo m, and A (i,j) is the difference between the average of embeddings of inclusive prompts correspondent those categories.
This loss is replaced by cosine similarity loss Less when it is undefined (i.e. when the batches are not diverse enough and do not contain samples for every altribute category). Given an image of a certain attribute category, we want to make it similar to all inclusive prompts that contain that attribute category.
The second loss is the semantic consistency loss, which acts as a regularizer by making the original text prompts similar to the inclusive prompts:
M
Deem =
Σ Σ m=1 Kiiiisks
max (0, A - (Frext (P), Frext (T))),
(2)
PEP"UPP
where T is the raw text prompt and P the inclusive text prompt for attribute category combination including category or j for attribute m and X is a hyperparameter.
In this way, the total loss for a batch is defined by:
= Ldir + Lyem Cair is defined, Loos Leom otherwise.
Ciotal
(3)
We invite the reader to check the original paper Zhang et al. 2023a) for more detailed explanation of the model.
At inference, the output of the generative model will reflect the attribute categories of the fair tokens. In this way, sampling over uniform distribution over all combinations leads to fair generation with respect to the attributes of interest. In addition, the text prompt can be the same that was used for learning (in-domain generation) or different. (Irain-once-for-all)
The generative model must be compatible with the encoder (CLIP in this case). Following the original paper, we use Stable Diffusion v1.4 (Rombach et al. 2022) for most of the experiments. We also show compatibility with models using additional conditions like ControlNet. (Zhang et al. 2023b) in plug-and-play manner.
3

---

Published in Transactions on Machine Learning Research (05/2024)
An alternative to ITI-GEN is HPS (Ding al. 2021 which works by specifying the categories directly in the original prompt. For example, we could consider the prompt "a headshot of woman" to generate a woman. One problem with this approach is that it does not handle negation properly and using a prompt like "a headshot of a person without glasses" will actually increase the chances of getting somcone with glasses. However, this can be circumvented by using the unconditional condilioning parameter, which is hard-coded to take in the empty string in the current Stable Diffusion sampler (PLMS). By providing the features that are not wanted we can prevent them from appearing in the generated images. For example, we pass "eyeglasses" to get headshot of someone without them.
A denoising step in Stable Diffusion takes an image æ, timestep and a conditioning vector This outputs another image that we will denote by f(x, (, c). The technique involves two conditioning vectors, e and č (the negative prompt), and outputs
-
- /(x,1,)) + (ax,,,
where is the scale, that we set the default 7.5 as we explain later in Section 3.3
We call this Hard Prompt Search with negative prompting and we will refer to it as HPSn throughout the paper.
3.2 Datasets
The authors provide four datascts reference images to train the model:
CelebA Liu et al.y 2015 a manually-labeled face dataset with 40 binary attributes. For each attribute, there are 200 positive and negative samples (400 in total).
FAIR (Feng et al. 2022). a synthetic face dataset classified into six skin tone levels. There are 703 almost equally distributed images among the six categories.
FairFace (Karkkainen & Joo 2021 face dataset that contains annotations for age (9 intervals), gender (male or female) and race (7 categories). For every attribute, there are 200 images per category.
Landscapes HQ (LHQ) (Skorokhodov et al. 2021 ,, dataset of natural scene images annotated using the tool provided in Wang et al. (2023). There are 11 different attributes, cach of them divided into five ordinal categories with 400 samples per category.
3.3 Hyperparameters
In order to reproduce the results of the original paper as closely as possible, we use the default train- ing hyperparameters from the code provided. In a similar way, for image generation, we use the default hyperparameters from the ControlNet (Zhang et al.) 2023b) and Stable Diffusion (Rombach et al. 2022) repositories (for HPS and HPSn). Morcover, we generate images with batch size of 8, which is the largest power of two that can fit. in NVIDIA A100 with 40 GB of VRAM.
3.4 Experimental setup and code
The code used to replicate and extend the experiments can be found in our GitHub repository We made two minor changes the authors' implementation: adding seed in the training loop of ITI-GEN to make it reproducible and fix bug in the script for image generation to handle batch sizes larger than 1. We also provide bash scripts to replicate all our experiments easily.
Moreover, the authors do not provide any code for combining ITI-GEN with ControlNet (Zhang et al, 2023b). Thus, we implement it ourselves on top the pre-existing ControlNet code in order to validate the plug-and-llay capabilities of ITT-GEN with other texl-to-image generation methods. More specifically,

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Inclusiveness with respect to single attributes. KL divergences between the obtained and the uniform distributions over the attribute category combinations. Reference images are from CelebA. To classify the images, CLIP and manual labeling are used. The text prompt is "a headshot of a person". An extended version of these results can be found in Appendix A
Method
Male
Young
Pale Skin
Eyeglasses
Mustache
Smiling
HPS
CLIP
0.000000
0.000000
0.000416
0.387506
0.158797
0.000046
Reported
0.000010
0.027000
0.002800
0.371000
0.241000
0.004400
ITI-GEN
CLIP
0.000000
0.026869
0.001156
0.015053
0.280577
0.000000
Manually labeled
0.000000
0.001156
0.000185
0.000000
0.000000
0.000000
Reported
0.000002
0.000200
0.000000
0.000200
0.000450
0.002500
we use the conditioning parameter of the DDIM sampler (which is meant to be initialized with the text prompl in the original implementation) to inject ITI-GEN's inclusive prompt. At the same time, we used the unconditional_ condiiiioonnnnngggggogo parameter of the sampler to guide the model not to generate, for example, black-and-white or sepia images. We run experiments using depth, canny edge, and human pose as addi- tional conditions for ControlNet. For more information about how to use the integration of ITI-GEN with ControlNet, invite the reader to check our GitHub repository
For reproducing the experiments using HPS (Ding et al. 2021 we use Stable Diffusion's 's code D We modify the text-to-image generation script to receive an optional negative prompt for HPSn.
We consider two aspects when we evaluate the results: fairness and image quality. To measure fairness, we first generate images for every category combination. Then we classify all images using CLIP (Cho et al. 2023 Chuang al.] 2023). As explained in the original paper, this turned out to be unreliable (even when not using negation), so we manually labeled the images for some attributes. Finally, compare the obtained distribution with uniform distribution with respect to all attribute categories of interest by calculating the Kullback-Leilh (KL) divergence. For quantifying the image qualily, computed the Fréchet Inception Distance (FID) score (Heusel et al., 2017 Parmar et al. 2022), which compares the distribution of generated images with the distribution of a set of real imagos. The score depends on the number of images involved so, for reproducibility reasons; we used the nunber of generated images mentioned by the authors (i.e. approximately 5,000). When we compare two methods (e.g. ITI-GEN and HPS), we also perform human evaluations by displaying images from the same attribute category combination for each method side by side, making the human subject choose the one with higher quality. We used 714 images for HPS, HPSn and ITI-GEN, respectively.
3.5 Computational requirements
We perform all experiments on an NVIDIA A100 GPU. Training a single attribute for 30 epochs takes around a minute for CelebA, 3 minutes for LHQ, 4 minutes for FAIR and less than 5 minutes for FairFace. For the four datasets, we use 200 images per category (or all of them if there are less than 200). Generating batch of 8 images takes around 21 seconds (less than 3 seconds per image). It is also possible to run inference Apple M2 chip, although it takes more than 30 seconds per image. In total, our reproducibility study adds up to at most 20 GPU hours.
All our experiments were run using the Snellius infrastructure located in the Amsterdam Data Tower. The data center reports a Power Usage Effectiveness (PUE) of 1.19 Thus, using the Machine Learning Emissions Calculator [Lacoste et al. 2019), we obtain that our experiments emitted around kg of CO2.
5

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Image generation quality, FID scores for CelebA dataset (lower is better). As reference dataset, we use the Flickr-Face-Bl Dataset (FFHQ) (Karras al. 2019 We set different seeds generate images, which might explain why we get a better score (the authors do not report their generation method).
FID score Reported
Method
Number of images
Ours
Vanilla Stable Diffusion
5, 040
78.34
67.40
HPS
5, 5,040
70.55
HPS (with negative prompting)
5, 040
65.08
ITI-GEN (30 epochs)
0.0
54.86
60.38
4 Results
Our reproducibility study reveals that the first four claims mentioned in Section 2 are correct, whereas the last one does not hold when the number of attributes increases. In this section, first highlight the results reproduced to support or deny the main claims of the authors. Then, we compare HPSn with ITI-GEN for the cases when ITI-GEN struggles to generate accurate images.
4.1 Results reproducing original paper
4.1.1 Inclusive and high-quality generation
To verify the first claim, we train ITI-GEN on every attribute of the CelebA dataset and generate 104 images (13 batches of size 8) per category. Considering there are 40 binary attributes, that makes a total of 40 x 2 x 104 = 8, 320 images.
For every attribute we have 104 x = 208 images that are classified using CLIP. Since this classification is not very reliable, we also label the images of selected attributes manually. After that, we compute the KL divergence. In Table we see how HPS struggles with attributes that require negation (e.g., "a headshot of person with no eyeglasses"), while ITI-GEN performs well.
Since the FID score reported in the paper uses around 5, 5,000 images, and we have 8, 320, we decide to compute the FID score using the last 63 images of every category, which results in a total of 5, 040 images. The results are in Table 2 where we can see that ITI-GEN obtains the best score.
We also carry out human evaluations to compare the quality of the images generated by HPS and HPSn with ITI-GEN's. The results show that, in 52.24 % of the cases, humans prefer images generated with HPS over ITI-GEN. The same thing happens with HPSn in 56.58% of the cases. This shows discrepancy with the interpretation of the FID scores, where ITI-GEN shows slightly better quality than the HPS methods. A possible explanation is that the reference datasets for the computation of the FID score and for training ITI-GEN (FFHQ and CclobA respectively) might be similar, which would imply a bias towards ITI-GEN.
4.1.2 Scalability to different domains
Figure D shows that it is possible to apply ITI-GEN multiple domains, as the second claim holds. For the human faces generated with the "Skin tone" altribule, we are able to compute the FID score in the same way before (using the FFHQ dataset), and obtain 46.177. For the natural scenes generated with the "Colorful" attribute, we compute the FID score by comparing the generated images with the training ones, obtaining 62.987. Note that both figures are within a similar range as the ones in Table 2
4.1.3 Plug-and-pin capabilities
We perform a number of experiments to verify the third claim, demonstrating the end that holds. More specifically, we show that inclusive tokens learned with one prompt can be applied to other (similar) prompts without retraining the model, it is shown in Figure We train the binary "Age" and "Gender"

---

Published in Transactions on Machine Learning Research (05/2024)
Skin tone
Colorfulness
(a) "Skin tone" applicd to "a headshot of person".
(b) "Colorful" applied to "a natural scene".
Figure 1: Scalability to different domains. Images generated with ITI-GEN in two different domains:
human faces and natural scenes. Each column corresponds to a different category of the attribute.
Young
Man
Woman
Old
DKL 0.000800
DKL. 0.000000
"Age" token applicd "a headshot of firefighter".
(b) "Gender" token applied "a headshot of doctor".
Figure 2: Plug-and-play capabilities. ITI-GEN is used to learn inclusive tokens for " "Age" and "Gender" using the text prompt "a headshot of a person". These tokens are then applied to other similar text prompts.
tokens with the prompt "a headshot of person" using CelebA dataset and apply them to "a headshot of firefighter" and "a headshot of doctor" prompts. However, while the generated images are still diverse, their quality diminished. Thus, we obtain slightly higher FID scores, i... 152.94 and 157.6 for the firefighter and doctor examples, respectively.
In addition, we run experiments to illustrate the compatibility of ITI-GEN with ControlNet (Zhang el al. 2023b) in a and-play manner. Figure depicts the desired behaviour: ITI-GEN improves diversity of ControlNet with respect to the inclusive tokens used. For this experiment, train the "Age" token with "a headshot of person" prompt using FairFace dataset, which has 9 categories for this attribute, and apply it to "a headshot of a famous woman" prompt.
However, we can observe that the newly introduced fair tokens may entangle other biases that are inherent from the reference images such as hair color, clothing style or skin tone. This is an observation the original authors also make but they did not try to find an explanation for it stating that the "disentanglement of attributes is not the primary concern of this study". Therefore, do believe ITI-GEN improves diversity of ControlNet, but only with respect to the attributes of interest, though it may bring some other biases taken from the reference images. For additional results, please refer to Appendix B
4.1.4 Data and computational efficiency
In order to verify the claim about data elliciency, we generate images for the altributes "Colorful", "Age", and "Skin tone" using different number of reference images per category, As in the original paper, we find ITI-GEN be robust in the low-data regime. The reader can refer Appendix D for more information. Regarding computational efficiency, training takes less than 5 minutes and generation takes around 2 seconds per image hardware, we show in Section 3.5

---

Published in Transactions on Machine Learning Research (05/2024)
Age
Figure Compatibility with ControlNet. We generate images using the prompt "photo of a famous woman" and human pose (left) as additional condition. The attribute of interest is * "Age", which is trained using the text prompt "a headshot of person".
4.1.5 Scalability to multiple attributes
Table iillustrates how ITI-GEN performs for some combinations of binary altributes. We can observe that while the KL divergence low when we use two attributes, it increases significantly when we keep adding attributes. Thus, this claim is not entirely correct, as we further investigate this observation qualitatively and quantitatively in the next section, where we also highlight more of ITI-GEN's pitfalls.
Table 3: Inclusiveness with respect to multiple attributes. Comparison of the KT. divergence between the obtained and the uniform distributions. All reference images are from CelebA datasct. The text prompt is 'a headshot of a person'. We use CLIP classify images. We also include the results for HPS with
negative prompting, which we further discuss in Section 4.2
Method
Male x Young
Male x Young x Eyeglass
Male x Young x Eyeglass x Smile
HPS
CLIP
0.000990
0.125403
0.032695
Authors
0.003500
0.399000
0.476000
HPSn
CLIP
0.000990
0.000475
0.000273
ITI-GEN
CLIP
0.051310
0.378709
0.356023
Authors
0.000130
0.061000
0.094000
4.2
Results beyond original paper
4.2.1 Proxy features
ITI-GEN might use some undesired features as proxy while learning the inclusive prompts for certain attributes. This is the case for attributes like "Bald", "Mustache" and "Beard", which seem to use "Gender" proxy. One could argue that this is not important as long the model is able to generate people with and without the desired attribute. However, it is actually a problem because as show in Figure 4 some pairs of attributes are strongly coupled and ITI-GEN is not able to generate accurate images for all elements in the joint distribution. HPSn, on the other hand, is able to disentangle the attributes. More examples are shown in Appendix C
Our main hypothesis that the reason for this lies within the reference images. If we inspoct them, we see that most of the bald people are men, which may be the reason why the model using gender as proxy. To delve into this, perform experiment using two variants of the "Eyeglasses" dataset (see Table 4 which is diverse and works fine when combined with the "Gender" attribute. Then, we test the ability of the model to learn the joint distribution using those reference images.
Figure Figure balllussrates with quantitative and qualitative results how ITI- ITI-GEN correctly handles the combination of the "Gender" and "Eyeglassos" attributes using the original reference datasct. If we alter the diversity of the dataset by entangling both attributes, the model fails learn the "Eyeglasses" attribute and uses "Gender" proxy feature, shown in Figure 5b (the generated samples of women with eyeglasses
8

---

Published in Transactions on Machine Learning Research (05/2024)
Not boild Bald
Nos bald Bold
Noi batal Bali
Man
Woman
Man
Woman
DEL. 0.000178
DeL. 0.401088
DKt. 0.009371
(a) ITI-GEN on "Bald" attribute.
(b) ITI-GEN on "Gender" x "Bald".
(c) HPSn on "Gender" x "Bald".
Figure 4: Proxy features are used by the model for certain attributes. (a) Generated samples using the fair tokens for "Bald". All positive samples men, whereas all negative samples women, which indicates that "Gender" might be used altribules, ITI-GEN fails to generale samples of bald women. (c) HPS with negative prompling is able to accurately generate bald women. In (b) and (c), the KL divergence is computed using 104 manually labeled samples.
proxy feature. (b) When combining the "Gender" and "Bald"
Table 4: Non-diverse "Eyeglasses" reference datasets. (a) is the original dataset form CelebA. In (b), the negative samples are only women, and, the positive ones, of men. In (c), only men are included.
Negative
Positive
Men without eyeglasses
Men with eyeglasses Women with eyeglasses
(a) Original
Women without eyeglasses
(b) Male-only
Women without eyeglasses
Men with eyeglasses
Men without eyeglasses
Men with eyeglasses
just men). However, if only men included in the reference images, the method does learn the feature correctly (Figure 5cb, because the only difference between the negative and positive samples are the glasses, and not the gender. This comparison demonstrates that we must be careful choosing the reference images, as the model might not learn what we expect, but the most salient difference between the datasets.
4.2.2 Handling multiple attributes
Computational complexity. The training loop iterates through all reference images, which implies a linear complexity with respect to the size of the training set. At the same time, it considers all category pairs within an attribute, which mcans it is quadratic with respect to that. It also iterates through all elements in the joint distribution, so it is exponential with respect to the number of attributes. This can problematic when we train ITI-GEN on many attributes. For example, in the case of binary attributes with the same number of reference images, the time complexity is O(Ne Figure S depicts.
Diversity issues in generated images. Figure 0 illustrates comparison between ITI-GEN and HPSn. We can observe that ITI-GEN struggles significantly to generate diverse images (the difference between old and young people is almost non-existent, there are many people without eyeglasses, etc.). On the other hand, images generated by HPSn reliably reflect all category combinations.
This behaviour can also be observed our quantitative results. More specifically, we compute the KT. divergence using 104 images in every category combination. The numerical results are displayed in Table 3 which can observe that ITI-GEN obtains KL divergence of approximately 0.3560, while HPSn's is almost equal to 0.
4.2.3 Combining ITI-Gen with HPSn
The main motivations for imago-guided prompting are handling negations and continuous attributes that are hard to specify with text. As have shown, negation can be effectively tackled using HPSn, but ITI-GEN
9

---

Published in Transactions on Machine Learning Research (05/2024)
Man
Man
Man
Wiman
Womma
Woman
Exeglasses
No eyeglasses
Eyeglasses
ryeg lagglssss
Eyeglasses
No eyeglasses
DKL 0.034039
DRL 0.237189
DKL 0.001636
(a) Original "Eyeglasses" dataset.
(b) Gender-blased " "Fyeglasses" dataset.
Male-only "Fyeglasses" dataset.
Figure 5: Ablation study on the diversity of reference datasets. Generated images for all category combinations of the "Gender" and "Eyeglasscs" attributes for all variations of the "Eyoglasscs" reference datasets introduced Table 0 The complete reference dataset for the "Gender" attribute used in the three cases. In the three cases, the KL divergence (DKL) is computed using 104 manually labeled samples
With eyeglasses
Without eyeglasses
Young
Oul
Young
Woman
Oli
Smiling
Not smiling
Smiling
Not smiling
ITI-GEN.
eyeglasses
Without eynglassee
Young
OlL
Young
Woman
Old
Smiling
Not smiling
Smiling
Not smiling
(b) HPS with negative prompting.
Figure 6: Qualitative results for multiple attributes. ITI-GEN and HPSn are compared on generating diverse images with respect to four binary attributes.
is still needed for continuous attributes. A natural question is whether we can combine both methods, and indeed, such integration is feasible.
This can achieved by appending ITI-GEN'S inclusive token after the (positive) prompt in HPSn. this way, we can just train ITI-GEN on the attributes that are hard to specify with text and leverage the training-free HPSn method for the others. In the experiment described in Figure we get a KL divergence close to zero, and FID score comparable to ITI-GRN's.
10

---

Published in Transactions on Machine Learning Research (05/2024)
With Without
With Without glusess
glasses
Age
Skin tone
DKL 0.000167 FID 57.07
DKL 0.000015 FID 49.92
(a) Glasses (HPSS) and Age (TTI-GEN)
(b) Classes (HPSS) and skin tone (1T1-GEX)
Figure 7: Combining ITI-Gen with HPSn. ITI-GEN's fair tokens used for the Age and Skin tone attributes, while the category of the Eyeglasses attribute is hard-prompted in the Stable Diffusion argunents.
5
Discussion
In this work, we conduct several experiments to check the validity of the main claims from the original paper. We find that most of them correct with small exceptions.
First, we show ITI-GEN (Zhang et al. 2023a is able to generate diverse high-quality images, according to the low KL divergence and FID scores we obtained (see Tables and 2 We also highlight that it works on multiple domains, such as human faces and landscapes. Moreover, ITI-GEN has plug-and-play capabilities when learned inclusive tokens are applied to similar text prompts (Figure 2). The fair tokens can also be integrated with different text-to-image generators (i.e. Stable Diffusion (Rombach al. 2022), ControlNet (Zhang et al. 2023b)), ) as displayed in Figures images required, and that training is computationally efficient (Figure 11
3 In addition, verify that only a few dozen reference
On the other hand, we show that the claim about scalability to multiple attributes does not completely hold. More specifically, our analysis reveals that ITI-GEN needs exponential training time with respect to the number attributes (Figure S and struggles to generate diverse images when we increase the number of attributes (Figure G). Moreover, we illustrate that ITI-GEN can struggle disentangle certain attributes (e.g. "Gender" and "Bald" in Figure + To solve these issues, we propose HPSn which does not require training and produces more accurate images. At the same time, it seems to handle negations better than ITI-GEN, especially when
large number of attributes (Figure 6 Table [h. However, limitation of HPSn is that it cannot handle attributes that are difficult to specify with text (e.g. skin tone, colorfulness), whereas ITI-GEN excels on these. As we show in Figure ITI-GEN and HPSn can be combined to generate images that are inclusive with respect to multiple attributes, taking advantage the strong aspects of each method.
5.1 What was easy
The paper was well-written and included many examples to make it easier to understand. Additionally, the original code was published on GitHub, with instructions on how to run the training, generation and evaluation scripts.
5.2
What was difficult
The main difficulty consisted in understanding the code, since there were some differences with the paper. More specifically, when the directional loss is undefined, it is replaced by a cosine similarity loss between the image and text embeddings.
Moreover, the evaluation script requires a list classes that is not specified. The authors mention that they had to change the text prompts to tackle the negative prompt issue. They also had use pre-trained classifiers combined with human evaluations. Thus, it was not possible for to casily reproduce the KL divergence results.
11

---

Published in Transactions on Machine Learning Research (05/2024)
1430
Empirical exeration time
SeV +
19.0
10:0
800
230-
Number allriintes (N)
Figure 8: Computational complexity. Average training time across 3 executions of ITI-GEN with respect to the number of binary attributes. 400 reference images are used for every attribute, equally distributed between the positive and negative categories.
5.3 Communication with original authors
We reached out to the authors via email correspondence regarding the quality of the generated images and the missing code for combining TEIGGEN with ControlNet. They replied quickly and cleared up the discrepancies in our results. However, we did not receive from the authors any additional code for integrating ITI-GEN with NontrolNet, because they were still working on it. They assured us they would upload the code on their original GitHub repository as soon as possible, so that we can cross-verify our implementations.
6 Ethical and Social Impact
As ITI-GEN is guided visually, we need a set of reference images. This raises some concerns regarding inclusiveness and privacy.
Even though the output images are usually inclusive with respect to the attributes of interest, they might not be with respect. to others. This be problematie, because someone aiming to increase diversity could actually be introducing other biases without rcalizing it. Thus, it is important to ensure that the reference images are not biased, and conduct some sort of quality evaluation on the outputs.
Another important aspect is data protection. If ITI-GEN is trained on publicly available dataset, we need to have permission to do so. Inspired by Carlini et al. 2023), we wonder if an attacker could recover the training images using the inclusive lokens, which would be problematic ITI-GEN was trained on private dataset.
All in all, agree with the original authors' that carefully considering potential risks of ITI-GEN is essential to avoid possible negative consequences.
Acknowledgments
We would like to thank Carlo Bretti from the University of Amsterdam for his useful feedback and guidance during the process of writing the paper.
12

---

Published in Transactions on Machine Learning Research (05/2024)
References
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models, 2023.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
Ching-Yar Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision- language models via biased prompts. ar preprint arXiv:2302.000 00070, 2023.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogviow: Mastering text-to-image generation via transformers. Advances in Newral Information Processing Systems, 2021.
Haiwen Feng, Timo Bolkart, Joachim Tesch, Michael J Black, and Victoria Abrevaya. Towards racially unbiased skin tone estimation via scene disambiguation. In ECCV, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. In WACV, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. CoRR, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. In ICCV, 2021.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
Cheng Zhang, Xuanbai Chen, Siqi Chai, Henry Chen Wu, Dmitry Lagun, Thabo Beeler, and Fernando De la Torre. ITI-GEN: Inclusive text-to-image generation. In ICCV, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, 2023b.
13

---

Published in Transactions on Machine Learning Research (05/2024)
A KL Divergences on the CelebA dataset
Table shows the performance on diversity of ITI-GEN and the baseline methods on binary altributes.
Table 5: Inclusiveness with respect to single attributes. KL Divergence for all attributes from CelebA dataset. We include results for ITI-GEN trained for 10, 20 and 30 epochs. The generated images (208 per attribute) were classified using CLIP, and, for some attributes, also manually (last column).
ITI-GEN
Attribute
SD
HPS
HPSu
10 epocla
20 epochs
30 epochs
30 (man.)
5 o'Clock Shadow
0.007833
0.076791
0.016782
0.000185
0.007833
0.008748
Arched Eyebrows
0.355570
D..24993
0.011881
0,101287
0.072061
0.083041
Attractive
0.072061
0.001156
0.036694
0,158797
0,130812
0.203595
Bags Under Eyes
0.325839
0. 0.125592
0.001156
0.255716
0.120493
0.034089
Bald
0.530121
0.048114
0.000000
0.022202
0.068321
0.101287
0.000178
Bangs
0.263821
0.263821
0.000000
0.000416
0.002962
0.003749
Big Lips
0, 1587779
0.064693
0.064693
0.001156
0.000416
0.002267
0.008915
Big Nose
0.355570
0.01 3420
0.013420
0.280577
0.316377
0.272110
0.201355
Black Hair
0.316377
0.110655
0.002962
0,196770
0.263824
0.232416
0.020136
Blond Hair
0. 6.639221
0.020527
0.001665
0.007833
0.006672
0.007833
Blurry
0.01 6782
0.217696
0.022544
0.036694
0.020527
0.031584
Brown Hair
0.4219588
O. 0.158797
0.003749
0.272110
0.224977
7.21
Bushy Eyebrows
0. 0.105913
0.376598
0.000046
0.002962
0.016782
0.011881
Chubby
0.298080
0.514949
0.000416
0.115515
0,164783
0.183554
Double Chin
0.272110
0.446529
0.002267
0.031584
0.026869
0.001665
Eyeglasses
0.398692
0.387506
0.000000
0.016782
0.020527
0.015053
0.000000
Goalee
0.345427
0.064693
0.010438
0.170903
0.158797
0.170903
Gray Hair
0.693147
0.365957
0.000046
0.079859
0.088094
0.092379
0.035989
Heavy Makeup
0.434070
0.136156
0.000185
0.096776
0.064693
0.045107
0. 01132232
High Cheekbones
0.472577
0.003749
0.096776
0.434070
0.335520
0.355570
Male
0.000740
0.000000
0.000000
0.000000
0.000000
0.000000
0,000000
Mouth Slightly Open
0.083921
0.1
0.110655
0.007833
0.000046
0.000185
Mustache
0.5459255
0. 0.158797
0.000000
0.280577
0.280577
0.280577
0.000000
Narrow Eyes
0.136156
0.051223
0.026869
0.026869
0.000000
0.006672
0,000000
No Beard
0.232416
0.500334
0.693147
0.013420
0.081584
0.042202
0.000000
Oval Face
0.376598
0.010438
0.170903
0.240016
0.101287
0.1 0.170903
Pale Skin
0,196770
0.000416
0.000416
0,004630
0.006672
0.001156
0.000185
Pointy Nose
0.500334
0.061169
0.164783
0.545925
0.562439
0.500334
Receding Hairline
0.410171
0.5149499
0.000740
0.136156
0.141624
0.141624
Rosy Cheeks
0.486221
0.051223
0.024658
0.315127
0.345-127
0.121958
Sideburns
0.514949
0.026869
0.034089
0.387506
0.410171
0.562439
Smiling
0.280577
0.000046
0.000046
0.000000
0.000000
0.000000
Straight Hair
0.057750
0.662690
0.000000
0. 0.002267
0.000185
0.002267
Wavy Hair
0.446529
0.240016
0.000185
0.115515
0.072061
0.064693
Wearing Earrings
0.355570
0.472577
0.000416
0.398692
0.446529
0.421958
Wearing Hat
0.662690
0.545925
0.000046
0.042202
0.136156
0.141624
Wearing Lipstick
0.579782
0.500334
0.000046
0.247782
0.307126
0.325839
Wearing Necklace
0.662690
0.307126
0.001156
0.355570
0.355570
0.325839
Wearing Necktie
0.514949
0.545925
0.001156
0.009088
0.001665
0.002267
Young
0.693147
0.000000
0.000000
0.000046
0.001665
0.026869
0.001156
B Compatibility with ControlNet
We present additional results on the compatibility of ITI-GEN with ControlNet (Zhang et al.) 2023b) in
Figure g
14

---

Published in Transactions on Machine Learning Research (05/2024)
Age
(a) Condition: depth map. Prompt: "a headshot of female",
Skin tone
(b) Condition: depth map. Prompt: "photo of famous woman".
Skin tonc
(c) Condition: human pose map. Prompt: "photo famous woman".
Colorfulness
(d) Condition: canny edge map. Prompt: "photograph of mount katahdin",
Figure 9: Compatibility with ControlNet. ITI-GEN is able to improve diversity in images generated with ControlNet with respect to the attributes of interest given different conditions. Human images (a), (b), (c) are generated using inclusive tokens trained on "a headshot of a person", while scene images (d) are generated using inclusive tokens trained on "a natural scene".
C Proxy features
We provide additional examples of proxy fcatures used by ITI-GEN in Figure 10
Low data requirements
Figure illustrates how ITI-GEN is able to improve diversity using only a few reference images.
15

---

Published in Transactions on Machine Learning Research (05/2024)
No beanll Board
No beard Beard
No Board
MED
Woman
Man
WomDnD
DRL. 0.000000
DKL. 0.288772
DkL. 0.113676
(al) ITI-GEN on "Beard" attribute.
(a2) ITI-CEN on "Gender" x "Baadd"
(a3) HPSn on "Gender" "Beard"
No must. Mist.
No must. Must.
No mist Muss..
Man
Woman
Man
Woman
DRL. 0.000000
DRL. = 0.449763
DKL = 0.187124
(b1) ITI-GEN "Mustache" attribute.
(b2) ITI-GEN "Gender" x "Mustache",
(b3) HPSn "Gender" "eander"""Munne "Mustache",
Figure 10: Proxy features are used by the model for certain attributes. In (al) and (b1) we see how ITI-GEN seems to be using "Gender" as proxy for "Beard" and "Mustache". In (a2) and (b2) we see how ITI-GEN fails to disentangle the attributes. In (a3) and (b3) we see HPSn's results. It struggles sometimes generate women with beard, but works well with the combination of "Gender" and "Mustache". In (b) and (c), the KL Divergence computed using 104 manually labeled samples.
16

---

Published in Transactions on Machine Learning Research (05/2024)
25
50
Age
(a) Prompt: "a headshot of a person".
25
50
Skin tone
(b) Prompt: a headshot of a person".
50
Colorfulness
(c) Prompt: "a natural soene".
Figure 11: Effect of of the size of the reference dataset. ITI-GEN is able improve diversity with respect to attributes such "Age", "Skin tone" and "Colorfulness" using only few reference images (10, 25 and 50).
17


Reviews:
[{"review_id": "1zAkaBHef8", "reviewer": "Reviewer_j9Wj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper conducts a lot of experiments related to the reproduction of “ITI-GEN: Inclusive Text-to-Image Generation”\n- The paper is clear with figures and tables\n- The paper proposes to use Hard Prompt Search with negative prompting, which does not require training and is more effective at handling negation than the standard Hard Prompt Search.\n\n\nWeakness:\n- The paper on reproducibility study lacks novelty as it primarily focuses on replicating experiments from a previous work without introducing any new methods, insights, or advancements to the field. As such, its contribution to the scientific community is limited, and it does not significantly add to the existing body of knowledge.\n- While I recognize the importance of reproducibility studies in validating previous research findings, the novelty of such studies lies in providing additional insights or uncovering discrepancies that significantly impact the original findings or methodology. Unfortunately, upon reviewing your manuscript, it appears that the findings do not substantially contribute to the existing body of knowledge or challenge previous results in a meaningful way.\n- Although the study is well-executed and adheres to rigorous methodology, it lacks innovation, new perspective, more insights, interpretations, or advancements that enhance our understanding of the subject matter.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a reproducibility study on another paper: “ITI-GEN: Inclusive Text-to-Image Generation\", it also has the following contributions:\n- Investigated the impact of diversity and entanglement in reference image datasets, particularly where ITI-Gen struggled.\n- Explored the use of negative prompts in image generation, comparing it with existing methods.\n- Enhanced model performance by fixing bugs, integrated with other models, and provided code for easy replication of experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The authors may need to convince us the novelty and importance of this reproductivity study", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9YTM7JEI6m", "reviewer": "Reviewer_Fur8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n+ This paper is well-written and easy to follow.\n+ They successfully reproduce the training, evaluation, and results. Apart from that, they also point out some potential issues, which are valuable for this research direction.\n+ They provide comprehensive quantitative analysis as well as qualitative comparison.\n+ They have open-sourced their reproduced code base.\n\n**Weakness**\n+ As they mention the training cost, the paper should contain learning curves for each dataset to indicate this issue. Also, the exact learning record (both training loss and validation performance) is crucial and helpful for this kind of rerpoduce report.\n+ The integration with ControlNet seems not that well. For example, in Fig. 3&8a, the girls change too much rather than just age. Furthermore, different attributes should be considered and demonstrated to support this integration.\n+ Since this tackles a real-world application/issue, it will be better to have a human evaluation using the reproduced results to investigate human preference in various aspects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to reproduce \"Inclusive Text-to-Image Generation\", which presents ITI-Gen to improve inclusiveness in text-to-image (T2I). From the experiments, most of the claims hold, including the improvement in quality as well as diversity, the scalability to different domains, and its plug-and-play capability. They also highlight the issue of undesired proxy features, poor disentanglement, and excessive training time costs, which are not covered in the original paper. Beyond this, they leverage hard prompt search with negative prompts, which may handle potential negation and address these issues.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think this version is in general pretty good. I propose some points that can make this draft more robust in weakness.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qxBPayg3wY", "reviewer": "Reviewer_RH1q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1)  The paper successfully replicates the results from the original study, validating the claims presented in Section 2 by reproducing select experiments from Zhang et al. (2023a).  \n(2) It thoroughly examines all contributions made by Zhang et al. (2023a), including aspects like inclusive and high-quality generation, scalability across different domains, plug-and-play capabilities, data and computational efficiency, and scalability to multiple attributes. The reproducibility scope is clearly defined and easy to follow.  \n(3) Motivated by the attributes where ITI-Gen underperforms, the authors conduct experiments to analyze the impact of diversity and entanglement in the reference image datasets.  \n(4) Beyond mere replication, the paper explores the addition of negative prompts to Stable Diffusion as an alternative approach to handling negations in natural language, comparing the outcomes with ITI-Gen.  \n(5) The code used for reproduction is made available. The authors have implemented and shared additional code for integrating ITI-Gen with other models like ControlNet.  \n\n\nWeaknesses: \n\n(1) The replication of an existing study lacks a comprehensive introduction to the model in the evaluated paper. Specifically, Section 3.1 provides an insufficient description of the model. It would be beneficial to include a detailed explanation of the model architecture, the functionality of each component, the loss function, the unique aspects of the ITI-GEN proposal, and how these features address existing issues. A thorough introduction and clarification would offer readers a clearer understanding of the research and ensure that the authors have accurately interpreted the evaluated paper.  \n(2) Reproducibility research aims to validate the claims regarding model correctness, code accuracy, and experimental parameters made in the published paper. Therefore, it is recommended to incorporate as many replication details as possible. For instance, the authors mention, \"The authors do not discuss the hyperparameters used for image generation. We found the scale hyperparameter $\\lambda$ to be the most important one after some experiments.\" It would be advisable to provide specifics on how the authors determined the importance of $\\lambda$ and the experiments conducted to reach this conclusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper attempts to replicate and evaluate the findings of \"ITI-Gen: Inclusive Text-to-Image Generation\" by Zhang et al. (2023a).\n\nThe paper confirms several of the original authors' claims about ITI-Gen, including (1) its ability to enhance the diversity and quality of generated images, (2) its scalability across different domains, (3) its plug-and-play capabilities, and (4) its computational efficiency.\n\nHowever, the paper also identifies some limitations of ITI-Gen. It occasionally relies on undesirable attributes as proxy features and struggles to disentangle certain correlated attributes, such as gender and baldness. Additionally, as the number of attributes considered increases, the training time rises exponentially, and ITI-Gen faces challenges in generating inclusive images for all elements in the joint distribution.\n\nTo address these issues, the paper proposes the use of Hard Prompt Search with negative prompting. This method does not require training and is more effective at handling negation than the standard Hard Prompt Search.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in the comments above,\n\n(1) The section describing the model could be elaborated upon. It should reflect the authors' understanding rather than being a mere paraphrase of the original paper.  \n(2) The paper could benefit from including more details on the reproduction process.  \n(3) The authors should clarify the computational and time complexities associated with using HPSn.  \n(4) While HPSn can handle attributes like \"Gender\" and \"Bald\" better than ITI-Gen, it struggles with attributes difficult to define textually, such as skin tone. Could the authors propose a solution that encompasses a broader range of attributes? This could be a significant new contribution beyond merely evaluating existing work.  \n(5) Besides FID, have the authors considered conducting human evaluations to assess the quality of the generated images?  \n(6) Regarding communication with the original authors of ITI-Gen, they seem cooperative. Although the requested additional code might not be available at the time of this submission, the authors could cross-verify their implementation with the ITI-Gen authors since they have developed the code.  \n(7) In addition to the issues identified in the paper, such as ITI-Gen's struggle to disentangle attributes like \"Gender\" and \"Bald,\" are there aspects of the ITI-Gen model that the authors believe could be improved? For instance, the authors mention problems when training ITI-Gen on numerous attributes; have any potential solutions or experiments been attempted to address this?  \n(8) It would be beneficial to include a discussion on the experimental results, specifically addressing which results align with the claims in the ITI-Gen paper and which do not, along with potential explanations for any discrepancies.  \n(9) For the authors' implementation, such as ControlNet, adding implementation details to the submission would be valuable. The reproduction work should be as comprehensive as possible.", "broader_impact_concerns": "Given that the generated images involve human faces, it is recommended to incorporate a Broader Impact Statement or an ethical statement in the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree with the contribution in reproducing the results. The concerns about the novelty are not issues of TMLR.
Decision: Accept (poster)