# Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation

Mario Michelessa, Jamie Ng, Christophe Hurter, Brian Y Lim

► To cite this version:

Mario Michelessa, Jamie Ng, Christophe Hurter, Brian Y Lim. Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation. DIS '25: Designing Interactive Systems Conference, Jul 2025, Madeira Portugal, Portugal. pp.1867 - 1885, 10.1145/3715336.3735847. hal-05176177

HAL Id: hal-05176177

https://enac.hal.science/hal-05176177v1

Submitted on 22 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation

**Mario Michelessa**
National University of Singapore
Department of Computer Science
Singapore, Singapore
IPAL
Singapore, Singapore
mario.michelessa@u.nus.edu

**Jamie Ng**
Agency for Science, Technology and Research (A\*STAR)
Institute for Infocomm Research (I2R)
Singapore, Singapore
jamie@i2r.a-star.edu.sg

**Christophe Hurter\***
Université de Toulouse
ENAC
Toulouse, France
IPAL
Singapore, Singapore
christophe.hurter@enac.fr

**Brian Y. Lim\dagger**
National University of Singapore
Department of Computer Science
Singapore, Singapore
brianlim@comp.nus.edu.sg

<figure><img src="image_2.png" /><figcaption>**Figure 1:** Varif.ai supports user-defined diversity to S1) generate images, S2) verify their diversity based on attributes, and S3) vary diversity iteratively. Left: Images generated with a state-of-the-art diffusion model [40] using the prompt “*a picture of a car*” lacks diversity. Right: Images generated with Varif.ai has more diversity in car color, car model, and background landscape.</figcaption></figure>

## Abstract

Diversity in image generation is essential to ensure fair representations and support creativity in ideation. Hence, many text-to-image models have implemented diversification mechanisms. Yet, after a few iterations of generation, a lack of diversity becomes apparent, because each user has their own diversity goals (e.g., different colors, brands of cars), and there are diverse attributions to be specified. To support user-driven diversity control, we propose Varif.ai that employs text-to-image and Large Language Models to iteratively i) (re)generate a set of images, ii) verify if user-specified attributes have sufficient coverage, and iii) vary existing or new attributes.

Through an elicitation study, we uncovered user needs for diversity in image generation. A pilot validation showed that Varif.ai made achieving diverse image sets easier. In a controlled evaluation with 20 participants, Varif.ai proved more effective than baseline methods across various scenarios. Thus, this supports user control of diversity in image generation for creative ideation and scalable image generation.

## CCS Concepts

* Human-centered computing → Interactive systems and tools; *Empirical studies in HCI*; *Applied computing* → Media arts.

## Keywords

Image generation, human-AI interaction, diversity, creativity tools

## ACM Reference Format:

Mario Michelessa, Jamie Ng, Christophe Hurter, and Brian Y. Lim. 2025. Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation. In *Designing Interactive Systems Conference (DIS ’25), July 05–09, 2025,*

<figure><img src="image_3.png" /></figure>

<footnote>\*Co-corresponding author</footnote>

<footnote>†Co-corresponding author</footnote>
Funchal, Portugal. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/3715336.3735847

# 1 Introduction

Generative text-to-image models have become highly popular for their ability to create high-quality images from natural language prompts [19, 44], while offering users versatility and control over the generation process. Various approaches have been developed to provide fine-grained control, including prompt engineering techniques [10, 25, 41, 51, 69], and direct manipulation techniques like in-painting [46, 71], image editing [31], and layout control [80].

However, current generative models often lack diversity. For example, when generating a set of cars, generative models frequently exhibit biases toward specific car models, car colors, or landscapes (see Fig.1, left). This can hinder creativity in design by limiting inspirational examples. The lack of diversity also extends beyond visual attributes and affects social ones, like gender [13, 68], geographic origin [7, 60, 79], or social background [9, 44]. Ensuring diversity in image collections is crucial not only for mitigating stereotypes and reducing biases [62], but also for supporting creative ideation for effective communication and design [20, 30, 36].

Several augmentation methods have been proposed to automatically increase diversity [8, 28], but they cannot encompass all diversity, since this depends on user needs and perception. Hence, diversification needs to be user-driven. With current techniques to control image generation, users can fine-tune each image, but this is tedious and not scalable to diversify a large set of images.

To support user-driven diversity, we propose Varif.ai to empower users to define, measure, and improve the diversity of image sets through an iterative three-step process: S1) (Re)Generate a set of images based on an initial prompt, S2) Verify whether the user-specified attributes have the desired diversity coverage, S3) Vary existing attributes and their labels, or introduce new attributes. Our approach is built on top of off-the-shelf foundation models, making it model-agnostic and not requiring model retraining.

We first conducted an elicitation study with 8 participants to determine the needs for diversity in image generation, and understand the limitations of standard prompt engineering. From these initial findings, we developed and designed Varif.ai to help users to verify attribute diversity through histogram visualizations and vary diversity by interactively changing the histograms to execute probabilistic prompt generation. Following this, we conducted a formative study with the same 8 participants to verify that Varif.ai improved their experience and ability to diversify images, and explore the diverse images they generated. Next, we conducted a controlled evaluation study with 20 participants to compare Varif.ai against baseline methods (simple prompt engineering, and automated-diversified GPT-4o [22] and Promptist [28]), and examined how Varif.ai supports diversity with different specification degrees (open-ended, attribute-specific, attribute-label-specific).

Our contributions are:

1) Insights into how users define, measure, and control diversity for generated images.

2) Varif.ai, an interactive visualization<fnref n="1" /> to generate diverse images through iterative verification and variation of attributes.

Evaluations show Varif.ai improved user-driven image diversity. We discuss its scope, limitations, generalization and future work.

# 2 Related Work

We discuss background work on text-to-image generative models and user interfaces, and methods to improve and evaluate diversity.

## 2.1 Prompt Engineering for Image Generation

Diffusion models [75], such as DALL-E 2 [56], Stable Diffusion [58], and SD-XL [54], have enabled the generation of remarkably high-quality images. Lay users can leverage these models through text-to-image prompt engineering. However, prompting can be difficult and unintuitive [76]. Liu and Chilton [41] emphasized the importance of iterating prompts multiple times, given the stochastic nature of the models. Oppenlaender [51] and Hutchinson et al. [32] highlighted the crucial role of detailed modifiers in prompts because images contain more information than simple text. While these approaches require human ingenuity to prompt effectively, an alternative approach is to leverage large language models (LLMs) to generate augmented prompts to improve image quality and fit [8, 28]. In this work, we use a similar approach to augment diversity.

## 2.2 Interaction Design for Image Generation

Sidestepping the challenges in prompt engineering, researchers have proposed visualization-based interfaces for human-AI collaboration with text-to-image models. PromptCharm [71] integrates multimodal techniques, such as inpainting [46] for modifying specific image regions, and attention-control mechanisms for tuning the intensity of concepts in images, surpassing traditional prompt engineering to better align with user preferences. PromptPaint [15] allows users to go beyond language to mix prompts that express challenging concepts through paint-like interactions. Once again, these tools focus on image quality rather than diversity. Targeting artists [14], designers [42, 43, 52, 63], and game developers [18], some tools have been proposed to enhance creativity. However, they focus on generating one image at a time, which can be tedious when diversifying a set of images together.

## 2.3 Image Diversification

Obtaining visual references is typically done via web search due to its accessibility and breadth. This has been widely adopted in image creativity tools, such as PicturePiper [24] with reconfigurable pipelines for interactive image retrieval, example-based design [36] with adaptive galleries, and MetaMap [34] for idea exploration via multimodal semantic, color, and shape spaces. However, image search is limited to existing content, restricting the exploration of novel or hybrid concepts. Generative models offer a more expressive alternative [49, 63], allowing users to synthesize and steer images through fine-grained control over visual attributes.

Yet, image generation introduces challenges such as distributional biases and limited support for user-driven diversity. State-of-the-art text-to-image models have been shown to be biased in different attributes, such as gender [13, 68], geographic origin [7, 60], social background [9, 44], or cultural artifacts [79]. Tests with lay-users also revealed that a lack of variability in AI-generated images significantly hinders creative outcomes [67]. Several approaches

<footnote n="1">¹Code available at https://github.com/mario-michelessa/varifai</footnote>
<table><caption>Table 1: Background of participants indicating their experience in design and AI, and selected topic in the elicitation study.</caption><thead><tr><td>Participant</td><td>Design experience</td><td>Image generative AI experience</td><td>Self-chosen image topic</td></tr></thead><tbody><tr><td>E1 (30M)</td><td>Professional artist for 4-5 years, mainly in paint.</td><td>None.</td><td>Poses of falling monkey.</td></tr><tr><td>E2 (24F)</td><td>Professional designer in advertising for 3 years, and children's book illustration.</td><td>None.</td><td>Poster of Asian friends traveling in Scotland.</td></tr><tr><td>E3 (23M)</td><td>Graduated in design, but no professional experience.</td><td>Some use of image generation AI.</td><td>Very tall red concrete building.</td></tr><tr><td>E4 (26F)</td><td>None.</td><td>None.</td><td>Outdoor community centers.</td></tr><tr><td>E5 (25F)</td><td>Professional designer and animator in film industry for 3 years.</td><td>None</td><td>Cockroaches drinking beer in a sewer.</td></tr><tr><td>E6 (24F)</td><td>Professional designer in textile industry for 4 years.</td><td>None</td><td>Dancing colorful frogs.</td></tr><tr><td>E7 (44F)</td><td>Professional Artist for 20 years.</td><td>Extensive use for ideation.</td><td>Ancient doorframe with intricate details.</td></tr><tr><td>E8 (27M)</td><td>None.</td><td>Some use of image generation AI</td><td>Logo for an AI startup</td></tr></tbody></table>

have been proposed to improve diversity in generated images, based on model fine tuning [48, 61] or prompt optimization [77] to align with predefined target distributions for specific attributes, in the pursuit of fairness. Ironically, some automatic approaches have sparked controversy of over-doing the diversity [26, 27, 50]. Indeed, user control is needed to manage the diversity to satisfy their specifications, which we support in our work.

## 2.4 Measuring and Visualizing Diversity

In order to improve diversity, one must first measure it. Several computational metrics have been defined for diversity [4, 33, 39, 53, 74], but as singular metrics, each does not encompass all definitions of diversity, and they are too abstract for lay users to understand. In contrast, more qualitative approaches, such as examples [78] or visualizations (e.g., histograms, clustering), offer richer, more accessible insights. Assogba et al. [3] used clustering in a semantically meaningful embedding space to support exploration and multi-criteria evaluation of generated images. IF-City [47] employed bar charts and heatmaps on 3D urban plans to explain fairness and inequality. FairSight [2] and FairVis [12] used multiple coordinated views of various visualizations to allow inspection of data and distributive biases. Inspired by these methods, we include computational metrics and visualizations to evaluate, measure, and communicate diversity in images as a key step in our work.

# 3 Elicitation user Study

In an elicitation study, we determined the needs for diversity in image generation and the limitations of basic prompt engineering,

## 3.1 Method

3.1.1 Participants. We recruited 8 participants (E1–E8) through snowball sampling for the elicitation study. It was conducted over Zoom for 30 minutes. Participants were compensated $22 USD in local currency. They were 5 females, median 28 years old, with varying experience in design and AI (see Table 1 for details).

3.1.2 Study procedure. The study consisted of two parts: i) semi-structured interview to understand participants’ needs for diversity, ii) user interaction with an image generation tool while thinking aloud to explore the limitations of prompt engineering. The interview centered on the following tasks and questions: *Recall a time when you needed to browse for image references. Explain how you did it and which tools you used. How many images did you review before deciding on the right one? How would you have liked to control the set of images proposed to you?* Participants shared their screen with the researcher so they could show references if they wished.

Next we asked participants to use a basic tool that we developed with a text input to enter prompts and a gallery displaying 10 generated images. They were tasked to generate images for inspirational references for any topic (see topics in Table 1). They were asked to think aloud to articulate their rationale when writing the prompts and examining the generated images, and to understand how they considered diversity and measured and (wanted to) control it.

## 3.2 Findings

We performed the thematic analysis [65] on user interactions and utterances. Thematic codes were identified by the first authors and verified by the last author. We found that participants i) iteratively refined their goals during the ideation process, ii) considered diversity in terms of attributes and labels, and iii) wanted to control diversity by defining labels and changing proportions.

3.2.1 Iterative Ideation and Need for Diversity. All participants emphasized the importance of having diverse images when selecting visuals. For example, in the context of creating communicative images for a presentation slide, E4 explained that he often needed to look at "*20 images, sometimes more, to see a wide range of possibilities before choosing what to include*". He also noted the role of serendipity, adding that sometimes "*I particularly like one of them and continue my search based on [that] image*".

Participants noted the iterative nature of their creative process as they refined or revisited their goals. E2 remarked that "*the first
<figure><img src="image_4.png" /><figcaption>Figure 2: The user interface of Varif.ai comprises four main panels. The Prompt text input (1) allows users to define a general description of the images and specify the number of images to generate. The Image gallery (2) displays the generated images and provides popup tooltips showing their specific descriptions and attribute labels. The Attributes panel (3) allows users to add attributes (e.g., Color, Period, Landscape), measure the distributions of their labels (e.g., Red, Black, etc., for Color) as vertical histograms, and modify the distributions as desired. The Action history panel (4) records the generated images after each iteration as the user modifies the attributes.</figcaption></figure>

[iteration] should be kind of unbiased, the first idea should be very
rough and very draft[-like]". Similarly, E7 shared that "at the start I
want to have references for different ideas quickly [...] and little by
little the exploration will be narrower and narrower, the more I add
criteria". E8 remarked that "it happens that my initial searches were
[sometimes] better, so I revert back to [them]. [...] I always keep the
first search close so that I can compare and revert to it".

3.2.2 *Definition of Diversity.* Participants sought for diversity in terms of attributes, often specifying several labels. Five out of 8 participants mentioned the style of the images, providing various desired labels: E2 mentioned "abstract, steampunk, realistic", E5 suggested "cartoony, dark, cute", and E1 referred to "pixel-art, video game, photography". Other frequently mentioned attributes included colors (5/8 participants) and background (3/8 participants). Some attributes were more specific to particular topics, such as ethnicity. For example, E8, when searching for images of a person sitting next to a machine, stated, "I want [the word] "people" to encompass all types of people. I want to specify that I want diversity in ethnicity".

To verify image diversity, participants applied their own criteria. Six out of 8 participants considered images diverse when they covered a wide range of different labels. When exploring communal spaces, E4 noted that she “imagined a variety, but it’s pretty much the same option in terms of types of activities.” Similarly, E3 observed a lack of diversity in images of buildings, stating that there was “not a lot of different architectural style, not enough variety”.

3.2.3 Methods for Controlling Diversity. Participants diversified their images by adjusting the proportions of each label, sometimes in an imbalanced manner, to prioritize some labels over others. While searching for a composition featuring a manhole, E5 stated that it would be better to have “most of the images with a centered composition [with the manhole in the center] and just some images with different compositions, with the manhole in one of the corners”.

Struggling with prompt-engineering, E2 obtained text suggestions to "help me improve my imagination for the prompt. Maybe I didn't think about pink or steampunk, and I can consider them now." Likewise, E6 wanted "suggestions of prompts, to see what images are possible to obtain and view them all at the same time".
<figure><img src="image_5.png" /><figcaption>Figure 3: User interaction flow for the doctor scenario described in Section 4.</figcaption></figure>

Finally, two participants wanted to modify multiple images as a batch. E1 “wanted to replace some [images] with the same [ones], but with the feet on top of the head” to diversify the orientation of a monkey, and E6 wanted to “have [the same frog] in different environments and settings”.

## 3.3 Design Goals

Building on the findings of the formative study, we propose several design goals to enable users to control image diversity through user-defined attributes to align with user specifications.

S1) **GENERATE** a collection of images. Users generate an initial set the images with a simple text prompt, view the images as a whole to observe their variability and assess the need for diversity.

S2) **VERIFY** image diversity based on attributes. Users specify diversity in terms of attributes they want to see vary, and view the prevalence of various labels in a histogram. Attribute labels are initially proposed using an LLM.

S3) **VARY** attribute label distribution. Users adjust the histogram to change labels or their proportions. These labels are sampled and appended to the original prompt as modifiers to adapt each image.

* **ITERATE** image generation. Users generate images based on the new attribute label distribution, and can repeat steps S1-S3. We also include an action history to review or revert changes.

## 4 Varif.ai Interaction Design

Following our design goals, we developed Varif.ai (see user interface in Fig. 2) to facilitate users to verify and vary the diversity of image generation. Here, we describe its capabilities with a scenario to generate diverse images of doctors with various ethnicity (see interaction flow in Fig. 3).

### 4.1 Initialization

To **GENERATE** an original image collection, Varif.ai allows the user to input a context prompt<fnref n="2" />, such as “*an image of a kind doctor during a consultation*”, specify 10 images, and click the **GENERATE** button.

The images are generated and displayed in a gallery for the user to review. Clicking on each image enlarges it for close inspection. In this scenario, the user identifies a lack of social diversity in the generated images, as most of them depicted white, old men.

### 4.2 Verify Diversity

As the user identifies specific attributes to explore, namely ethnicity, gender and age, the user starts by adding one to the Attributes tab. After entering “Ethnicity,” Varif.ai suggests a set of labels generated by a large language model (LLM): Caucasian, Black, Asian, Hispanic, and Middle-Eastern. The user can modify, add, or remove any of the labels. Furthermore, if the user needs help, the user can click the **SUGGEST** button, to ask the LLM (like [29, 64]) to provide possible attributes based on the context prompt. Varif.ai then classifies the generated images to the labels, counts them and visualizes their

<footnote n="2">²Similar to popular prompt-based interfaces [5, 10, 25].</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 4: Conceptual diagram of probabilistic prompts. For each attribute (e.g., color), Varif.ai samples labels (e.g., red, green, blue) according to user-defined probabilities (e.g., 40%, 50%, 10%) and replaces the placeholder in the context prompt. When sampled repeatedly, generated image sets reflect the specified distributions.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5: Varif.ai architecture. S1) **GENERATE** an image collection with an initial prompt $t_0$, duplicated (Π) $n$ times and fed into an image generation model $G_{img}$ to obtain generated images $\tilde{X}$. S2) **VERIFY** attribute diversity, each from an attribute $t_a$ and the prompt $t_0$, combined in a prompt template $\mathcal{A}$ and fed into a language model $G_{LLM}$ to obtain $n_a$ labels. Each image is then labeled using a vision-language model $C_a$, and the distribution are obtained by counting ($\Sigma$) the labels in the image set. S3) **VARY** attribute diversity to generate images following the user-defined distribution, where $n$ labels are sampled ($\sim$) and appended ($\|\|$) to the original prompt $t_0$ to obtain extended prompts, and executing Step S1b and S2c again.</figcaption></figure>

distribution in a histogram for each attribute (VERIFY). Hovering the mouse over each image shows a pop-up stating its predicted labels. Hovering over each histogram bin highlights images with that attribute label. In this scenario, the user examines the histogram and observes that over 80% of the images are classified as Caucasian, with the remaining images as Hispanic and Middle-Eastern. This indicates a significant lack of diversity in the image set.

## 4.3 Vary Diversity

Each attribute histogram is interactive with sliders (Fig. 3c) that the user can VARY diversity to achieve goals such as increasing fairness, prioritizing specific labels, improving coverage, or removing undesired labels. We include a convenient Balance button to set the histogram to a uniform distribution. In this scenario, the user improves coverage and fairness by adding the “Native American” label and by balancing all attribute labels across ethnicities.

## 4.4 Iterate

After clicking the **GENERATE** button again, Varif.ai generates a new set of images aligned to the user's specified distribution. The histograms and pop-up tooltips are updated to reflect the new distributions and labels. The user can track changes and branch from

any previous iteration using the Message History bar. Examining the updated histogram, the user verifies that Varif.ai has successfully diversified ethnicities. Subsequently, the user can **ITERATE** to verify and vary the diversity of other attributes—gender and age.

# 5 Technical Approach

We describe our technical approach to support the design features of Varif.ai to: S1) generate image sets, either initially or iteratively, using a text-to-image diffusion model, S2) verify diversity through image-to-text modeling, and S3) vary diversity by sampling the text prompt distribution based on the user-defined attribute histograms, and reapplying steps S1 and S2. Architecture shown in Fig. 5.

## 5.1 Generate Original

Given a description prompt provided by the user, the first step is to generate an image collection using a diffusion model. 1a) The user defines a text prompt $t_0$ that describes the common theme of the images, and the number of desired images $n$ in the user interface. 1b) $t_0$ is duplicated $n$ times and each prompt is input to a text-to-image generation model $G_{img}$. Since our method only modifies the text prompts, any text-to-image generation model can be used. 1c) The generated images $\tilde{X}$ are displayed in the user interface.
<table><caption>**Table 2:** Example labels proposed by $G_{LLM}$ for the same attribute and different contexts.</caption><thead><tr><th>Context t<sub>0</sub></th><th>Generated labels v̂ for attribute t<sub>a</sub> "age"</th></tr></thead><tbody><tr><td>Person</td><td>Child, Adolescent, Young Adult, Middle-Aged, Elderly</td></tr><tr><td>Doctor</td><td>30s, 40s, 50s, 60s, 70s</td></tr><tr><td>Bridge</td><td>Newly built, Recent, Old, Ancient, Historic</td></tr><tr><td>Car</td><td>New, Used, Old, Classic, Vintage</td></tr></tbody></table>

## 5.2 Verify Diversity

To determine diversity along an attribute, multiple labels for the attribute must be measured. Since the attribute $t_a$ is open-ended, Varif.ai uses an open-world large-language model (LLM) to propose $n_a$ labels. 2a) The attribute $t_a$, the context $t_0$, and the number of labels $n_a$ are combined in a prompt template $A$ (see Appendix A), and 2b) input to LLM $G_{LLM}$ to generate attribute-label prompts $v̂$ (examples in Table 2). If users need attribute suggestions, Varif.ai can input the context $t_0$ to $G_{LLM}$ to generate a suggested attribute $t_a$.

For each attribute, an image $\hat{x} \in \hat{X}$ is labeled via classification using CLIP [55], since we are using user-defined, open-domain class labels instead of predetermined fixed classes with pretrained classifiers. 2c) For each attribute label, we compute the CLIP similarity score between the image $\hat{x}$ and label $v̂$, then choose the label with the highest similarity $\hat{y}_a$. This is the predicted label for the image. 2d) We then count the labels in the image set to determine the distribution $\hat{p}_a$, which is visualized in the attribute histogram.

## 5.3 Vary Diversity

To vary image diversity based on attributes, 3a) the user can adjust the sliders in the histogram, which modifies the distributions $\hat{p}_a$. Varif.ai generates prompts probabilistically by 3b) sampling $n$ labels from the distribution of attribute labels (see Fig. 4) and appending to the original context prompt $t_0$, and 3c) reapplying step 1b to 3d) generate a new image set to better align with the new distribution. 3e) Step S2c is reapplied to verify the distributions and update the histograms. 3f) The user can also vary diversity by adding labels in $v̂$ to 3g) generate images representing the new labels.

## 5.4 Implementation Details

We implemented Varif.ai as a responsive web application. The front-end is implemented with Material UI components<fnref n="3" />, the back-end server with Python Flask, and all machine learning models with Pytorch<fnref n="4" /> and Diffusers<fnref n="5" />. For specific models, we used the open-source SD-XL Lightning [40] for text-to-image generation, and the open-weight LLM LLaMA-2 [66] to generate suggestions. We used 8-bit quantized versions of models to accelerate inferences. To ensure the LLM returns lists of labels, we employed the guidance library <fnref n="6" />, with the specific instructions provided in Appendix A. Varif.ai ran on an Nvidia H100 GPU, and generating 10 images

takes around 6 seconds. Our code is available at: https://github.com/mario-michelessa/varifai

# 6 Formative User Study

We conducted an elicitation study to examine whether Varif.ai satisfies our design goals from Section 3 and determine its usability.

## 6.1 Participants and Study Procedure

We re-invited the same 8 participants from the formative study to use Varif.ai using the same or similar scenario that they had explored previously. Each session lasted 30 minutes on Zoom with recorded screenshots and using the think-aloud protocol. At the end of the session, we conducted a semi-structured interview, focusing on Varif.ai's usability and the helpfulness of its components.

## 6.2 Findings

We performed a thematic analysis and affinity diagramming [45] of participant usage and utterances. We report the usage patterns of Varif.ai, its perceived usefulness, weaknesses, and feature requests.

### 6.2.1 Usage Patterns of Varif.ai
Participants exhibited different usage patterns (Fig. 6). Six out of 8 participants iteratively added attributes to explore "with and without the attribute" [E3] and test if "adding one [attribute] doesn't mess everything up" [E4]. Participants iteratively adjusted the attribute distributions while monitoring if changes were effective or non-responsive. For example, wanting images of colorful frogs, E6 first measured the attribute "vibrant colors" (Fig. 6a, Iteration 1) and observed too many green frogs. Despite balancing the proportion of "this category [...] because there are still too many [green frogs]" (Iteration 2). She overcompensated to further reduce green frogs by setting the label a minority.

In addition to interacting with the attribute histograms, some participants modified the original context prompt. For example, E5, an artist searching for images of cockroaches with a beer, initially adjusted attributes to generate images without beer (see Fig. 6b, Iterations 1-2). She then modified the prompt to include beer (Iterations 3-4), explaining that she "wanted to see how the system would do without the beer first, and then add something more complex".

### 6.2.2 Feedback on Feature Helpfulness
Participants generally found Varif.ai's components helpful, and readily used labels and histograms.

**Attributes.** Participants appreciated the ease of use of the Attributes tab in Varif.ai. E6 noted, "once I understood how it works it's a lot easier to explore what I want. It's so much easier to type [the attribute] instead of typing [each label] one by one". However, two participants reported ambiguity when using intricate prompts or many attributes, as it was unclear to which object an attribute affected. For example, E6 observed that the attribute "Vibrant colors" applied to the background instead of the frogs (see Fig. 6a, iteration 4). Similarly, E3 "want[ed] the color for the building, not the background, [...] I'll just try to add "building color"", which ended up properly targeting buildings.

**Attributes Labels.** Participants appreciated both the label suggestions and editability. While looking for community centers, E4 found the suggested labels for "Crowd" (Crowded, Empty, Busy, Vibrant, Quiet) "very smart, I would have never thought about all of them". Yet, participants may want more than what was suggested.

<footnote n="3">³mui.com</footnote>

<footnote n="4">⁴pytorch.org</footnote>

<footnote n="5">⁵github.com/huggingface/diffusers</footnote>

<footnote n="6">⁶github.com/guidance-ai/guidance</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 6: Examples of participants iteratively adding and adjusting attribute distributions, and revising the context prompt.</figcaption></figure>

While looking for cockroach references, E5 added unconventional labels under the “Image Style” attribute, such as “Peppa Pig” or “Looney Tunes” (see Fig. 6b, iteration 1) to “*see if it gives a more cartoony and childish vibe*” to the images. She liked that label editing “*helps you to really customize and add your own vision inside. What [generated images] depict is very common, and so it helps getting inspired to put something completely off*”.

**Distribution of Attribute Labels.** Participants appreciated being able to control the proportions of attribute labels using the sliders. Three participants deliberately assigned low probabilities to labels

they “*don’t want, [but still] prefer to have some examples to compare with what I want*” [E3]. While exploring images of falling monkeys, E1 strongly preferred a pixel-art style but still included other styles “*just to keep options open a little bit*” (Fig. 6c, Iteration 1).

### 6.2.3 Usability issues.
Participants noted some weaknesses due to technical issues in the underlying models affecting the UI behavior.

**Inaccurate Attribute Label Distributions.** Some participants found that the measured attribute histograms were inconsistent with the actual labels of the generated images. For example, E6 observed
<table><caption>Table 3: Criteria for the three diversity specification degrees (columns) and the three scenarios (rows.)</caption><thead><tr><th>Scenario</th><th>Open-ended</th><th>+ Attribute-specific</th><th>+ Attribute-label-specific</th></tr></thead><tbody><tr><td></td><td>"You are ...</td><td>"Increase diversity in ...</td><td>"Satisfy the following ...</td></tr><tr><td>Doctors</td><td>"a kid book illustrator wanting to generate inspiration images for a kid book, portraying some doctors."</td><td>"gender, race, environment, and image style."</td><td>"75% women and 25% men, have 33% Black, 33% Asian and 33% White people. Have 50% hospital, 25% home consultation and 25% office. Have 70% photos and 30% cartoon."</td></tr><tr><td>Birds</td><td>"an elementary teacher, you want to show your students the variety of birds that exist."</td><td>"type of birds, habitat, and pose."</td><td>"100% photos, Have 40% forests, and 10% each: savannah, desert, polar, swamp, river and sea. Have 25% each: flying, on a branch, on the ground, or on water. Have 10% tall, small birds, 70% others."</td></tr><tr><td>Cars</td><td>"a car amateur wanting to see cool references for a car poster/wallpaper."</td><td>"color, environment, and weather conditions."</td><td>"100% photorealistic images. Have 20% each: blue, red, yellow, green, or purple. Have 40% sunny, and 20% each: snowy, cloudy, or rainy. Have 75% in the city, 25% in the countryside."</td></tr></tbody></table>

that frogs were consistently classified as green, despite most not appearing so (see Fig. 6a, iteration 4). She speculated, “I don’t know why it keeps saying that most frogs are green. Maybe it’s because of the grass,” highlighting the challenge of accurately attributing labels to specific objects within an image.

**Entangled Attributes.** Two participants found it limiting that attributes were treated independently, and wanted to conjoin them. For instance, E4 sought examples of communal spaces that were “crowded by day but empty at night”, but could not achieve this by adding the attributes “Crowd” and “Time of the Day.” Conversely, latent interactions between attributes occasionally led to unintended but welcomed diversity. While searching for an image of friends in Scotland for an ad campaign, E2 observed that diversifying the attribute “Weather” resulted in “a lot more diversity [than at the beginning], not just in weather but also in landscapes.”

6.2.4 *Feature Requests.* Several participants requested additional features. Artist E1, who had a distinctive personal style, expressed interest in using custom *seed images*, asking whether it was possible “to put my own images in the interface, and try from there.” E2, encountering undesirable outcomes, wanted to *suppress or prevent certain attribute labels to remove completely the buildings.*

# 7 Summative user study

Having found that users can use Varif.ai in our formative user study, we next conducted a controlled study to investigate whether Varif.ai improves *image diversity* compared to the baseline prompt engineering, and under what settings. Furthermore, our earlier elicitation study found that using prompt engineering was tedious, so we also examined whether Varif.ai improves *user engagement* in terms of time engaging on the task and number of iterations.

For external validity, we note that users may want to specify diversity with different degrees: from only desiring some variety, to defining specific attributes to diversify, to prioritizing specific attribute labels or requiring equal representation across labels. Specific to Varif.ai, we evaluated the diversity of attributes conceived by users, *alignment* to the target distribution of the aforementioned diversity degrees, and user effort to modify and finetune attributes.

<figure><img src="image_9.png" /><figcaption>Figure 7: Prompt-only UI without diversity controls.</figcaption></figure>

We focus on simpler prompt tasks with basic scenarios to limit overfitting to contexts. Our research questions are:

RQ1. Does Varif.ai improve a) image diversity and b) user engagement compared to prompt engineering?

RQ2. How are a) image diversity, b) attribute count, c) diversity alignment, and d) user engagement affected by the degree of diversity specification?

## 7.1 Experiment Method

7.1.1 *Experiment Design.* We varied Technique and Diversity Specificity Degree with Scenario as random variable, and measured Image diversity, Task time, Iteration count, and Attribute count.

*Technique (IV1).* We compared Varif.ai to Prompt-only (see Fig.7), which is an ablated version of Varif.ai with only the prompt text entry and no Attributes tab to view or add attributes. This was done within-subjects with counterbalancing to mitigate learning effects. We also compare against the original images generated from the initial prompt, without participant interaction, to analyze how much Prompt-only and Varif.ai increased diversity.

*Diversity Specificity Degree (IV2).* We examine how different degrees of specificity affect participants’ use of Varif.ai. *Open-ended* provides minimal guidance, offering a one-sentence scenario to explain the need for diversity and engage participants [35]. *Attribute-specific* defines several attributes for participants to focus on when
<figure><img src="image_10.png" /><figcaption>Figure 8: Results of a) Image diversity, b) Task completion, and c) Iteration count with different techniques. Dotted lines indicate extremely significant comparisons (*p* < .0001); otherwise, the *p*-value stated when comparisons are very significant. Solid lines indicate no significance (*p* > .01). Error bars represent a 90% confidence interval. Black vertical line separates the theoretical baseline (no users) from the user results.</figcaption></figure>

increasing diversity in addition to the open-ended criteria. *Attribute-label-specific* defines precise attribute labels and their distributions that participants are asked to satisfy in addition to the attribute-specific criteria. To avoid biasing participants in the open-ended condition, we present the degrees sequentially, from least to most detailed. To avoid overwhelming participants with a 3 × 2 condition setup, we only tested the Prompt-only interface with the open-ended specification degree, since we learned from our formative study that it would be too tedious for users to specify at scale.

*Scenarios (RV).* We use three scenarios—*Doctors*, *Birds*, and *Cars*—treated as a random variable between-subjects. These scenarios require no prior knowledge and are common enough to be generated well by diffusion models [59]. We include a social diversity scenario (*Doctors*) due to biases in generated images toward specific populations [7], and scenarios with animate subjects (*Birds*) and inanimate objects (*Cars*) to explore potential differences in how participants approach diversity. Table 3 summarizes the criteria provided for each diversity specification degree and scenario.

*Computed Image Diversity (DV).* We collect all the images generated by the users to measure their diversity. We augment each participant's image collection to 50 total images with the original attribute specification. This increased sample size allows us to compute diversity more precisely. From our elicitation study, participants generally considered diversity in terms of coverage. We thus measure how widely the concepts in the images are spread out as a proxy of diversity. We use the diversity *span* metric [11, 16], which represents the "radius" of the images' distributions to estimate coverage. More precisely, given image embedding *z* from the semantic image feature encoder CLIP [55], *span*(*z*) = *percentile*_{95%} ||*z* − *z*||. For a fair evaluation, similar to [61], we use a more powerful CLIP model[^7] than the one used in Varif.ai.

*Attribute Count (DV).* We measure the attribute counts and added labels from participants. Higher numbers suggest higher diversity for the open-ended degree, but the attributes may be extraneous for more specific diversity specification degrees.

*Diversity Alignment (DV).* To assess whether participants could satisfy precise distributions in the Attribute-label-specific diversity degree, we measured the alignment between the target distribution (Table 3, right column) and the participant's attribute distribution.

To quantify the distance between two probability distributions, we use the Kullback-Leibler divergence (KLD) and compute its inverse as *Diversity alignment*. Lower KLD indicates more alignment.

*Task Completion Time (DV)* and *Iteration Count (DV)* were used to measure user engagement. Since we are not focusing on efficient image generation, and creative ideation requires reflection, we consider the time taken to generate images as indicative of engagement. We expect more inspired participants to ideate for longer, and frustrated participants to engage less.

*Attribute Label Modifications (%) (DV)* measures the effectiveness of the LLM in providing aligned labels. We considered it tedious if users need to modify a large amount of labels manually. We expect users to modify labels most often for the Attribute-label-specific diversity degree due to its stringent requirements.

### 7.1.2 Procedure. Each participant went through the following:

1) *Introduction (10 min)* to the research background, including the motivation and study protocol. Demographic information was collected, and participants provided consent to record their operations and results for analysis.

2) *Tutorial (5 min)* on the main interactions, explaining their purpose and functionality.

3) *Main Session (40 min).* Randomly assigned a scenario task to generate images.

a) Generate images iteratively for the Open-ended degree of specification using Prompt-only, then Varif.ai (or in reversed depending on random assignment).

b) Generate images iteratively for the Attribute-specific degree for Varif.ai only.

c) Generate images iteratively for the Attribute-label-specific degree for Varif.ai only.

4) *Free Session (Optional)* to use Varif.ai on a scenario of their choice. Eight participants explored on their own briefly.

## 7.2 Analysis and Results

We recruited 20 participants through a local university mailing list and mutual connections. Participants ranged in age from 21 to 33 (Median = 23), were 12 female, and had diverse educational backgrounds: social sciences (6), medicine (3), arts (2), business (3), and STEM (6). Three participants reported some experience in image creation. Nine participants had no prior experience with

<footnote n="7">⁷huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k</footnote>
<figure><img src="image_11.png" /><figcaption>Figure 9: Results of a) Image diversity b) Diversity alignment c) Task completion and d) Iteration count for different specificity degrees. Dotted lines indicate extremely significant comparisons (*p* < .001); otherwise, the *p*-value stated when comparisons are very significant. Solid lines indicate no significance (*p* > .05), and error bars 90% confidence interval.</figcaption></figure>

text-to-image models, while the others had only used them a few times. No one reported regular or professional use of text-to-image models. The experiment, conducted over Zoom, lasted one hour with $15.30 USD in local currency as compensation. Participants shared their screens and were asked to think aloud.

For statistical analysis, we fit linear mixed-effects models for each dependent variable (DV) and performed post-hoc contrast tests for specific differences that we highlight. Next, we discuss significant results (*p* < .01).

### 7.2.1 Varif.ai Improved Diversity and Engagement Compared to Prompt-Only (RQ1). Participants unanimously found Varif.ai better than Prompt-only to diversify images with an open-ended goal.

*Computed Image Diversity (RQ1a).* Varif.ai significantly improved diversity (span mean *M* = 0.65) compared to Prompt-only (*M* = 0.45; *p* < .0001) and compared to Original (*M* = 0.35; *p* < .0001); see Fig. 8a. Participants were more satisfied with the diversity they accomplished with Varif.ai than with Prompt-only. P2, while diversifying images of birds using Varif.ai, commented that “[the images] are a lot more diverse than at the beginning, there is a lot more variety of birds and backgrounds”. There was no significant difference across scenarios, indicating the generality of the effect.

*Task Time (RQ1b).* Participants were engaged more when using Varif.ai compared to Prompt-only (*M* = 10.2 vs. 6.3 min, *p* = .019); see Fig. 8b. Indeed, some participants gave up earlier trying with Prompt-only, e.g., P16 “[didn’t think the AI understood what I wanted to do” after generating images of crowds of diverse doctors instead

<figure><img src="image_12.png" /><figcaption>Figure 10: Word cloud of attributes defined by participants for the three scenarios a) Doctors, b) Birds, c) Cars and for the Open-ended degree. The font size is proportional to the number of participants who specified the attribute.</figcaption></figure>

of one doctor per image, P2 “[didn’t] know how to make it clearer; I don’t know why it keeps giving me so many [doctors]”.

*Iteration Count (RQ1b).* There was no significant difference in number of iterations whether using Varif.ai (*M* = 3.7) or Prompt-only (*M* = 3.5); see Fig. 8b. Participants iterated as many times on Varif.ai (min=1, *M*=3, max=9) as on Prompt-only (min=2, *M*=3.5, max=9); see Fig. 8c. See examples of iterative prompt refinements from the Prompt-only trials in Appendix B.

### 7.2.2 Varif.ai Improves Diversity and Control for More Specific Degrees of Diversity Goals (RQ2). Participants could better diversify images when attributes were provided, and participants could align diversity to precise criteria using Varif.ai.

*Computed Image Diversity (RQ2a).* When given the goal of diversifying images with Attribute-specific degree, participants generated more diverse images than with Open-ended degree (span *M* = 0.80 vs. *M* = 0.66, *p* = .009); see Fig. 9a. Diversity span was not significantly higher for Attribute-label-specific degree (*M* = 0.71), as non-uniform requirements did not necessarily lead to high diversity outcomes. The results generalized across scenarios, having no significant difference among them. On their own, some participants initially struggled to identify attributes to diversify with Open-ended degree, such as P11, who wanted, “*cars, trucks, sports cars, but I don’t know how to say that*”. However, participants were more inspired when guided with specific attributes to consider. For example, P16 “*didn’t think about changing the background of the doctors, that’s a good idea*”.

*Attribute Count (RQ2b).* Though not significantly different, participants diversified *M* = 3.7 attributes for each specification degree, with values more varied for the Open-ended degree (*M* = 3.86, min = 1, max = 7) compared to the Attribute-specific degree (*M* = 3.64, min = 3, max = 4) and the Attribute-label-specific degree (*M* = 3.70, min = 2, max = 4); see Table 4, first row. Interestingly,
<figure><img src="image_13.png" /><figcaption>Figure 11: Results of image diversity from the summative user study using Varif.ai for one image set of 20 images in the Open-ended diversity degree (blue), compared to automatic diversification by Promptist [28] (red) and GPT-4o [22] (orange) across scenarios a) Doctors, b) Birds, and c) Cars. Each bar is a single measurement, so there are no error bars.</figcaption></figure>

beyond a few attributes commonly chosen by most participants, the selection of attributes varied widely. The attributes selected with the Open-ended degree and for each scenario are shown in Fig. 10. A few participants (P7, P18, P20) used the open-ended nature of attributes to add scenario-specific details. P7 included "toys [to] help with the child book". Participants in the Doctors scenario primarily focused on social diversity. P1 and P5 actively reduced the proportion of labels representing stereotypical images, increasing representation for minorities to "compensate for the stereotypical images" [P5]. Additionally, participants identified more attributes to diversify animate Birds compared to inanimate Cars.

*Diversity Alignment (RQ2c).* When prompted with the Attribute-label-specific degree, participants generated image sets that more closely matched the target distributions compared to the Attribute-specific degree (alignment *a* = 0.79 vs. *a* = 0.74, *p* < 0.001) and the Open-ended degree (*a* = 0.79 vs. *a* = 0.73, *p* < 0.001).

*Task Time (RQ2d).* Using Varif.ai, participants were engaged for a longer time during Open-ended compared to Attribute-specific (M = 10.2 vs. 8.2 min, *p* = n.s.) and Attribute-label-specific (M = 10.2 vs. 6.9 min, *p* = n.s.); see Fig. 9c. Although the difference is not significant, this task time decreased as the specificity degree increased, which can possibly be influenced by the think-aloud protocol, where participants tended to discuss the Open-ended degree more extensively, and by the practice effect of having more experience in earlier sessions and higher-degree specifications coming later. Iterations followed a similar trend, because participants explored more for the Open-ended degree.

*Attribute Label Modification (RQ2d).* Participants retained the attribute labels suggested by Varif.ai in 86% of the cases for Open-ended and Attribute-specific, and significantly less for Attribute-label-specific with 57.3% (*p* = 0.006); see Table 4, second row. Participants modified labels the most for Attribute-label-specific to match the labels required in the criteria instructions. However, this can be tedious. P20 struggled to get the right labels when adding the attribute "type" in the Birds scenario with Attribute-label-specific degree, and "finally [gave] up and enter[ed] them one by one".

### 7.2.3 *Comparison with Automatic Diversified Prompts*
Our user study results had shown that using Varif.ai, participants can improve diversity compared to plain prompt engineering. However, recent text-to-image generative models, such as DALLE-3 [8], implicitly use LLMs to suggest diversified prompts to generate the image. We compared Varif.ai against two more advanced baselines using GPT-4o [22] and Promptist [28] as LLMs to prompt the same diffusion model as before, SD-XL Lightning [40]. For each scenario, we prompted the two models to "provide diverse prompts to generate images of [scenario]" and generated 50 image prompts to generate 50 images with the diffusion model. This matches the augmented sample for each participant in the summative study. We measured the image diversity of each image set to indicate the open-ended diversity, and diversity alignment to indicate how well the image set satisfied the user specifications in terms of attributes.

*Open-Ended Image Diversity.* Fig. 11 compares computed image diversity of each Varif.ai user participant to the automatic diversity baselines. While these automated baselines had higher diversity than most users, some Varif.ai participants did achieve even higher diversity (Fig. 11a, b), or only marginally lower (Fig. 11c). Although automatic diversity can be effective, it does not necessarily support specific diversity needs of users, which we examine next.

*User-Defined Diversity Alignment.* We evaluated how automated diversification baselines compare with Varif.ai in achieving user-defined diversity. To standardize the comparison, we focused on the three most common attributes by participants in the summative user study (Fig. 10), LLM-generated 5 labels for each attribute, and set the label distribution to be uniform. To measure the actual counts of attribute labels, the first author manually annotated each image, hence the number of attributes analyzed was small for this study. With these counts, we measure the diversity alignment of each attribute, and average them for each diversification method.

Fig. 12 shows that Varif.ai has higher diversity alignment than the automatic baselines, particularly for the Car and Doctor scenarios. This indicates that automatic diversity does not target user-driven attribute-based diversity well. Furthermore, the diversity alignment of automated baselines for the Doctor scenario was especially low
<figure><img src="image_14.png" /><figcaption>**Figure 12:** Diversity alignment to uniform attribute distributions of images generated with user-driven Varif.ai (blue), automated diversified Promptist [28] (red) and GPT-4o [22] (orange) for different scenarios. Each point corresponds to one attribute.</figcaption></figure>

(M = 0.37), highlighting the remaining gap to satisfy user-specific requirements of diversity for this socially-sensitive domain.

## 7.3 Analysis of CLIP Performance on Diversity

From our formative study, we noted that the attribute label histograms may be inaccurate. This could be due to CLIP sometimes mislabeling images. This can affect the performance of the verification and varying steps in Varif.ai. To investigate the impact of this issue, we conducted a sensitivity analysis on the effect of CLIP labeling accuracy on image diversity measurement and generation, by measuring the diversity alignment based on CLIP labels and ground truth labels, respectively.

7.3.1 *Method.* To cover a range of accuracies, we sampled 12 attributes from Fig. 10 to diversify in Varif.ai. Attribute labels suggested by the LLM are left unchanged. Across the scenarios, for each attribute, we generate 20 images with the simple base prompt appended with the attribute and randomly sampled attribute values. We set Varif.ai to vary the labels to have a diverse uniform distribution in one iteration. With the verification step, we note the CLIP-based attribute label (*CLIP label*) that were measured for each image. The distribution of attribute labels may still not be uniform, indicating a limit in measurement or manipulation which we investigate here. For each image, we obtain the ground truth attribute label (*Actual label*) with the first author's manually annotation. Independent variable *CLIP label accuracy* is calculated as how often the CLIP label matches the Actual label. Dependent variables for *Diversity alignment* based on CLIP or Actual labels are calculated as the KL divergence between a uniform attribute label distribution and the respective distributions.

7.3.2 *Results.* Fig. 13 shows how diversity alignment changes with CLIP label accuracy for CLIP-based and Actual labels. As expected at high accuracies (>0.8), both types of alignments are high. We performed a linear fixed effects regression to determine if the effects of CLIP accuracy are significant. CLIP-based alignment decreases significantly with decreasing CLIP accuracy ($\beta$ = 0.76, *p* < .001), indicating that measurement accuracy of labels decreases as CLIP

<figure><img src="image_15.png" /><figcaption>**Figure 13:** Effect of CLIP-based image-attribute classification accuracy on diversity alignment calculated by CLIP-based labels (red) or Actual labels (blue). Lines show the linear fit with 95% confidence interval (shaded area).</figcaption></figure>

accuracy is lower. In contrast, Actual alignment does not significantly change ($\beta$ = 0.18, *p* = n.s.), suggesting that the diverse generation is successful regardless of CLIP measurement accuracy. Perhaps, this is due to the diffusion process using CLIP on the full prompt appended with the attribute label to ensure convergence of the generated image, while our verification step uses only CLIP between the image and the short-text attribute label alone. Future work could verify based on the long-form appended prompts instead.

# 8 Discussion

We have introduced Varif.ai to help users generate images that are diverse and aligned with their defined attribute distributions. We discuss its scope, limitations, future work, and generalization.

## 8.1 Toward User-Controllable Diversity

The need for diversity has driven several methods to implicitly diversify generated images [22, 28]. However, these methods pre-determine the criteria for diversity, which may be too open-ended and exclude specific requirements from users. Varif.ai supports user-specified diversity based on attributes and targeted label distributions. However, in our formative study, some participants lacked knowledge or goals to diversify effectively when unguided; they did not utilize the feature for *suggesting attributes*, which was shown to be useful in text-based LLMs for proposing criteria to generate perspectives [29] and proposing dimensions for creative writing [64].

## 8.2 Scalability to More Attributes and Images

We explored controlling diversity with a limited number of attributes at a time. Under the open-ended specification degree, summative study participants diversified up to 9 attributes, reflecting both their preferences and time constraints. While participants in the formative study did not express a need to control many attributes, this capability could be crucial for more complex tasks, such as ensuring fairness or creating datasets. Below, we discuss
how to scale the processes of verifying and varying diversity across
a larger number of attributes.

Verifying diversity across many attributes poses a significant
interaction challenge, requiring users to interpret multiple distribu-
tions. One potential solution is progressive disclosure: presenting
a single summary metric per attribute and revealing detailed his-
tograms only upon user interaction. As in Lyu et al. [47], diversity
metrics can help users quickly assess which attributes have been
sufficiently diversified, allowing users to focus on problematic ones.
Future work should explore scalable interaction designs that sup-
port effective engagement with a large number of distributions.

Our probabilistic prompting approach is scalable to any num-
ber of attributes. However, as more attributes are sampled and
appended to the prompt, the resulting prompt length can exceed
what current image generation models can handle. Addressing this
may involve improving the capacity of the image generator or re-
fining the prompt strategy, e.g., by sampling a subset of attributes
to make the prompt brief while preserving some diversity.

Managing many attributes requires sufficient images to diver-
sify. While our main experiments were limited to 20 images, we
conducted preliminary tests with three users generating 100–200
images using Varif.ai. The images were generated within 90 sec-
onds, and participants reported no major difficulties. Histograms
and brushing interactions remained effective for verifying diversity
even with large image sets. For instance, users employed brushing
to locate images with underrepresented labels, enabling them to
verify concept coverage. Future work should explore interaction
strategies that scale with both the number of attributes and the
number of images, enabling users to effectively verify and adjust
attribute distributions without sacrificing usability.

## 8.3 Improving User Control of Image Sets

In our user studies, several participants expressed a need for more
control over image collections and improved modeling.

8.3.1 *Multimodal Inputs for Image Collection.* Participants wanted greater control over the images, including the ability to modify their own images. While text-based descriptions were useful, they were often insufficient. Text can be ambiguous, and even precise captions may produce unintended results [32]. Concepts such as layout or composition are especially difficult to convey through text alone. Although technical solutions like layout control exist [31, 81], they require new interaction designs to scale effectively for image collections. Future work should explore multimodal input mechanisms to improve user control over image collections.

8.3.2 *Control Multi-Attribute Entanglement.* From our formative study, participants desired more nuanced control over how attributes interact. This parallels the technical challenge of entanglement in image generation. Disentangling latent spaces remains a key objective in improving generative model interpretability and editability [73], and recent work has begun exposing these concepts to users through interactive interfaces [23]. Our approach treated attributes as independent, assuming that concepts were fully disentangled. To provide control over the attributes' relationship, we could explore more sophisticated visualization techniques. While histograms assume attributes behave independently, future systems

could incorporate richer visualizations—such as high-dimensional
correlation matrices [17, 21]—to reveal dependencies between at-
tributes. However, this may increase the complexity of interactions
and limit accessibility for non-expert users.

Some participants also requested the ability to exclude specific
attributes or labels from generation. A possible solution would be to
incorporate negative prompts in the user interface [6], for example
by allowing users to set the probability of a label to zero, effectively
blacklisting it from appearing in generated images.

8.3.3 Associate Attributes to Parts of the Image. Attributes served as an intuitive mechanism for verifying and varying diversity. However, some participants reported difficulty in linking attributes to parts of the image, such as changing the color of the frogs rather than the background. While rephrasing attributes (e.g., using “frog color” instead of “color”) can help, this may not generalize. Future work should explore techniques for explicitly binding attributes to image regions, such as alternative strategies for constructing probabilistic prompts beyond simple concatenation to the context.

8.3.4 *Improve Classification Accuracy for Distribution Verification.* We have exclusively used CLIP [55] for image representation, despite its known biases [1, 72]. Although CLIP provided acceptable accuracy in our tests, misclassifications can undermine trust in the visualizations and skew diversity assessments. Future work should consider more robust classification or ensemble methods, and incorporate representations sensitive to emotional tone [69] or structural similarity [37]. Enhancing classifier reliability is critical for ensuring accurate and trustworthy diversity verification.

## 8.4 Generalization to Other Models and Media

Although we have implemented Varif.ai with one image genera-
tion model and text processor, our framework is agnostic to these
underlying models. Future work can investigate using Varif.ai as a
benchmark platform to compare the diversity of image generation
models, and to assess their amenability toward varying diversity.

While our focus has been on image generation, there is a grow-
ing need to support diversity in text generation as well, such as for
improving the impact of motivational messages [16, 70], increasing
educational value by offering diverse course descriptions [38], or
supporting dataset evaluation [57]. Adapting user-defined attribute
control to text media may be feasible with interface-level adjust-
ments. Varif.ai architecture would need minimal changes to adapt
to other domains, as it remains agnostic to the underlying large
language model, image generator, and classifier. This flexibility en-
ables integration with newer or more specialized models as needed.
While we currently host open-source models on a server, Varif.ai
could also run on a variety of setups by leveraging API calls, making
it adaptable to different computing environments.

## 9 Conclusion

As AI facilitates the creation of images for effective communication,
it is important to ensure that the images are generated diversely, for
fairness and for creative ideation. Varif.ai represents a significant
advancement in enabling user-defined control over the diversity
of generative images by providing an interactive visualization for
defining, verifying, and varying attributes and their labels. The
results from our formative study demonstrate that Varif.ai allows users to diversify image sets in real use cases, and our summative study shows Varif.ai outperforms traditional prompt-engineering in terms of generated diversity and engagement. Future work can explore refining the system's scalability and extending its applicability to more types of data such as text. Varif.ai presents a promising direction for ensuring diverse, user-driven outputs, with potential for further enhancements.

## Acknowledgments

This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2023-01-040-J), and is part of the programme DesCartes which is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.

## References

[1] Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications. arXiv:2108.02818 [cs.CV] https://arxiv.org/abs/2108.02818

[2] Yongsu Ahn and Yu-Ru Lin. 2020. FairSight: Visual Analytics for Fairness in Decision Making. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2020), 1086-1095. https://doi.org/10.1109/TVCG.2019.2934262

[3] Yannick Assogba, Adam Pearce, and Madison Elliott. 2023. Large Scale Qualitative Evaluation of Generative Image Model Outputs. arXiv:2301.04518 [cs.HC] https://arxiv.org/abs/2301.04518

[4] Abolfazl Asudeh, Zhongjun Jin, and H. V. Jagadish. 2019. Assessing and Remediying Coverage for a Given Dataset. In 2019 IEEE 35th International Conference on Data Engineering (ICDE). 554-565. https://doi.org/10.1109/ICDE.2019.00056

[5] AUTOMATIC1111. 2022. Stable Diffusion Web UI. Retrieved 14 April 2025 from https://github.com/AUTOMATIC1111/stable-diffusion-webui

[6] Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Boqing Gong, and Cho-Jui Hsieh. 2024. Understanding the Impact of Negative Prompts: When and How Do They Take Effect?. In Computer Vision - ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXIX (Milan, Italy). Springer-Verlag, Berlin, Heidelberg, 190-206. https://doi.org/10.1007/978-3-031-73024-5_12

[7] Abhipsa Basu, R. Venkatesh Babu, and Danish Pruthi. 2023. Inspecting the Geographical Representativeness of Images from Text-to-Image Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 5113-5124. https://doi.org/10.1109/ICCV51070.2023.00474

[8] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023. Improving Image Generation with Better Captions. Computer Science. https://doi.org/10.1145/3593013.3594095

[9] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FACCT'23). Association for Computing Machinery, New York, NY, USA, 1493-1504. https://doi.org/10.1145/3593013.3594095

[10] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 96, 14 pages. https://doi.org/10.1145/3586183.3606725

[11] Nathan Brown, Stavros Tseranidis, and Caitlin Mueller. 2015. Multi-Objective Optimization for Diversity and Performance in Conceptual Structural Design. Proceedings of IASS Annual Symposia 2015, 20 (2015), 1-12. https://www.ingentaconnect.com/content/iass/piass/2015/00002015/00000200/art00012

[12] Angel Alexander Cabrera, Will Epperson, Fred Hohman, Minsuk Kahng, Jamie Morgenstern, and Duen Horng Chau. 2019. FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning. In 2019 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE Computer Society, Los Alamitos, CA, USA, 46-56. https://doi.org/10.1109/VAST47406.2019.8986948

[13] Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models. In 2023

IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 3020-3031. https://doi.org/10.1109/ICCV51070. 2023.00283

[14] John Joon Young Chung and Eytan Adar. 2023. Artinter: AI-powered Boundary Objects for Commissioning Visual Arts. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS '23). Association for Computing Machinery, New York, NY, USA, 1997-2018. https://doi.org/10.1145/ 3563657.3595961

[15] John Joon Young Chung and Eytan Adar. 2023. PromptPaint: Steering Text-to- Image Generation Through Paint Medium-like Interactions. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 6, 17 pages. https://doi.org/10.1145/3586183.3606777

[16] Samuel Rhys Cox, Yunlong Wang, Ashraf Abdul, Christian von der Weth, and Brian Y. Lim. 2021. Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 393, 35 pages. https://doi.org/10.1145/3411764.3445782

[17] Rene Cutura, Michaël Aupetit, Jean-Daniel Fekete, and Michael Sedlmair. 2020. Comparing and Exploring High-Dimensional Data with Dimensionality Reduction Algorithms and Matrix Visualizations. In Proceedings of the 2020 International Conference on Advanced Visual Interfaces (Salerno, Italy) (AVI '20). Association for Computing Machinery, New York, NY, USA, Article 10, 9 pages. https://doi.org/10.1145/3399715.3399875

[18] Hai Dang, Frederik Brudy, George Fitzmaurice, and Fraser Anderson. 2023. WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 63, 17 pages. https://doi.org/10.1145/3586183.3606772

[19] Prafulla Dhariwal and Alex Nichol. 2021. Diffusion models beat GANs on image synthesis. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS '21). Curran Associates Inc., Red Hook, NY, USA, Article 672, 15 pages.

[20] Steven P Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L Schwartz, and Scott R Klemmer. 2010. Parallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-efficacy. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 4 (2010), 1-24.

[21] Niklas Elmqvist, Pierre Dragicevic, and Jean-Daniel Fekete. 2008. Rolling the Dice: Multidimensional Visual Exploration using Scatterplot Matrix Navigation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1539-1148. https://doi.org/10.1109/TVCG.2008.153

[22] OpenAI et al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

[23] Noyan Evringen and Xiang 'Anthony Chen. 2023. GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 19, 15 pages. https://doi.org/10.1145/3544548.3581226

[24] Adam M. Fass, Eric A. Bier, and Eyton Adar. 2000. Picturepiper: Using a ReConfigurable Pipeline to Find Images on the Web. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (San Diego, California, USA) (UIST '00). Association for Computing Machinery, New York, NY, USA, 51-62. https://doi.org/10.1145/354401.354411

[25] Yingchaojie Feng, Xingbo Wang, Kam Kwai Wong, Sijia Wang, Yuhong Lu, Minfeng Zhu, Baicheng Wang, and Wei Chen. 2024. PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation. IEEE Transactions on Visualization and Computer Graphics 30, 1 (Jan. 2024), 295-305. https://doi.org/10.1109/TVCG.2023.3327168

[26] Joe Foley. 2024. It Seems AI Image Generators Still Have a Diversity Problem. Retrieved 14 March 2025 from https://www.creativebloq.com/news/meta-ai-image-generator-diversity

[27] Google. 2024. Gemini Image Generation Got It Wrong. We'll Do Better. Retrieved 14 March 2025 from https://blog.google/products/gemini/gemini-image-generation-issue/

[28] Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2023. Optimizing Prompts for Text-to-image Generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 2923, 17 pages. https://doi.org/10.5555/3666122.3669045

[29] Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Rajagopal, and Dongyeop Kang. 2023. How Far Can We Extract Diverse Perspectives from Large Language Models?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 5336-5366. https://doi.org/10.18653/v1/2024.emnlp-main.306
[30] Scarlett R. Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P. Bailey. 2009. Getting inspired! understanding how and why examples are used in creative design practice. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Boston, MA, USA) (CHI '09). Association for Computing Machinery, New York, NY, USA, 87–96. https://doi.org/10.1145/1518701.1518717

[31] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. 2022. Prompt-to-Prompt Image Editing with Cross-Attention Control. In *The Eleventh International Conference on Learning Representations*.

[32] Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran. 2022. Under- specification in Scene Description-to-Depiction Tasks. In *Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (Eds.). Association for Computational Linguistics, Online only, 1172–1184. https://doi.org/10.18653/v1/2022.aacl-main.86

[33] Marius Kaminskas and Derek Bridge. 2016. Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems. *ACM Trans. Interact. Intell. Syst.* 7, 1, Article 2 (Dec. 2016), 42 pages. https://doi.org/10.1145/2926720

[34] Youwen Kang, Zhida Sun, Sitong Wang, Zeyu Huang, Ziming Wu, and Xiao-juan Ma. 2021. MetaMap: Supporting Visual Morphology Ideation through Multi- dimensional Example-based Exploration. In *Proceedings of the 2021 CHI Con- ference on Human Factors in Computing Systems* (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 427, 15 pages. https://doi.org/10.1145/3411764.3445325

[35] Sari Kujala et al. 2002. *User studies: A practical approach to user involvement for gathering user needs and requirements*. Helsinki University of Technology.

[36] Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klem- mer. 2010. Designing with Interactive Example Galleries. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 2257–2266. https://doi.org/10.1145/1753326.1753667

[37] Seongwon Lee, Suhyeon Lee, Hongje Seong, and Euntai Kim. 2023. Revisiting Self-Similarity: Structural Embedding for Image Retrieval. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 23412–23421. https://doi.org/10.1109/CVPR52729.2023.02242

[38] Yoonjoo Lee, John Joon Young Chung, Tae Soo Kim, Jean Y Song, and Juho Kim. 2022. Promptverse: Scalable Generation of Scaffolding Prompts Through Human-Ai Hybrid Knowledge Graph Annotation. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*. 1–18.

[39] Joel Lehman and Kenneth O. Stanley. 2011. Abandoning Objectives: Evolution Through the Search for Novelty Alone. *Evol. Comput.* 19, 2 (June 2011), 189–223. https://doi.org/10.1162/EVCO_a_00025

[40] Shanchuan Lin, Anran Wang, and Xiao Yang. 2024. SDXL-Lightning: Progressive Adversarial Diffusion Distillation. arXiv:2402.13929 [cs.CV] https://arxiv.org/ abs/2402.13929

[41] Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems* (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 384, 23 pages. https://doi.org/10.1145/3491102.3501825

[42] Vivian Liu, Han Qiao, and Lydia Chilton. 2022. Opal: Multimodal Image Gener- ation for News Illustration. In *Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology* (Bend, OR, USA) (UIST '22). Asso- ciation for Computing Machinery, New York, NY, USA, Article 73, 17 pages. https://doi.org/10.1145/3526113.3545621

[43] Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2023. 3DALL- E: Integrating Text-to-Image AI in 3D Design Workflows. In *Proceedings of the 2023 ACM Designing Interactive Systems Conference* (Pittsburgh, PA, USA) (DIS '23). Association for Computing Machinery, New York, NY, USA, 1955–1977. https://doi.org/10.1145/3563657.3596098

[44] Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable Bias: Evaluating Societal Representations in Diffusion Models. In *Proceedings of the 37th International Conference on Neural Information Processing Systems* (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 2458, 14 pages.

[45] Andrés Lucero. 2022. Using Affinity Diagrams to Evaluate Interactive Proto- types. In *Human-Computer Interaction – INTERACT 2015*. Springer-Verlag, Berlin, Heidelberg, 231–248. https://doi.org/10.1007/978-3-319-22668-2_19

[46] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. 2022. RePaint: Inpainting using Denoising Diffusion Prob- abilistic Models. In *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 11451–11461. https://doi.org/10.1109/CVPR52688.2022.01117

[47] Yan Lyu, Hangxin Lu, Min Kyung Lee, Gerhard Schmitt, and Brian Y. Lim. 2024. IF-City: Intelligible Fair City Planning to Measure, Explain and Mitigate Inequal- ity. *IEEE Transactions on Visualization and Computer Graphics* 30, 7 (July 2024), 3749–3766. https://doi.org/10.1109/TVCG.2023.3239909

[48] Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. 2024. Training Diffusion Models Towards Diverse Image Gen- eration with Reinforcement Learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 10844–10853.

[49] Mohammad Amin Mozaffari, Xinyuan Zhang, Jinghui Cheng, and Jin L.C. Guo. 2022. GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network. In *Proceed- ings of the 2022 CHI Conference on Human Factors in Computing Systems* (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 537, 15 pages. https://doi.org/10.1145/3491102.3517511

[50] Matt O'Brien. 2024. Google Says Its AI Image-Generator Would Sometimes 'overcompensate' for Diversity. Retrieved 14 March 2025 from https://finance. yahoo.com/news/google-says-ai-image-generator-225023065.html

[51] Jonas Oppenlaender. 2024. A Taxonomy of Prompt Modifiers for Text-to-image Generation. *Behaviour & Information Technology* 43, 15 (2024), 3763–3776. https://doi.org/10.1080/0144929X.2023.2286532

[52] Xiaohan Peng, Janin Koch, and Wendy E. Mackay. 2024. DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI. In *Proceedings of the 2024 ACM Designing Interactive Systems Conference* (Copenhagen, Denmark) (DIS '24). Association for Computing Machinery, New York, NY, USA, 804–818. https://doi.org/10.1145/3643834.3661588

[53] Owen L. Petchey and Kevin J. Gaston. 2002. Functional diversity (FD), species richness and community composition. *Ecology Letters* 5, 3 (2002), 402–411. https://doi.org/10.1046/j.1461-0248.2002.00339.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1046/j.1461-0248.2002.00339.x

[54] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In *The Twelfth International Confer- ence on Learning Representations*. https://openreview.net/forum?id=di5zR8xgf

[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhi Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In *Proceedings of the 38th Inter- national Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139)*, Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763. https://proceedings.mlr press/v139/rafdord21a.html

[56] Aditya Ramesh, Prafulla Dharwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. *arXiv preprint arXiv:2204.06125* (2022). https://doi.org/10.48550/arXiv.2204.06125

[57] Emily Reif, Minsuk Kahng, and Savvas Petridis. 2023. Visualizing Linguistic Diversity of Text Datasets Synthesized by Large Language Models. In *2023 IEEE Visualization and Visual Analytics (VIS)*. IEEE Computer Society, Los Alamitos, CA, USA, 236–240. https://doi.org/10.1109/VIS54172.2023.00056

[58] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE Computer Society, Los Alamitos, CA, USA, 10674–10685. https://doi.org/10.1109/ CVPR52688.2022.01042

[59] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. 2024. Generating images of rare concepts using pre-trained diffusion mod- els. In *Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelli- gence and Thirty-Sixth Conference on Innovative Applications of Artificial Intel- ligence and Fourteenth Symposium on Educational Advances in Artificial Intel- ligence (AAAI'24/IAAI'24/EAAI'24)*. AAAI Press, Article 522, 9 pages. https://doi.org/10.1609/aaai.v38i5.28270

[60] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. 2023. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 22522–22531. https://doi.org/10.1109/CVPR52729.2023.02157

[61] Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. 2023. Finetuning Text-to-image Diffusion Models for Fairness. *arXiv preprint arXiv:2311.07604* (2023).

[62] Pedro Silva, Bhawna Juneja, Shloka Desai, Ashudeep Singh, and Nadia Fawaz. 2023. Representation Online Matters: Practical End-to-End Diversification in Search and Recommender Systems. In *Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency* (Chicago, IL, USA) (FACCT '23). Association for Computing Machinery, New York, NY, USA, 1735–1746. https://doi.org/10.1145/3593013.3594112

[63] Kihoon Son, DaEun Choi, Tae Soo Kim, Young-Ho Kim, and Juho Kim. 2024. GenQuery: Supporting Expressive Visual Search with Generative Models. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems* (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 180, 19 pages. https://doi.org/10.1145/3613904.3642847

[64] Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems* (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 644,
26 pages. https://doi.org/10.1145/3613904.3642400

[65] Gareth Terry, Nikki Hayfield, Victoria Clarke, Virginia Braun, et al. 2017. The- matic analysis. *The SAGE handbook of qualitative research in psychology* 2, 17-37 (2017), 25.

[66] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL] https://arxiv.org/abs/2307.09288

[67] Samang Wadinambiarachchi, Ryan M. Kelly, Saumya Pareek, Qiushi Zhou, and Eduardo Velloso. 2024. The Effects of Generative AI on Design Fixation and Divergent Thinking. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems* (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 380, 18 pages. https://doi.org/10.1145/3613904.3642919

[68] Jialu Wang, Xinyue Liu, Zonglin Di, Yang Liu, and Xin Wang. 2023. T2IAT: Mea- suring Valence and Stereotypical Biases in Text-to-Image Generation. In *Findings of the Association for Computational Linguistics: ACL 2023*, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Lin- guistics, Toronto, Canada, 2560–2574. https://doi.org/10.18653/v1/2023.findings- ac160

[69] Yunlong Wang, Shuyuan Shen, and Brian Y Lim. 2023. RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23)*. Association for Computing Machinery, New York, NY, USA, Article 22, 29 pages. https://doi.org/10.1145/3544548.3581402

[70] Yunlong Wang, Priyadarshini Venkatesh, and Brian Y Lim. 2022. Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22)*. Association for Computing Machinery, New York, NY, USA, Article 183, 28 pages. https://doi.org/10.1145/3491102.3517551

[71] Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. 2024. PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24)*. Association for Computing Machinery, New York, NY, USA, Article 185, 21 pages. https://doi.org/10.1145/3613904.3642803

[72] Robert Wolfe and Aylin Caliskan. 2022. American = White in Multimodal Language-and-Image AI. In *Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society* (Oxford, United Kingdom) (AIES '22). Association for Computing Machinery, New York, NY, USA, 800–812. https://doi.org/10.1145/3514094.3534136

[73] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. 2023. Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 1900–1910. https://doi.org/10.1109/CVPR52729.2023.00189

[74] Peipei Xia, Li Zhang, and Fanzhang Li. 2015. Learning Similarity with Cosine Similarity ensemble. *Inf. Sci.* 307, C (June 2015), 39–52. https://doi.org/10.1016/j.ins.2015.02.024

[75] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. 2023. Diffusion Models: A Comprehensive Survey of Methods and Applications. *ACM Comput. Surv.* 56, 4, Article 105 (Nov. 2023), 39 pages. https://doi.org/10.1145/3626235

[76] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23)*. Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/3544548.3581388

[77] Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, and Fernando De La Torre. 2023. ITI-Gen: Inclusive Text-to-Image Gen- eration. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*. 3946–3957. https://doi.org/10.1109/ICCV51070.2023.00367

[78] Enhao Zhang and Nikola Banovic. 2021. Method for Exploring Generative Adversarial Networks (GANs) via Automatically Generated Image Galleries. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems*

(Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 76, 15 pages. https://doi.org/10.1145/3411764.3445714

[79] Lili Zhang, Xi Liao, Zaijia Yang, Baihang Gao, Chunjie Wang, Qiuling Yang, and Deshun Li. 2024. Partiality and Misconception: Investigating Cultural Represen- tativeness in Text-to-Image Models. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24)*. Asso- ciation for Computing Machinery, New York, NY, USA, Article 620, 25 pages. https://doi.org/10.1145/3613904.3642877

[80] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K. Wong. 2023. Uni-ControlNet: all-in-one control to text- to-image diffusion models. In *Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23)*. Curran Associates Inc., Red Hook, NY, USA, Article 491, 24 pages.

[81] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. 2023. LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Gener- ation. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 22490–22499. https://doi.org/10.1109/CVPR52729.2023.02154

# A Prompts

## A.1 Prompts for Suggesting Attribute Labels.

**System Prompt:** "You are a useful assistant. You give very brief answers, in very few words, no need to be polite, do not provide explanations."

**Instruction prompt:** "For the extracted attribute $t_a$ in the context of $t_0$, suggest possible labels for the attribute and provide a precise definition. Consider general knowledge or common scenarios for accuracy. Example: for the attribute 'color' in the context of 'sky', some possible labels are blue, white, grey, orange, and red. For the attribute 'ethnicity' in the context of 'person', some possible labels are Caucasian, Black, Asian, Hispanic, and Middle Eastern."

**Answer template:** Here are $n_a$ possible labels of attribute $t_a$ in the context of $t_0$: for i in range(n-a): "i." + [GENERATE 4 TOKENS]

## A.2 Prompts for Suggesting Attributes.

**System Prompt:** "You are a useful assistant. You give very brief answers, in very few words, no need to be polite, do not provide explanations."

**Instruction prompt:** "For the context of $t_0$, suggest possible at- tributes to diversify. Consider general knowledge or common sce- narios for accuracy."

**Answer template:** Here are 3 possible attributes $t_a$ in the context of $t_0$: for i in range(n-a): "i." + [GENERATE 4 TOKENS]

# B Additional Results of Participants

Fig. 14, 15, 16 show participants' image collections for the scenarios of the summative study. Fig. 17 shows participant iterative prompt- engineering during the Prompt-only condition in the summative study.
<figure><img src="image_16.png" /><figcaption>Figure 14: Examples of iterations for different conditions on the *Birds* scenario.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 15: Examples of iterations for different conditions on the *Cars* scenario.</figcaption></figure>
<figure><img src="image_18.png" /><figcaption>Figure 16: Examples of iterations for different conditions on the *Doctors* scenario.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 17: Three examples of prompts iterations for each scenario for the technique Prompt-only. Modifications from one iteration to another are highlighted in green.</figcaption></figure>