StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images

## Abstract

Abstract Understanding the semantics of visual scenes is a fundamental challenge in Computer Vision. A key aspect of this challenge is that objects sharing similar semantic meanings or functions can exhibit striking visual differences, making accurate identification and categorization difficult. Recent advancements in text-to-image frameworks have led to models that implicitly capture natural scene statistics. These frameworks account for the visual variability of objects, as well as complex object co-occurrences and sources of noise such as diverse lighting conditions. By leveraging large-scale datasets and cross-attention conditioning, these models generate detailed and contextually rich scene representations. This capability opens new avenues for improving object recognition and scene understanding in varied and challenging environments. Our work presents StableSemantics , a dataset comprising 224 thousand human-curated prompts, processed natural language captions, over 2 million synthetic images, and 10 million attention maps corresponding to individual noun chunks. We explicitly leverage human-generated prompts that correspond to visually interesting stable diffusion generations, provide 10 generations per phrase, and extract cross-attention maps for each image. We explore the semantic distribution of generated images, examine the distribution of objects within images, and benchmark captioning and open vocabulary segmentation methods on our data. To the best of our knowledge, we are the first to release a diffusion dataset with semantic attributions. We expect our proposed dataset to catalyze advances in visual semantic understanding and provide a foundation for developing more sophisticated and effective visual models. https://stablesemantics.github.io/StableSemantics/

## 1 Introduction

Visual scene understanding is a complex task that requires the integration of cues, context, and prior knowledge to navigate the inherent variability and complexity of the visual world. This complexity is particularly evident when considering the diversity of visual appearances that can correspond to a single semantic concept. For instance, entities that correspond to "man-made structures" can have vastly different visual appearances, ranging from sleek skyscrapers to rustic cottages. Similarly, objects that serve the same purpose, such as "containers," can have diverse shapes, sizes, and materials. This disconnect between semantic meaning and visual appearance poses a significant challenge for Computer Vision systems [, , , ], requiring the disentanglement of the underlying semantic structure from visual differences [, , , , , ]. To overcome this challenge, recent advances in Computer Vision have adopted data-driven approaches, which learn to recognize patterns and relationships in large datasets of images and annotations. However, the reliance on large datasets of images and annotations poses a significant challenge in the development of Computer Vision systems. Acquiring and annotating such datasets can be a time-consuming and resource-intensive process, requiring careful consideration of data quality and diversity.

This limitation has sparked interest in exploring alternative approaches that can reduce the need for large human-annotated datasets. One promising direction is the use of generative models, which have shown impressive results in translating between semantic meaning and visual appearance [, , , ]. In particular, diffusion-based text-to-image synthesis models have demonstrated an impressive ability to generate highly realistic images from textual descriptions, suggesting that these models must possess an implicit understanding of the semantic structure of the visual world, and have learned to associate words and phrases with specific visual concepts. By leveraging cross-attention mechanisms, these models learn to link textual input to visual representations and enable the generation of images that are grounded in the semantic content of the input text [].

Figure: Figure 1: Images and maps corresponding to select noun chunks from StableSemantics. Images are generated using natural language captions derived from human generated and curated prompts. For reproducibility, seeds are recorded for each generation. Noun chunks are extracted by performing dependency parsing the natural language captions. Semantic maps corresponding to each noun chunk is computed using the cross-attention maps with the DAAM [] method. Only a single attention map is shown here for each image, please see below for additional examples. Yellow indicates high relevance, black indicates low relevance.
Refer to caption: x1.png

In this work, we introduce StableSemantics, a dataset that consists of human-generated and curated prompts, natural language captions, images generated from the captions, and attention attribution maps corresponding to objects in the captions. Unlike prior work which sourced unfiltered human-generated prompts, we source our prompts from a pool of images that have been evaluated by humans for their visual appeal and interest, resulting in a dataset that mirrors the types of images people find engaging. As the original prompts may not always reflect natural language, we employ a large language model to paraphrase and refine them into fluent and natural-sounding captions, thereby bridging the gap between human-generated prompts filtered for visual appeal and naturalistic language. Each natural language caption is provided to a Stable Diffusion XL model to generate high-resolution and reproducible images. Finally, we explicitly record the dense text-to-image cross-attention maps used to condition the image generation process. We visualize the distribution of semantics across images, evaluate the spatial distribution of semantic classes within images, and evaluate the alignment of current captioning and open-set segmentation models on our dataset. To our knowledge, our dataset is the first to systematically record the spatial distribution of cross-attention activations corresponding to individual noun chunks. We hope that StableSemantics will inspire future research on the visual distribution of semantic concepts and the development of more interpretable text-to-image synthesis models.

## 2 Related work

#### Natural Scene Statistics.

Natural image statistics have been a long-standing area of research in computer vision and neuroscience. The human visual system is thought to be adapted to the statistical properties of natural images, which are characterized by complex dependencies between pixels [, ]. The power law distribution of gradient magnitude statistics is thought to be a result of the hierarchical, self-similar structure of natural images, which arises from the presence of edges, textures, and other features at multiple scales. Understanding natural image statistics has important implications for image recognition tasks [, , ], and has inspired the development of a range of algorithms and models that are tailored to the statistical properties of natural images [, , , , , ]. Other work has also explored the semantic structure of visual data, seeking to understand how higher-level categories and concepts are reflected in the statistical patterns present in images. This work has shown that different categories of images, such as scenes and objects, exhibit distinct statistical patterns [, ]. These semantic statistics have important implications for the development of models that can effectively represent and analyze visual data.

#### Deep Image Generative Models.

Recent progress on generative models has enabled the generation of images, video, text, and audio [, , , , , , , , ]. Models rely on a variety of different mathematical assumptions and architectures. Variational autoencoders [, , , , ] and flow-based models [, , , ], while highly efficient, tend to produce lower-quality samples. Generative Adversarial Networks (GANs) [, , , , , , ] can yield high-fidelity samples but may neglect modes in the data and can exhibit unstable training dynamics. Auto-regressive methods [, , , , ], although capable of producing high-quality samples, typically experience slow sampling. Recent progress in energy/score/diffusion models [, ] has given us methods that are simultaneously stable during training and yield high-quality samples.

#### Visual Datasets.

Deep learning models have achieved remarkable results by leveraging vast amounts of data. There has been a significant push to collect large-scale datasets. Earlier works such as LAION-5B [], Flickr Caption [] and YFCC100M [] scrape real-world data of image-caption pairs from web sources. COCO [] goes a step further to also provide pixel-level segmentation masks on top of the image-caption pairs. [, , , , ] introduce datasets specifically for the task of VQA. Given the difficulty of collecting real data, recently there has been a shift towards synthetic datasets. StableRep [] also demonstrated the usefulness of Stable Diffusion images in training contrastive image models. Pick-a-Pic [] provides a dataset of image-caption pairs where each sample contains a pair of diffusion-generated images and the human preference between those images. JourneyDB [] and DiffusionDB [] are the closest works to ours that release large-scale datasets of synthetic image-caption pairs.

## 3 Data collection

In this section, we provide details on the collection and creation process of our dataset. Our data originates from human-generated and curated prompts submitted publicly by users online for Stable Diffusion XL. We describe our prompt collection process in section 3.1. The prompts are filtered and transformed into natural language captions, and we describe our procedure in section 3.2. Finally, we generate images and compute noun-chunk to image saliency maps via cross-attention in section 3.

Figure: Figure 2: Data collection and generation process. (1) We collect our data from Stable Diffusion Discord, specifically the showdown and pantheon channels which are derived from user rankings of images generated from public prompt submissions. (2) The prompts are cleaned using regex to remove common errors, and further processed using an LLM to generate natural language captions. (3) The natural language captions are provided to a Stable Diffusion XL model, while we record the attention attribution maps corresponding to noun chunks.
Refer to caption: x2.png

### 3.1 Collecting human curated prompts

Our dataset is collected from the Stable Diffusion discord server, where users can publicly submit prompts to generate images using a discord bot. After users submitted prompts using the /dream command, the bot would return images corresponding to a prompt. Beyond accepting a prompt, users could also submit negative prompts, and image styles which were achieved via a prefix/affix pattern of text to the original prompt. These style patterns were not visible to the users.

We started our data collection after the Stable Diffusion XL 1.0 [] candidate was made available via bots. The data was continuously collected from July 11, 2023 (a day after SDXL 1.0 candidate bots were launched) until Feb 07, 2024 (SDXL bot shutdown). Users were allowed to submit prompts to bot-# channels where # corresponds to a number. We observed that the number of channels varied over time, and generally remained at slightly over $10$. For each prompt, the bot would return $2$ images. Users were asked to select which image was better by clicking on a button corresponding to an image, without explicit guidance on what "better" meant. Our understanding from discussions with members of staff was that these prompts and image pairs were used for fine-tuning the SDXL candidates using RLHF/DPO  [, ], selection of model candidates, and selection of generation hyperparameters.

Prior work has also collected user-generated prompts from discord servers for MidJourney and Stable Diffusion [, ]. Our work goes further by only collecting prompts that were human-curated. The Stable Diffusion discord followed a three-tier hierarchy for prompts, where users first submit and rate images in bot-#, with highly rated images from all bot channels going into a single showdown channel every $15$ minutes. The showdown channel was reset every $30$ minutes and had the history wiped. In the showdown channel, 2 images and their respective prompts were placed side-by-side. Users again were asked to select the images that were more visually appealing. Every $30$ minutes, the top-ranked images and prompts would go into the pantheon channel. The pantheon had history going back to inception May 02, 2023. We note that strictly speaking the showdown to pantheon selection process was not fair to images that came in at the second $15$ minute slice, as they were given less time to be voted upon. Due to this, we do not further distinguish between prompts collected from these two sources. Our data collection process ran every $14$ minutes on the showdown channels and ran once on the pantheon channel. This was sufficient as after the initial collection date, new pantheon entries were a strict subset of showdown prompts. Visual inspection of the generations from showdown and pantheon suggest that these images were generally more artistic and contained more interesting visual compositions than the bot-# channel. We collect a total of 235k unique user-generated prompts, which is further filtered according to NSFW ratings and caption length.

### 3.2 Obtaining natural language captions

As shown in Figure 4, the user-submitted prompts generally took a tag-like format, with descriptors being separated by commas. Such prompts are convenient for users to specify and likely achieve good results due to the use of CLIP text networks for conditioning, which can operate like bag-of-words models [, ]. However, such prompts generally perform poorly when typical NLP pipelines are used for analysis. These prompts further may not explicitly specify needed visual relationships in the text, and instead excessively rely on the prior learned by the diffusion model to disambiguate relationships.
In order to mitigate this issue, we utilize an LLM model to clean up the original raw user-generated prompts. We use Gemini 1.0 Pro for this task, as it performed competitively against other models at the time of our work [] and offered a free API. The model was instructed to take the user-generated prompts and transform them into natural language captions. To enhance the results, we augment the prompt via in-context learning from GPT-4 input/output pairs. To remove NSFW prompts, we record the Gemini API safety ratings for each input prompt, and remove the prompts where a 4 out of 4 rating was given on the axes of sexuality/hate speech/harassment, or if the model itself produced a refusal, or if the prompt was repeatedly returned with an error (blocked by Google). Please see Figure 2 for a visualization of the pipeline.

### 3.3 Image generation and semantic attribution

**Table 1: Size of the different components of StableSemantics. Our captions are selected by humans to correspond to visually interesting images. We are the only dataset to provide dense open-set spatial semantic maps. Our maps are derived from the cross-attention maps in Stable prompts. Note that 235k unique captions are collected, 224k remain after NSFW filtering and only 200k captions are used for image generation after filtering for length.**
| \addstackgapDataset | Total Images | Total Captions | Caption Source | Human Preferred | Open-set Semantics |
| --- | --- | --- | --- | --- | --- |
| COCO 2017 [] | \addstackgap123k | 617k | H | N | N |
| LAION-COCO [] | 600M | 600M | M | N | N |
| DiffusionDB [] | 14M | 1.8M | H | N | N |
| JourneyDB [] | 4.7M | 1.7M | H+M | N | N |
| StableSemantics | 2M | 224k | H+M | Y | 10.8M |

Figure: Figure 3: Histogram of dataset statistics. (a) We visualize the cosine CLIP similarities between generated images and original captions. (b) Number of tokens in the captions. (c) NSFW scores of the captions after LLM filtering. Scores measured by LLaMA Guard 2 for sexuality and hate.
Refer to caption: x3.png

To provide a fully reproducible pipeline for the images and maximize the usability of our dataset, we generate the images ourselves using open weights and record the random seed for each generation. Images are generated using sdxl lightning 4step unet [], a few-step distilled version of Stable Diffusion XL. For each prompt, we perform parsing using spaCy en_core_web_lg to extract noun chunks. To obtain mappings from noun chunks to spatial attributions, we use Diffusion Attentive Attribution Maps which measures the cross-attention from tokens in the language condition to the UNet. Specifically, we used the improved DAAM-i2i guided heatmap variant [, ] which improves object localization. We observe that unrelated articles like "a", and "the" and possessive determiners like "his", "her", "our", "their" are not typically localized to a specific object, but rather have attribution maps diffuse over the background or various objects in the image. While similar phenomena has been noted in ViTs [] and pure text LLMs [, , ], our observation is novel in that text-to-image diffusion models are encoding contextual information in these "filler" words. For effective localization, we remove articles and possessive determiners if they are the first word of a noun chunk.

## 4 Dataset exploration

Figure: Figure 4: Example of SDXL generated images from the captions, raw user prompts and LLM processed captions. Raw prompts from users often contain typos or take the form a non-natural language tag-like format. We instruct an LLM to transform the prompts into a natural language caption. Noun chunks (bolded and underlined) are derived from dependency parsing. Images are generated from the captions, with diffusion attribution maps recorded for the noun chunks.
Refer to caption: x4.png

We first evaluate the CLIP similarity between the generated images and the captions, and further characterize the safety and length of the captions. We then explore the semantic distribution of the images and the captions using CLIP, and visualize the spatial distribution of objects in a scene using diffusion attribution maps. Finally, we evaluate the performance of captioning and open-set segmentation models on our dataset. These characterizations demonstrate how StableSemantics can be a promising dataset for advancing visual semantic understanding. The data will be released under a CC0 1.0 license.

### 4.1 Dataset characterization

After deduplication and LLM NSFW filtering, we have 224 thousand natural language captions. We evaluate the similarity of the SDXL-lightning generated images and the captions in Figure 3a using OpenAI’s CLIP ViT-B/16 []. We find that the CLIP similarity peaks at $0.34$, which is similar to CLIP scores achieved using SDXL. These scores typically range from $0.2$ to $0.5$ which means that our prompts and images can be interpreted to be semantically very similar given the higher range of scores. We visualize the token length of the captions in Figure 3b. Note that we do not generate images for captions exceeding $77$ tokens post-padding. This yields a total of 200k captions which are used for image generation. In Figure 3c, we plot the NSFW scores of the captions used for image generation, as evaluated using the state-of-the-art Meta Llama Guard 2 model. We define the unsafe categories to the 3 official categories relating to sexual content, and the 1 official category related to hate speech. The scores are the "unsafe" softmax outputs between the "safe" and "unsafe" tokens. We find that the captions used for generation are overwhelmingly safe. In Figure 4, we provide examples of the images, the original human generated prompts which may often contain typos or tags, and the LLM output natural language prompts. Likely due to human preference, we observe a higher ratio of images with visually interesting compositions.

Figure: Figure 5: Visualization of the dataset. We show example captions used for image generation, images generated from the captions, and select noun chunks and their corresponding attention attribution maps. We find that our dataset contains accurate localizations for different semantic concepts.
Refer to caption: x5.png

Figure: Figure 6: UMAP visualization of dataset CLIP embeddings. We use OpenAI CLIP ViT-B/16 to compute embeddings for both the generated (a) images and the (b) text. UMAP with the cosine metric is used to perform dimensionality reduction. We observe that images describing people, scenes, text, and animals occur with high frequency.
Refer to caption: x6.png

### 4.2 Semantic exploration of the dataset

In Figure 6 we visualize the semantic distribution of whole images and the captions used to generate the images. We utilize UMAP [] with a cosine metric applied to CLIP embeddings for this visualization with wizmap []. We find that the distribution of both images and text exhibit peaks in concepts such as people, scenes, text, and animals (cats and dogs). These peaks likely reflect the effect of human preference on visually interesting images.

The semantic maps we provide in our dataset help localize image regions corresponding to specific noun chunks from the prompts. In Figure 5 we visualize the captions used from image generation, the generated RGB image, and attention attribution masks corresponding to noun chunks shown in bold. We find that our dataset can provide semantic attributions that are well aligned to objects in the scene. This is likely due to the nature of Stable Diffusion, which leverages cross-attention guidance to generate complex compositional images.

Figure: Figure 7: Spatial distribution of semantic concepts. For each concept, we visualize an example image containing the concept, as well as the spatial distribution averaged over occurrences. We utilize CLIP text similarity to select the top-100 most similar noun chunks and average those occurrences.
Refer to caption: x7.png

We use these maps to analyze whether our images exhibit a trend of certain concepts being generated in specific regions on average. In Figure 7 we aggregate the masks of the top 100 noun chunks that have the highest CLIP similarity scores with concepts of interest. We apply this similarity based matching to allow for inexact matching. Figure 7 clearly shows that the spatial distribution of concepts can be highly non-uniform. This bias likely reflects the distribution of concepts in natural images [, ]. For instance, it makes sense for the sunset to always be on the top, the walls towards the sides, and the floor towards the bottom. It is also very common to find human beings as the primary subjects in images which explains the placement of hair surrounding a central region. Finally, many images exhibit food on top of a table and cars on roads. In these scenarios, these semantic concepts typically occupy the bottom half of the visual field. We visualize [] the frequency distribution of nouns grouped by wordnet hierarchy in Figure 8. Our dataset could be used to understand the spatial and visual bias present in natural images.

Figure: Figure 8: Frequency of nouns visualized with wordnet hierarchy. We parse the sentences and extract the nouns. The hierarchy is from wordnet []. The circle size corresponds to frequency.
Refer to caption: x8.png

### 4.3 Evaluation of models

In this section we evaluate the performance of state-of-the-art open-vocabulary image segmentation and captioning models on our dataset.
For open-vocabulary segmentation methods, we evaluate the standard mean Intersection over Union (mIOU), where discrete masks are computed by taking the argmax over all noun chunks’ continuous masks for a given prompt. As these methods also produce soft masks, we also evaluate the pearson correlation of the attribution maps from our datasets. In Table 3, We find that recent open-vocabulary segmentation models which modify CLIP (LSeg, SCLIP) or leverage text-to-image diffusion models (ODISE) perform better than their peers.

We also evaluate recent vision-language models for image captioning in Table 3. We evaluate the generated captions against the original captions using state-of-the-art E5-Mistral model to evaluate cosine similarity ($\times 100$ for clarity), BLEU-4, and CIDEr scores. These results suggest that while the captions predicted by models may use different wording from the original caption, they can be semantically very similar.

## 5 Discussion

Limitations and Future Work. Our work relies on human-submitted prompts, which may exhibit non-natural semantic co-occurrences. During the data collection process, we also observed a strong shift in the semantic distribution of prompts and images around holidays (Thanksgiving, Christmas). This suggests that continual data collection is required to mitigate bias.
Conclusion. We introduce StableSemantics, the first large-scale dataset that combines natural language captions, synthetic images, and diffusion attribution maps. Our work goes beyond prior datasets by providing spatially localized noun chunk to image region mappings. We explore the semantic distribution of whole images and objects within an image. The availability of this dataset will allow for the use of synthetic visual data in additional domains.

## A.1 Additional dataset visualization

In this section, we provide additional visualizations of the natural language captions generated using a large language model from the raw user prompts, the RGB image, and semantic masks corresponding to select noun chunks in the image in Figure S.1 and Figure S.2.

Figure: Figure S.1: Visualization of additional dataset examples. We show the natural language caption used for the image generation, the image, and masks corresponding to select noun chunks.
Refer to caption: x9.png

Figure: Figure S.2: Visualization of additional dataset examples. We show the natural language caption used for the image generation, the image, and masks corresponding to select noun chunks.
Refer to caption: x10.png

## A.2 Visualization of object distributions

In this section, we visualize the spatial distribution of various noun chunks in Figure S.3. We note that several types of objects exhibit highly non-uniform spatial distributions.

Figure: Figure S.3: Additional examples on the spatial distribution of concepts. We provide additional examples of images containing a concept, and the average distribution of the top-100 images containing the most similar noun chunks as evaluated using the CLIP text model.
Refer to caption: x11.png

## A.3 Comparison of open vocabulary segmentation methods

In this section, we provide additional visualizations of the semantic maps from our dataset, and segmentation outputs in Figure S.4 and Figure S.5. We note that in general, the semantic masks in our dataset can accurately localize objects. However, there are specific cases (Lone Man) where the attention maps corresponding to noun chunks can include other contextual objects. We believe this occurs when the diffusion model tries to generate co-occurring scene and image parts that are not explicitly mentioned in the caption.

Figure: Figure S.4: Comparison of semantic maps and open vocabulary segmentation methods. We visualize the semantic attribution maps corresponding to noun chunks from our dataset, and the segmentation maps produced by various state-of-the-art methods.
Refer to caption: x12.png

Figure: Figure S.5: Comparison of semantic maps and open vocabulary segmentation methods. We visualize the semantic attribution maps corresponding to noun chunks from our dataset, and the segmentation maps produced by various state-of-the-art methods. Note that the Lone Man illustrates how the semantic attribution maps can be imperfect. In this case it includes additional background.
Refer to caption: x13.png

## A.4 Prompt used for language model cleanup of raw prompts

We utilize the following prompt followed by the raw user prompt to obtain a natural language caption. Note that our raw prompts undergo simple regex based processing to remove some obvious errors before being provided to the language model.
You are going to be provided with the description of an image. You will transform and edit the description as needed into a cohesive natural language sentence or sentences without elaborating. If the original is mixed language, then your output should also be mixed language. Do not elaborate, do not provide information about people mentioned in the description.
Make a best effort to use ALL WORDS AND DETAILS from the original description. DO NOT MAKE UP DETAILS unless absolutely necessary. BE AS CONCISE AS POSSIBLE, WHILE ATTEMPTING TO INCLUDE ALL WORDS AND DETAILS FROM THE ORIGINAL DESCRIPTION. You may only omit details or words if they are nonsensical or form a contradiction. Attempt to fix typos and remove invalid punctuation. Omit emojis in your output.
When you output, use [START] before the output, and include [END] after the output. You may retain hash tags in the output only if hash tags were used in the original prompt. Here is the original description:

## A.5 Dataset documentation

This document is based on Datasheets for Datasets by Gebru et
al. [].

[linecolor=violet]

## MOTIVATION

For what purpose was the dataset created?
Was there a specific task in mind? Was there
a specific gap that needed to be filled? Please provide a description.
This dataset consists of raw human-generated prompts that underwent human curation (via human voting to the showdown and pantheon channels), natural language captions generated using a large-language model on these prompts, multiple images generated using the natural langauge captions, and the semantic maps from diffusion cross-attention maps for the noun chunks parsed from the captions. To our knowledge, no other dataset has systematically recorded the cross-attention maps from a text-to-image generative model. Our dataset provides a basis for improving open vocabulary segmentation methods and for contrastive learning by leveraging the multiple images generated for each caption.

Who created this dataset (e.g., which team, research group) and on behalf
of which entity (e.g., company, institution, organization)?
This dataset was created by a team from Carnegie Mellon University and leveraged CMU computational resources for the image generation.

What support was needed to make this dataset?
(e.g.who funded the creation of the dataset? If there is an associated
grant, provide the name of the grantor and the grant name and number, or if
it was supported by a company or government agency, give those details.)
No specific grant was used. We utilized GPU computational resources available to CMU students for image generation.

[linecolor=violet]

## COMPOSITION

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)?
Are there multiple types of instances (e.g., movies, users, and ratings;
people and interactions between them; nodes and edges)? Please provide a
description.
The dataset is composed of the raw prompts generated by humans, the natural language captions derived from those prompts using a large language model, the ten images for each caption generated using a Stable Diffusion XL based model, and the individual semantic maps corresponding to the noun chunks for each caption for each generated image.

How many instances are there in total (of each type, if appropriate)?
There are 10 instances of generated images for each caption. Each of these captions has an average of  5 noun chunks and thus 5 semantic maps for a generated image. Thus, on average, there are about 10 generated images and 50 semantic maps for each prompt. After NSFW and caption length filtering, 200 thousand such captions are used for image generation. Hence, about 2 million generated images and about 10 million semantic maps. We also share the raw versions of the prompts, thus there are also 224 thousand raw prompts (post NSFW filtering).

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set?
If the dataset is a sample, then what is the larger set? Is the sample
representative of the larger set (e.g., geographic coverage)? If so, please
describe how this representativeness was validated/verified. If it is not
representative of the larger set, please describe why not (e.g., to cover a
more diverse range of instances, because instances were withheld or
unavailable).
We process the dataset to remove NSFW prompts and captions that exceed the Stable Diffusion XL 77 token limit from the image generation process.

What data does each instance consist of?
“Raw” data (e.g., unprocessed text or images) or features? In either case,
please provide a description.
The generated image instances and the semantic maps are of "jpg" type. Both the raw prompts and natural language prompts are stored in a "pickle" file.

Is there a label or target associated with each instance?
If so, please provide a description.
The generated image, corresponding semantic maps, the corresponding natural language text prompt, the raw prompt are associated with a unique hash key. This key helps identify and extract the corresponding instances. There are also "pickle" files present for each image that help identify the semantic map for a given noun chunk in the prompt. The seed used for image generation is also saved.

Is any information missing from individual instances?
If so, please provide a description, explaining why this information is
missing (e.g., because it was unavailable). This does not include
intentionally removed information, but might include, e.g., redacted text.
We perform some simple heuristic-based filtering to remove certain noun chunks corresponding to non-physical objects. We only save the semantic maps for the remaining noun chunks. However, we will also provide the original captions so all noun chunks can be extracted if needed.

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)?
If so, please describe how these relationships are made explicit.
Yes, the relations are made explicit. The instances of generated images, prompts, and semantic maps are linked by a unique hash key. The relation between noun chunks and their corresponding semantic maps can be identified with the provided "offset" pickle file.

Are there recommended data splits (e.g., training, development/validation,
testing)?
If so, please provide a description of these splits, explaining the
rationale behind them.
No, there are no recommended splits. The user is free to split the data as per their need.

Are there any errors, sources of noise, or redundancies in the dataset?
If so, please provide a description.
We observe that some raw prompts are highly similar, which seems to happen as a user quickly iterates on a prompt and creates prompts with a similar theme. We do not explicitly filter or remove such conceptually similar prompts. We also note that large language models are not perfect, and may create errors or deviations of the generated caption from the raw user prompts.

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?
If it links to or relies on external resources, a) are there guarantees
that they will exist, and remain constant, over time; b) are there official
archival versions of the complete dataset (i.e., including the external
resources as they existed at the time the dataset was created); c) are
there any restrictions (e.g., licenses, fees) associated with any of the
external resources that might apply to a future user? Please provide
descriptions of all external resources and any restrictions associated with
them, as well as links or other access points, as appropriate.
It is self-contained and doesn’t depend on any other resource. The resources used to create/modify the dataset have been mentioned above, however, they are no longer needed to use the dataset.

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ non-public
communications)?
If so, please provide a description.
No, the data was created by using publicly available information and doesn’t involve any confidential or non-public information.

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety?
If so, please describe why.
We have used Gemini 1.0 Pro to process the dataset to ensure that there are no harmful or offensive prompts. The prompts were then used to generate images from Stable Diffusion XL, which has an NSFW filter. The data is however subject to the failure of these models.

Does the dataset relate to people?
If not, you may skip the remaining questions in this section.
As the prompts in the dataset were extracted from human written prompts in a discord server, the images generated and the prompts in the data may contain information about people.

Does the dataset identify any subpopulations (e.g., by age, gender)?
If so, please describe how these subpopulations are identified and
provide a description of their respective distributions within the dataset.
The dataset is not targeted towards any subpopulation. However, it is subject to the distribution and preferences of the users of the discord server.

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from
the dataset?
If so, please describe how.
The dataset is not targeted towards any individual, and we remove the information of the person submitting the prompt from the public release. The prompts themselves are subject to the distribution and preferences of the users of the discord server.

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)?
If so, please provide a description.
The prompts used are extracted from users’ input in the discord server. We have used Gemini 1.0 Pro to process the dataset to ensure that there are no harmful or offensive prompts. The prompts were then used to generate images from Stable Diffusion XL, which has an NSFW filter. The data is however subject to the failure of these models and the bias/preferences of the humans.

[linecolor=violet]

## COLLECTION

How was the data associated with each instance acquired?
Was the data directly observable (e.g., raw text, movie ratings),
reported by subjects (e.g., survey responses), or indirectly
inferred/derived from other data (e.g., part-of-speech tags, model-based
guesses for age or language)? If data was reported by subjects or
indirectly inferred/derived from other data, was the data
validated/verified? If so, please describe how.
The data was collected from the showdown and pantheon channels of the official Stability.ai discord server. In the discord server, users publicly posted prompts to obtain image generations and competed against other users for the chance to have prompts and images featured in the pantheon channel. We used DiscordChatExporter software to obtain the public prompts.

Over what timeframe was the data collected?
Does this timeframe match the creation timeframe of the data associated
with the instances (e.g., recent crawl of old news articles)? If not,
please describe the timeframe in which the data associated with the
instances was created. Finally, list when the dataset was first published.
This data was collected from the showdown channel from July 11, 2023, which is 1 day after the SDXL 1.0 candidates were made available via bots on discord; until Feb 07, 2024 which is when the SDXL bots were shut down. As pantheon was derived from showdown, and maintained the history from the start of channel creation, we scraped the pantheon channel only once, which collected posts from May 02, 2023 onwards.

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software
API)?
How were these mechanisms or procedures validated?
We used DiscordChatExporter executed on a Linux machine on a $14$ minute cron job to capture all posts from showdown. The showdown was updated every $15$ minutes, with history cleared every $30$ minutes.

What was the resource cost of collecting the data?
(e.g. what were the required computational resources, and the associated
financial costs, and energy consumption - estimate the carbon footprint.)
We executed the data collection from Discord on a machine that was on regardless of usage, we do not believe this imposed significant CPU usage or energy cost.

If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?
We filtered the dataset using a large language model, which caused around $4.6\%$ of the 235k prompts to be removed due to safety reasons (not transformed to natural language captions), yielding 224k natural language captions. Furthermore, captions longer than $77$ tokens were not used for image generation, and roughly 200k captions were used for images.

Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much
were crowdworkers paid)?
The data collection was performed by students as part of their research.

Were any ethical review processes conducted (e.g., by an institutional
review board)?
If so, please provide a description of these review processes, including
the outcomes, as well as a link or other access point to any supporting
documentation.
No IRB was required as this data collection did not directly involve human subjects.

Does the dataset relate to people?
If not, you may skip the remainder of the questions in this section.
Not directly, however, the prompts were submitted by users.

Did you collect the data from the individuals in question directly, or
obtain it via third parties or other sources (e.g., websites)?
They were collected from the chat messages publicly submitted by users to Discord.

Were the individuals in question notified about the data collection?
If so, please describe (or show with screenshots or other information) how
notice was provided, and provide a link or other access point to, or
otherwise reproduce, the exact language of the notification itself.
No notification was provided as the messages were publicly shared, and expected to be visible and voted upon by other users. We notified moderators of the Stability Discord server that we were performing this data collection effort and did not receive any objection.

Did the individuals in question consent to the collection and use of their
data?
If so, please describe (or show with screenshots or other information) how
consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the
individuals consented.
Users consented to their prompts being publicly visible by submitting prompts for image generation. Other users were explicitly encouraged to view and vote on the prompts and images.

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis)been conducted?
If so, please provide a description of this analysis, including the
outcomes, as well as a link or other access point to any supporting
documentation.
We performed an NSFW evaluation using Llama Guard 2 and found that our captions post-filtering were overwhelmingly safe. Visual examination of over a thousand randomly selected captions also confirmed that our captions were largely safe.

[linecolor=violet]

## PREPROCESSING / CLEANING / LABELING

Was any preprocessing/cleaning/labeling of the data
done(e.g.,discretization or bucketing, tokenization, part-of-speech
tagging, SIFT feature extraction, removal of instances, processing of
missing values)?
If so, please provide a description. If not, you may skip the remainder of
the questions in this section.
We performed our prompt cleaning in three steps. (1) We use a set of handcrafted grep filters to remove common prompt corruptions, mispunctuation, extra spaces, and non-text Unicode. 2 We deduplicate the prompts after grep filtering. (3) These prompts are provided to a large language model, where they are filtered and transformed into natural language captions.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)?
If so, please provide a link or other access point to the “raw” data.
Yes, the raw prompts will be shared.

Is the software used to preprocess/clean/label the instances available?
If so, please provide a link or other access point.
Yes, the code will be publicly provided after paper acceptance.

[linecolor=violet]

## USES

Has the dataset been used for any tasks already?
If so, please provide a description.
We use our dataset to evaluate captioning and open vocabulary segmentation models. However, the dataset has not been used to train new models as of writing.

Is there a repository that links to any or all papers or systems that use the dataset?
If so, please provide a link or other access point.
The link to the dataset will contain papers that use our dataset: [https://stablesemantics.github.io/StableSemantics/](https://stablesemantics.github.io/StableSemantics/)

What (other) tasks could the dataset be used for?
The dataset can be used for training/finetuning and evaluating scene understanding, semantic segmentation, and object detection/classification models. It can also be used for developing models for tasks like Visual Question Answering (VQA), visual grounding, inpainting, etc. It can also be used for understanding the distribution of natural language prompts and the biases present in them.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?
For example, is there anything that a future user might need to know to
avoid uses that could result in unfair treatment of individuals or groups
(e.g., stereotyping, quality of service issues) or other undesirable harms
(e.g., financial harms, legal risks) If so, please provide a description.
Is there anything a future user could do to mitigate these undesirable
harms?
The raw prompts collected are human written prompts and are then processed by Gemini 1.0 Pro. These prompts are then used to generate the images from Stable Diffusion XL based model. These models were used following their license terms.

Are there tasks for which the dataset should not be used?
If so, please provide a description.
Preventative steps have been taken to ensure there is no harmful content in the data as explained in the data processing part. However, we still recommend the data should not be used for any harmful or illegal purpose.

[linecolor=violet]

## DISTRIBUTION

Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organization) on behalf of which the dataset
was created?
If so, please provide a description.
The dataset will be under CC0 1.0 Universal Public Domain Dedication license and thus will distributed openly without any restriction.

How will the dataset be distributed (e.g., tarball on website, API,
GitHub)?
Does the dataset have a digital object identifier (DOI)?
The dataset will be shared via a torrent and hosted via http(s). The links to access the dataset will be available on our website which will be always maintained.

When will the dataset be distributed?
The complete dataset will be made publicly available by the end of June 2024.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?
If so, please describe this license and/or ToU, and provide a link or other
access point to, or otherwise reproduce, any relevant licensing terms or
ToU, as well as any fees associated with these restrictions.
It will not be under any copyright or intellectual property (IP) license. The dataset will be released under the CC0 1.0 Universal Public Domain Dedication license.

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?
If so, please describe these restrictions, and provide a link or other
access point to, or otherwise reproduce, any relevant licensing terms, as
well as any fees associated with these restrictions.
No.

Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances?
If so, please describe these restrictions, and provide a link or other
access point to, or otherwise reproduce, any supporting documentation.
No.

[linecolor=violet]

## MAINTENANCE

Who is supporting/hosting/maintaining the dataset?
The authors will be hosting and maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?
The authors should be contacted via email for any questions/problems regarding the dataset.

Is there an erratum?
If so, please provide a link or other access point.
There is no identified erratum with the dataset so far. However, the user must consider any possible errors associated with the LLM and Diffusion models used.

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?
If so, please describe how often, by whom, and how updates will be
communicated to users (e.g., mailing list, GitHub)?
The dataset will be updated by the authors if there is any error detected. The information will be announced on our website from where people can access the dataset.

If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were individuals
in question told that their data would be retained for a fixed period of
time and then deleted)?
If so, please describe these limits and explain how they will be enforced.
The dataset doesn’t collect any individual’s personal information.

Will older versions of the dataset continue to be
supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence
will be communicated to users.
The older versions of the dataset will continue to be supported. Any updates made in the dataset will be hosted as separate instances so that users of the previous version can still access it.

If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so?
If so, please provide a description. Will these contributions be
validated/verified? If so, please describe how. If not, why not? Is there a
process for communicating/distributing these contributions to other users?
If so, please provide a description.
We have no restriction on others augmenting this dataset. However, we don’t aim to officially merge the extended/augmented data to avoid any possible inclusion of offensive/harmful or illegal instances.

## A.6 Broader Impacts

StableSemantics consists of raw prompts, natural language captions derived from the prompts, $10$ images per caption generated using a Stable Diffusion XL based model, and semantic maps corresponding to the noun chunks in the caption for each image generation. The availability of this dataset is expected to improve visual semantic learning, open vocabulary segmentation methods, and improve the characterization of text-to-image generative models. We expect StableSemantics to help develop models that can understand complex visual scenes. This has the potential to increase the robustness of deep learning-based visual systems.

## A.7 Dataset website

Note that the dataset will be soon released. For now, we have released a sample consisting of images corresponding to $100$ samples of $5$ images each on the website. [https://stablesemantics.github.io/StableSemantics/](https://stablesemantics.github.io/StableSemantics/)

## A.8 Statement of responsibility

Authors accept all responsibility for potential violation of rights.

## A.9 Dataset license

We release our dataset under a CC0 1.0 Universal Public Domain Dedication license following Stable Diffusion’s [discord policy](https://stability.ai/discord-tos) and other work collecting data from Stable Diffusion’s discord [].

## A.10 Hosting plan and dataset format

We plan to host our dataset via an http(s) link, and simultaneously distribute the data via a torrent where we permanently seed the files. The captions are stored in a Python pickle file, while images and semantic maps are jpg files.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: What guarantee is there that the generated images contain the relationships specified in the prompt? Even though the prompts have undergone multiple stages of human and automated filtering, without knowing the exact seed used to generate these images, it is impossible to produce images identical to those that received high ratings on Discord. Moreover, neither CLIP score nor DAAM accurately interprets these relationships and can easily make errors. Consequently, human evaluation is still needed to filter out images that do not align with their prompts.
soundness: 1
strengths: 1- The StableSemantics dataset maps out exactly which parts of synthetic images correspond to specific objects and concepts. It contains 2 million images paired with 10 million attention maps that show where different words and phrases appear in each image. Unlike previous datasets, it provides detailed spatial information about a wide range of objects and concepts.

2- The  StableSemantics is the first diffusion dataset to record spatial distribution of cross-attention activations for individual noun chunks, positioning it as a pioneering resource for exploring spatial semantics in synthetic data​
summary: The paper presents "StableSemantics," a synthetic dataset aimed at improving visual scene understanding by combining human-curated text prompts, synthetic images, and dense attention maps at the noun-chunk level. The dataset is built using cross-attention mechanisms within diffusion models, theoretically allowing alignment between text prompts and visual details in generated images. It claims to address limitations in existing datasets by offering more nuanced semantic attributions.
weaknesses: 1- As noted in recent studies [1,2], text encoders in models like CLIP have difficulty preserving detailed relational attributes between objects. Since Stable Diffusion XL also uses a CLIP encoder, there is a high probability that generated images include all requested objects and attributes but fail to capture accurate spatial or functional relationships. For example, when prompted with “a banana with checkered skin,” the model may produce an image with a banana on a checkered background, misrepresenting the intended attribute. This limitation restricts the model’s ability to generate complex scenes that require precise relationships between elements. Given this limitation, a human evaluation step is necessary to filter out images that do not align precisely with their prompts, ensuring better relational consistency in the dataset.

2- The paper relies on CLIP scores and DAAM attribution maps as primary metrics to assess image quality. However, these metrics primarily capture object presence rather than the accuracy of spatial or relational context. To ensure that each generated image reflects the intended relationships and layout specified by the prompt, relation-specific metrics or additional human assessment are recommended. This would enhance the dataset’s value for training models capable of nuanced scene understanding and relational accuracy.

[1] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2022). When and why vision-language models behave like bags-of-words, and what to do about it?. arXiv preprint arXiv:2210.01936.

[2]  Hsieh, C. Y., Zhang, J., Ma, Z., Kembhavi, A., & Krishna, R. (2024). Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - [intro] "To overcome this challenge, recent advances have adopted data-driven approaches, which learn to recognize patterns and relationships in large datasets of images and annotations." - what are the works which do this?
- the authors mention that converting the prompts into captions is necessary because the prompts often have misspellings and unnatural sentence structure: why does this matter? is it because it is hard to parse the nouns with incorrect spelling?
soundness: 4
strengths: - this paper was clear in its dataset collection process and I particularly appreciated how comprehensive the data analysis was
- the addition of semantic attribution maps could be valuable in providing more fine grained semantic labels
- the dataset collection process as described requires very little human annotation and takes from a live datasource; I can see a promising direction in periodically recollecting data to ensure data freshness
summary: This paper introduces StableSemantics, a synthetic dataset which consists of the user prompt, the derived caption, the generated image and attention maps for each noun in each prompt. This can be used to find the semantic distribution of images, object relations, and provide a useful benchmark for dense visual semantic tasks.
weaknesses: My main concerns surround the value add of this dataset in comparison to DiffusionDB and JourneyDB 
- the authors claim that StableSemantics differs from datasets like JourneyDB because StableSemantics only collects prompts which are human generated. However, from looking through the JourneyDB collection process it seems that they also collect user prompts from discord servers: is the difference that JourneyDB also contains prompts which are not human generated? If so, this seems like a very small delta. 
- the authors show in table 8 that their prompts and captions have a lower perplexity score and thus is more similar to human language than DiffusionDB and JourneyDB, but it is unclear to me why this is desired. 
- In figure 7 the authors show that the captions are a subset of JourneyDB's, indicating to me that these captions don't capture a different distribution than existing datasets. This seems like an argument that this dataset provides little value
- there is a lot of mention about the distribution of prompts and captions being specifically objects/styles/scenes that users would find visually appealing. While i appreciate the authors honesty, I do think this reduces the value of this dataset as it only captures a small distribution compared to natural images which are often inputs into captioning and segmentation models. 

While an interesting idea, I do not think that this paper has shown that it provides substantial value when compared to existing datasets. That being said, I hope the authors can convince me otherwise by giving some applications of when someone would prefer this dataset over the others. One immediate thing I can think of is to generate the attribution maps for all or a subset of JourneyDB and see if the model lineup changes for the captioning and segementation models. If these differ and the authors show that the evaluation on their dataset provides insights into these models that JourneyDB does not, I would be compelled to improve my score. I am also happy to clarify any of my points in the rebuttal if they were unclear

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Did the authors perform any experiments to quantify the quality of the semantic maps, assuming that the authors wish to use them to characterize the spatial extent of objects? For instance, one way to mitigate concerns about quality would be to manually annotate a small number of images and compute mIoU metrics between the manual annotations and the semantic maps.

2. Given the noise issues discussed above, are there other potential applications of the semantic maps that could be useful for the community? One experiment that could demonstrate value would be to show that the authors could train on these maps and improve accuracy on a hand-annotated test set. (This would be similar to the MaskCLIP experiment you conduct in L518, but would change the test set to use trusted annotations.)

3. Using an LLM to rewrite the human prompts seems likely to subtly change the distribution of generated images. Do the authors have any ideas for how you could quantify this distribution shift? (One option might be to characterize differences between images generated by the prompts and images generated by the captions).  To give a specific example, the user asks for a hospital gaate (sic) in Figure 4, but the LLM caption rewriting changes this to “hospital entrance”. This is not necessarily what the user wanted. 

4. This is a question rather than a comment since I’m not a lawyer. Do the authors have the rights to release this dataset under a CC0 1.0 license, as proposed in L308-L309? Reading the Stable Diffusion discord terms of service (https://stability.ai/discord-tos, accessed 30th October 2024) my understanding is that Stability owns the content on the server. A CC0 1.0 license would make this discord server content available for downstream commercial use.

5. This is a low importance question (the authors may feel free to ignore it if preferred). In L467, it says “it makes sense for the sunset to always be on the top, the walls towards the sides, and the floor towards the bottom.” However, examining Figure 10, the “Wall” map seems to be strongest at the top of the image. Why do the authors think this is the case?
soundness: 3
strengths: Significance: Although there are some limitations to the use of cross-attention maps (see discussion below), I think there is considerable value in constructing synthetic media datasets.  In particular, I agree with the authors’ observation (L088-L089) that generative media has proliferated over social media, advertisements and news media. Consequently, it is important that the community develops methods that work well on such data. I think this direction is relatively undervalued within the Computer Vision community, and this dataset represents a step in this direction. 

Originality: As noted by the authors, this dataset appears to be the first large-scale synthetic dataset to contribute semantic maps in addition to images and captions. Although I have concerns about the use of these maps (see discussion below), this is an interesting contribution. If the maps were of high quality, I think this could have been particularly impactful.

Quality: The dataset itself is relatively large-scale (224K curated human-written prompts). This quantity brings a certain quality by enabling more robust statistical analysis of the data than would otherwise have been possible with smaller datasets.  

Clarity: Overall, the paper is fairly clearly written.  The figures are nicely captioned and presented. I particularly appreciated the inclusion of failure cases in the appendix (e.g. the “Lone Man” examples in Figure S.5), which helps to give the reader a sense of the robustness of the semantic maps.
summary: This work contributes a large-scale synthetic language-vision dataset called StableSemantics. The dataset is constructed in several stages. First, human-written prompts are curated from the Stable Diffusion discord server in a manner that accounts for user rankings of the resulting image. Each prompt is post-processed (using a combination of regular expressions and LLM re-writing) to form a caption, and this caption is used to synthesize images using a distilled variant of Stable Diffusion XL. Finally, diffusion attribution maps are computed for noun chunks appearing in the caption to generate dense semantic attribution labels. 

The key contribution claimed by the authors is the construction of the first large-scale synthetic dataset that includes semantic maps in addition to images and captions. In support of this contribution, the authors conduct various analyses of StableSemantics. These include analysis of the spatial distribution of cross-attention activations corresponding to individual noun chunks, visualization of CLIP embeddings and perplexity measurements from prompts and captions. As an application of the dataset, the authors evaluate open-vocabulary segmentation methods and captioning models on their dataset.
weaknesses: 1) One of the key contributions of this work, relative to prior work, is the inclusion of semantic maps. In L095-L096, the authors write: “Specifically, there are no existing large-scale generated datasets that contribute semantic maps in addition to images and their corresponding captions.”  In writing this weakness, I’m interpreting the authors’ claimed contribution to be semantic maps that are similar in usefulness/accuracy to semantic segmentation or instance segmentation annotations.  This interpretation comes from two sources: (1) the listed applications in L111-L112 “StableSemantics can be utilized in future research on various vision tasks such as object detection, semantic segmentation, semantically meaningful representation learning, image-inpainting and object removal, object editing, etc.“, (2) the use of the semantic maps to evaluate open-set segmentation models in Figure 14. 

    Unfortunately, relative to traditional instance or semantic segmentation annotations, the semantic maps provided are pretty noisy in capturing the spatial extent of objects or "stuff". As such, it is unclear how useful they are.   A good example is the “mirror” mask in Figure 5, which does not segment the mirror, and instead encodes the surrounding mirror frame.  One consequence of this noise is that it means that downstream evaluations (such as those conducted on open-set segmentation models in Figure 14) don’t seem particularly informative. 

   As noted above, I’m listing this point as a weakness on the assumption that the authors intend to frame these semantic maps as being essentially semantic segmentation annotations. It may be that this is a misinterpretation on my part, and they solely wish to claim that they are providing generic semantic attribution maps that don’t aim to capture the spatial extent of objects. This would remove the weakness related to noise, but it brings another problem: it makes the dataset less useful overall.

2) I’m slightly concerned that the generated media is specific to a single model (namely sdxl lightning 4step unet). This may not be an issue - perhaps this model is highly representative of many modern diffusion models. However, it is difficult to ascertain from the paper whether this dataset will immediately go “stale” when new models are released. One possible way to mitigate this would be to generate images from multiple models and demonstrate that similar qualitative insights can be drawn from the data associated with each model.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: 1. The paper is easy-to-understand.
2. Captioning and segmentation models are evaluated.
3. The prompts are manually filtered to ensure quality.
summary: This paper introduces a new large-scale dataset that combines text-to-image generations with semantic annotations, comprising 224K human-curated prompts and 2M synthetic images. The dataset provides fine-grained semantic attributions at the noun-chunk level, along with 10 million attention maps that link textual descriptions to specific image regions. The authors analyze the semantic distribution of generated images and benchmark various vision tasks.
weaknesses: My major concern is the significance of the proposed dataset. The main difference between the proposed dataset and previous counterparts is the additional dense open-set spatial semantic maps. The authors claim that "we expect StableSemantics to catalyze advances in visual semantic understanding and provide a foundation for developing more sophisticated and effective visual models". However, the reviewer remains unclear about the usage of semantic maps. How are semantic maps being used in existing model training (e.g., VLMs, MLLMs)? In addition, whether the proposed dataset can actually benefit downstream tasks remains doubtful. More experiments are needed to demonstrate the effectiveness.