SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Negation: Embeddings may not be able to distinguish between negated and non-negated sentences. Sentences are encoded similarly, even though they have opposite meanings.
2. Subtle Differences: In some cases, the embedding model fails to capture the nuances between different actions or activities that may appear similar.
3. Spatial Relations: The model may not encode sentences with clear spatial relationships accurately. This failure may lead to problems in generating images or videos with correct spatial relationships.
4. Attribute Differences: The embedding model tends to overlook specific details or attributes mentioned in the sentences. This failure would result in generating images or videos that may not accurately depict the mentioned details or attributes.
5. Near Synonyms: The embedding model may encode different words that have similar meanings, or synonyms, as if they were identical. This could cause problems for future tasks related to visual generation because it could result in the model generating incorrect images or videos.
6. Numerical Differences: : The model fails to differentiate between sentences involving singular and plural instances. The embedding model does not adequately encode the presence or absence of multiple instances, potentially leading to incorrect visual generation.
7. Subject Identity (Gender, Age): The model might fail to encode the syntactic structure of a sentence, leading to confusion between different concepts. For example, in the pairs "A man in a white shirt is walking across the street" and "A woman in a white shirt is walking across the street," the model might not differentiate between "man" and "woman," leading to ambiguity.
8. Granularity (Intensity): The model encodes sentences describing actions or movements similarly. The embedding model does not effectively capture the distinctions in actions or movement, which can result in inaccurate visual representations.
C.3 Ablation study on using different corpus and LLM
Mean, std and success rate of each LM-corpus pair We measure the mean, standard deviation and success rate of each LM-corpus pair uncovered systematic failure in Table 1. The table contains numbers that produces results in Figure 3. Our findings indicate that, despite identifying fewer systematic failures, the quality of systematic failures produced by Claude is comparable to that of GPT-4. Meanwhile, GPT-3.5 lags behind in this respect.
Systematic Failure	GPT-4	Claude	GPT-3.5
Mean	Std	Suc.	Mean	Std	Suc.	Mean	Std	Suc.
Negation	0.952	0.019	100%	0.928	0.027	95.1%	0.923	0.039	89.0%
Temporal Differences	0.924	0.033	96.2%	0.941	0.025	98.7%	-	-	-
Quantifier	0.950	0.029	98.7%	0.873	0.037	43.9%	-	-	-
Bag-Of-Words	0.928	0.029	91.5%	0.951	0.026	98.6%	-	-	-
Absence Vs Presence	0.933	0.029	91.5%	0.936	0.027	96.1%	-	-	-
Homonyms	0.758	0.079	1.2%	0.859	0.094	47.9%	-	-	-
Subtle Differences	0.917	0.032	86.6%	0.941	0.033	93.9%	0.910	0.044	79.5%
Spatial Relations	0.930	0.047	89.6%	0.922	0.049	81.4%	0.926	0.038	87.8%
Attribute Differences	0.823	0.093	35.3%	-	-	-	0.841	0.052	18.4%
Near Synonyms	0.887	0.056	65.9%	-	-	-	0.874	0.053	56.1%
Numerical Differences	0.906	0.052	72.0%	-	-	-	0.897	0.063	68.5%
Action State / Differences	0.854	0.073	41.5%	0.886	0.051	59.8%	-	-	-
Subject Identity	0.875	0.064	62.2%	0.923	0.047	81.7%	0.855	0.073	48.8%
Granularity (Intensity)	0.887	0.060	62.5%	0.883	0.060	64.6%	0.841	0.092	42.3%
Table 1: We measure the quality of each LM-corpus pair uncovered systematic failure with their mean CLIP similarity, standard deviation and success rate (Suc.) across new generated pairs.
Distribution of similarity of generated individual failures We plot the distribution of CLIP similarities of generated individual failures in Figure 7. These failures, categorized and generated by GPT-4, have been divided into two groups for improved clarity. The first group consists of systematic failures with a success rate below  $80\%$ , while the second group comprises systematic failures with a success rate exceeding  $80\%$ . Examination of the plot reveals that the majority of systematic failures are capable of generating high-quality individual failures.
(a) Systematic Failures with success rate  $< 80\%$
Figure 7: Distribution of Similarity Scores for Generated Individual Failures.
(b) Systematic Failures with success rate  $\geq 80\%$
C.4 Ablation study on description using LLM
We turn our attention to the quality of the descriptions associated with the summarized systematic failures. Although large language models are capable of categorizing systematic failures, the nature of their descriptions can influence the generation state of MULTIMON. Our focus is on the five systematic failures that are categorized by these three language models. We then compare the quality of the individual failures that each of GPT-4, Claude, and GPT-3.5 generate from the disparate descriptions, as detailed in Table 2. GPT-4 and Claude produce equally good descriptions, while GPT-3.5 produces slightly worse descriptions.
Systematic Failures	GPT-4	Claude	GPT-3.5
Mean	Std	Suc.	Mean	Std	Suc.	Mean	Std	Suc.
Negation	0.952	0.019	100%	0.928	0.027	95.1%	0.923	0.039	89.0%
Subtle Differences	0.917	0.032	86.6%	0.941	0.033	93.9%	0.910	0.044	79.5%
Spatial Relations	0.930	0.047	89.6%	0.922	0.049	81.4%	0.926	0.038	87.8%
Subject Identity	0.875	0.064	62.2%	0.923	0.047	81.7%	0.855	0.073	48.8%
Granularity (Intensity)	0.887	0.060	62.5%	0.883	0.060	64.6%	0.841	0.092	42.3%
Table 2: This table showcases our comparison of description quality among systematic failures detected by each language model. We employ GPT-4 to generate individual failures grounded in the systematic failures each language model reveals, and then we calculate the mean, standard deviation, and success rate (Suc.).
C.5 Ablation study on using different LLM as generator
Here, we study using different language models to generate individual failures from the same systematic failures. We choose the first 7 systematic failures categorized by GPT-4 and generate individual failure instances using GPT-4, Claude and GPT-3.5 respectively. Results are summarized in Table 3. We observe that GPT-4 and Claude are both good generator, whereas GPT-3.5 is less competent.
These results also demonstrate that we could be underestimating the true success rate of MULTIMON; better models may be more faithful to the descriptions of systematic failures, and more reliably produce pairs that contain failures.
Systematic Failures	GPT-4	Claude	GPT-3.5
Mean	Std	Suc.	Mean	Std	Suc.	Mean	Std	Suc.
Negation	0.952	0.019	100%	0.938	0.027	100%	0.951	0.025	100%
Temporal Differences	0.924	0.033	96.2%	0.941	0.025	97.0%	0.693	0.104	4.2%
Quantifier	0.950	0.029	98.7%	0.900	0.063	65.8%	0.743	0.071	0.0%
Bag-of-Words	0.928	0.029	91.5%	0.959	0.017	100%	0.907	0.054	76.4%
Absence Vs Presence	0.933	0.029	91.5%	0.919	0.027	90.2%	0.837	0.036	11.4%
Homonyms	0.758	0.079	1.2%	0.882	0.069	51.1%	0.742	0.076	0.0%
Subtle Differences	0.917	0.032	86.6%	0.962	0.018	100%	0.911	0.052	80.3%
Table 3: We use GPT-4, Claude and GPT-3.5 to generate new individual failures categorized by GPT-4. GPT-4 and Claude are on par with each other as generator, while GPT-3.5 is less competent.
C.6 Ablation study on no corpus
To study the importance of scraping corpus data and find failure instances, we prompt language model (GPT-4) to produce systematic failures without including examples from the corpus. We use prompts from Appendix B.1 without parts related scraped failure instances from corpus. We found that the model comes up with homonyms and subtle differences. We evaluate these two systematic failures using GPT-4 to generate new individual failures. Results can be found in Table 4, but find an average success rate of 29.3. This verifies the importance of corpus dataset when generating systematic failures.
Systematic Failures	Mean	Standard Deviation	Success Rate
Homonyms	0.760	0.069	4.9%
Subtle Differences	0.877	0.071	53.7%
Table 4: We prompt GPT-4 to categorize systematic failures without corpus data. We then generate individual failure instances and measure mean, standard deviation and success rate of generated new individual failures by GPT-4.
C.7 Steering MULTIMON
Steering Scraping When scraping datasets, we additionally ask a zero-shot GPT-3.5 model
Please respond with either "yes" or "no" to the following:
Is the difference between "input 1" and "input 2" important for [dir]?
Where dir is the direction we hope to steer in (in this case, self-driving cars). With this steering scraping, we categorized 5 systematic failures that are relevant to self-driving cars:
1. Negation handling: The model may struggle to encode negation or opposite meanings, such as "The car is stopping" and "The car is not stopping." These sentences convey contrasting concepts, but the embeddings might be too similar, leading to incorrect visual generation.
2. Temporal ambiguity: The model might not differentiate between present and future events, such as "The car is turning left" and "The car will turn left." In a self-driving context, distinguishing between present and future actions is crucial for accurate visual representation and decision-making.
3. Quantitative differences: The model may struggle with encoding differences in quantity, such as "The car is moving slowly" and "The car is moving very slowly." This could lead to issues with visual generation, as the rate of movement is important in a self-driving context.
4. Spatial relationships: The model may not accurately capture spatial relationships between objects, such as "The car is following the truck closely" and "The car is following the truck at a safe distance." This is particularly important for self-driving applications, as accurate spatial understanding is critical for safe navigation.
5. Object-specific attributes: The model may not differentiate between important attributes of objects, such as "The pedestrian is crossing the street" and "The cyclist is crossing the street." These distinctions are crucial for self-driving cars to make appropriate decisions based on the varying behaviors of different road users.
We further generate new individual failures and measure the mean, standard deviation and success rate of the generated new individual failures under the context of self-driving cars. We also measure relevance rate by asking GPT-3.5 model the following question and measure the ratio of generated individual failures that are relevant to self-driving,
Is the difference in the following pair of sentences salient to [dir]? {"prompt1}" {"prompt2"} Please answer YES or NO
We summarize results in Table 5, and include further results in Table 8. Results show that we can effectively steer MULTIMON towards a direction (e.g. self-driving cars) by steering scraping.
Systematic Failures	Mean	Standard Deviation	Success Rate	Relevance Rate
Negation	0.953	0.023	100%	100%
Temporal Differences	0.953	0.019	100%	100%
Qualitative Differences	0.962	0.033	96.3%	100%
Spatial Relationship	0.951	0.025	100%	100%
Object Specific Attributes	0.854	0.076	41.0%	92.3%
Table 5: We steer scraping towards self-driving cars and categorize systematic failures based on the steering scraping failures. We then generate individual failures and measure the mean, standard deviation, success rate and relevance rate, which we report here.
Steering generation. Next, we test whether evaluators can steer towards individual failures relevant to self-driving. We edit the generation stage of our pipeline by appending "Keep in mind, your examples should be in the context of self-driving" to the prompt from Appendix B.2. We measure the mean, std, success rate and relevance rate of the generated failures in Table 6. The results show that the systematic failures we find using normal corpus data can be applied to specific applications using steering generation, obtaining an average success rate of  $74.56\%$  and average relevance rate of  $95.01\%$ .
Systematic Failures	Mean	Standard Deviation	Success Rate	Relevance Rate
Negation	0.941	0.020	100%	92.7%
Temporal Differences	0.951	0.029	95.1%	92.7%
Quantifier	0.915	0.048	77.1%	100%
Bag-of-Words	0.909	0.038	79.5%	48.7%
Absence Vs Presence	0.927	0.034	87.8%	92.7%
Homonyms	0.802	0.090	19.5%	34.2%
Subtle Differences	0.875	0.076	53.9%	76.9%
Spatial Relations	0.916	0.073	71.1%	86.9%
Attribute Differences	0.920	0.052	92.0%	82.5%
Near Synonyms	0.856	0.077	46.3%	85.9%
Numerical Differences	0.878	0.091	63.4%	92.7%
Action State / Differences	0.882	0.054	61.0%	95.1%
Subject Identity	0.865	0.062	51.2%	87.5%
Granularity (Intensity)	0.857	0.060	38.5%	87.2%
Table 7: We steer evaluators towards Pokemon Go. We then measure mean, standard deviation, success rate and relevance rate. MULTIMON generates individual failures with both high success rate and relevant to Pokemon Go.
Systematic Failures	Success Rate	Relevance Rate
Negation	100% ± 0.0%	100% ± 0.0%
Temporal Differences	100% ± 0.0%	100% ± 0.0%
Qualitative Differences	96.3% ± 2.1%	100% ± 0.0%
Spatial Relationship	100% ± 0.0%	100% ± 0.0%
Object Specific Attribute	41.0% ± 5.5%	92.3% ± 3.0%
Average	87.5%	98.5%
Table 8: Success and relevance rates when steering MULTIMON towards self-driving-related systematic failures. The systematic failures consistently have high success and relevance rates.
Systematic Failures	Mean	Standard Deviation	Success Rate	Relevance Rate
Negation	0.968	0.019	100%	100%
Temporal Differences	0.949	0.021	100%	97.6%
Quantifier	0.959	0.015	100%	100%
Bag-of-Words	0.937	0.022	97.1%	85.7%
Absence Vs Presence	0.875	0.053	51.2%	100%
Homonyms	0.830	0.085	27.0%	70.3%
Subtle Differences	0.913	0.049	82.9%	100%
Spatial Relations	0.938	0.042	93.8%	96.8%
Attribute Differences	0.867	0.073	51.2%	97.6%
Near Synonyms	0.831	0.046	17.0%	92.8%
Numerical Differences	0.886	0.038	63.2%	100%
Action State / Differences	0.942	0.039	94.7%	100%
Subject Identity	0.904	0.037	71.1%	92.1%
Granularity (Intensity)	0.930	0.029	94.6%	97.3%
Table 6: We steer evaluators towards self-driving cars. We then measure mean, standard deviation, success rate and relevance rate. MULTIMON generates individual failures with both high success rate and relevant to self-driving cars.
We also steer generation towards concepts beyond the distribution of the original corpus data, such as Pokemon Go. We measure the mean, std, success rate and relevance rate of the generated failures in Table 7. The results show that systematic failures categorized can also be used to generate failures containign concepts out side the corpus data.
D Additional Results on Downstream Failures
D.1 Additional manual study details
We generate 100 pairs with MULTIMON and 100 pairs with the baseline. The baseline scrapes random pairs from MS-COCO, then categorizes into systematic failures and generates individual failures normally. We then randomly select choose one of the four text-to-image models (Stable Diffusion 2.1, Stable Diffusion 1.5, Stable Diffusion XL, MidJourney 5.1) to generate images and ask the annotator the following questions
- Is the image generated by prompt 1?
- Is the image generated by prompt 2?
- Is the image generated by neither prompts?
- Would the prompts generate visually identical images?
An example of the labeling interface is in Figure 8. Two authors labeled all 400 images, and the labels of the two authors were added together.
Figure 8: Annotator interface for our manual evaluation.
D.2 Additional manual evaluation results
Ratio of visually identical images verses the DistilRoBERTa similarity threshold Here, we plot the number of visually identical prompts on each DistilRoBERTa similarity interval in Figure 9. On all DistilRoBERTa similarity intervals, most of the generated pairs are visually different, leading us to avoid choosing a threshold.
Ratio of downstream failures verses the CLIP similarity threshold Here, we plot the number of visually identical prompts on each CLIP Similarity Interval in Figure 10. Over  $65\%$  of the individual examples in pairs with a CLIP similarity around 0.88 are failures. Since there is an abrupt shift at this threshold, we select it for the success rate. This manual evaluation offers vital insights into the sensitivity of contemporary text-to-image models in relation to input CLIP text embeddings.
The outcomes suggest that when the similarity between two text embeddings surpasses 0.88, caution is required due to the heightened probability that the generated text may not correspond with the given input. Note however that this threshold is model dependent; so long as the CLIP embeddings aren't identical, in principle a downstream system could leverage the small difference in embedding to generate separate images.
Results of manual evaluation We measure and analyze the number of failure pairs, where the annotator selects an incorrect prompt, or chooses neither. Results are summarized in Table 9. The table shows that MULTIMON generate individual instances that largely result in failures. Whereas
Figure 9: Ratio of visually identical prompts on each DistilRoBERTa Similarity Interval
Figure 10: Ratio of mistakes annotator makes on each CLIP Similarity Interval. The figure shows that for pairs with clip similarity over 0.88, there is more than  $60\%$  chance of making mistakes.
MidJourney 5.1
"one shoe on the floor"
DALL-E (New Bing)
"the car is far from the stop sign"
"awomanprepare toputon makeup"
"ashort pine tree"
"amanclosingabook"
Figure 11: More examples of inputs that MULTIMON generates used in text-to-image models.
"acatchasesadog
"a table not set for dinner"
"adeer chased by a leopard"
Stable Diffusion XL
"a small box on the table"
"a squirrel falling off the tree"
"awoman teaches a man
"abirthdaycakewith7candles
Stable Diffusion 2.1
"aslightly tall building
"a slice of pizza on the table
"akitchenwithoutrefrigerator"
"aplane flying below the sky
text-to-image models normally does not lead to failure, as demonstrated by baseline results. We also found that around  $9\%$  of the prompts generated by MULTIMON are labeled as "visually identical". This indicates that only a small portion of the generated prompts are not suitable for downstream text-to-image generation, whereas the majority that good examples of failure in text-to-image models.
# of Failure Pairs / # of Pairs	# of Failure Pairs / Total # of Failure Pairs
MULTIMON	80.00%	79.61%
Baseline	20.50%	20.39%
Table 9: Comparison of Mistakes generated by MULTIMON and baseline
D.3 Additional results on text-to-image models
We provide more MULTIMON generated individual failures applied to text-to-image models (Mid-Journey 5.1, DALL-E from New Bing, Stable Diffusion XL and Stable Diffusion 2.1) in Figure 11.
D.4 Additional results on text-to-3D models
We provide more MULTIMON generated individual failures applied to text-to-3D models in Figure 11.
Figure 12: More examples of errors in Shap-from inputs that MULTIMON generates.
D.5 Additional results on the individual failures generated by MULTIMON
Here, we show some of the individual failures generated by MULTIMON via GPT-4 as categorizer and generator.
- ("A child opening a birthday present", "A child about to open a birthday present")
- ("A runner crossing the finish line", "A runner who has just crossed the finish line")
- ("A flower blooming in spring", "A flower that will bloom in spring")
- ("A couple getting married", "A couple who are about to get married")
- ("A tree shedding its leaves in autumn", "A tree that has shed its leaves in autumn"),
- ("A bowl with many apples", "A bowl with few apples")
- ("A park with some people", "A park with many people")
- ("A table with several books", "A table with a few books")
- ("A room with a couple of chairs", "A room with several chairs")
- ("A street with numerous cars", "A street with a handful of cars")
- ("A man teaching a woman", "A woman teaching a man")
- ("A girl pushing a boy", "A boy pushing a girl")
- ("A waiter serving a customer", "A customer serving a waiter")
- ("A lion hunting a gazelle", "A gazelle hunting a lion")
- ("A spider catching a fly", "A fly catching a spider")
- ("A landscape with a river", "A landscape without a river")
- ("A forest filled with trees", "A forest with no trees")
- ("A sky with clouds", "A sky without clouds")
- ("A room containing furniture", "A room with no furniture")
- ("A playground with children", "A playground without children")
- ("A slightly annoyed person", "A furious person")
- ("A person looking mildly surprised", "A person looking shocked")
- ("A slightly cloudy day", "A heavily overcast day")
"A curious cat", "A scared cat"
- ("A partially filled glass of water", "An almost full glass of water")
- ("A cat sitting on top of a car", "A cat sitting underneath a car")
- ("A bookshelf next to a window", "A bookshelf far from a window")
- ("A ball rolling in front of a child", "A ball rolling behind a child")
- ("A vase of flowers beside a lamp", "A vase of flowers across from a lamp")
- ("A tree near the edge of the lake", "A tree far from the edge of the lake")
- ("Two children playing soccer", "Four children playing soccer")
- ("A street with one traffic light", "A street with three traffic lights")
- ("A painting with six birds", "A painting with eleven birds")
- ("A man juggling three balls", "A man juggling five balls")
- ("A picnic with eight people", "A picnic with twelve people")
- ("A car driving down the road", "A car parked on the side of the road")
- ("A dog barking at the mailman", "A dog sleeping on the porch")
- ("A plant growing in a pot", "A plant wilting in a pot")
- ("A child running in the park", "A child sitting on a bench in the park")
- ("A waterfall flowing rapidly", "A waterfall frozen in winter")
- ("A person gently stroking a cat", "A person vigorously petting a cat")
- ("A light rain falling on the street", "A heavy downpour on the street")
- ("A person slowly stirring a pot", "A person quickly mixing ingredients in a pot")
- ("A car driving at a leisurely pace", "A car speeding down the road")
- ("A soft breeze blowing through the trees", "A strong wind gusting through the trees")
E Using MULTIMON to Find Failures Beyond CLIP
We next use MULTIMON to identify failures of text-to-image systems that encode inputs with a non-CLIP embedding model. We do this by either (i) using MULTIMON to find failures of the non-CLIP embedding directly or (ii) transferring CLIP failures to other systems.
Finding failures directly. We first study whether MULTIMON can be used to find failure modes of other systems directly. To do so, we evaluate the DeepFloyd text-to-image system [Shonenkov et al., 2023], which encodes inputs with the T5 embedding model [Raffel et al., 2020]. To find failures, we apply each step of the MULTIMON pipeline as described in Section 3, except we use T5 instead of CLIP in the scraping step. We use GPT-4 as the categorizer and generator.
We find that MULTIMON is able to find systematic failures of the T5 embedding model, some of which do not arise with CLIP. MULTIMON finds eight systematic failures of T5, which have an average success rate of  $77.3\%$ . The systematic failures are as follows:
1. Failure to distinguish temporal differences: The model fails to distinguish the time of day, despite the sentences mentioning 'sunrise' and 'midnight', respectively. This is critical in visual representation, as these times would significantly change the lighting, color scheme, and potentially the activity depicted in the image.
2. Negation and Antonyms handling: The model does not adequately handle negation. The phrases 'likes cats' and 'doesn't like cats' have nearly opposite meanings. If this embedding model is used to generate images, it could generate an image of a person happily interacting with a cat in both scenarios, which is clearly incorrect.
3. Misinterpretation of homonyms: The word 'orange' is used differently in each sentence, once as a color and once as a fruit. This could lead to significant issues in visual representation as one would expect to see a color theme in the first sentence and a piece of fruit in the second.
Figure 13: Examples inputs that MULTIMON generates using T5 as the encoder. These inputs produce failures on T5-based DeepFloyd (top row), but not CLIP-based Stable Diffusion-XL (bottom row).