SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Some of these systematic failures that MULTIMON recovers are similar to those found in Section 4.1, but the descriptions tend to be different; for example, MULTIMON identifies "attribute differences" with and without steering, but outputs the description The model may not differentiate between important attributes of objects, such as "The pedestrian is crossing the street" and "The cyclist is crossing the street." when steered towards self-driving.
Steering towards individual failures. We next steer towards individual failures that are related to self-driving, by editing the generation stage of our pipeline with the method described in Section 3.3. Using the systematic failures from Section 4.1 and the modified generation stage, we find that the generated instances are often failures and related to self-driving;  $74.6\%$  of the instances are successful (i.e. have
high CLIP similarity), while  $95.0\%$  of pairs are relevant. Though relevance is computed with the GPT-3.5 classifier automatically, we empirically find the examples we generate are consistently related to self-driving; for example, using the systematic failure "action state differences", MULTIMON generates examples such as "Autonomous vehicle approaching a stop sign" and "Autonomous vehicle ignoring a stop sign".
Steering generation also allows MULTIMON to generate failures that are not in the distribution of the original corpus. We show this by steering towards failures relevant to "Pokemon Go", which was released after both of the corpora we test. We manage to obtain an average success rate of  $66.9\%$  and relevance rate of  $82.5\%$ , and find examples like "Team Mystic dominating a Pokémon Go gym", "Team Mystic not dominating a Pokémon Go gym".
We include additional experimental details, results, and generated individual failures in Appendix C.7.
5 Failures of CLIP lead to Failures Downstream
We next check that the failures generated by MULTIMON produce errors not just in the CLIP embeddings, but in downstream state-of-the-art multimodal systems. Through manual labeling, we find that text-to-image models fail frequently (i.e., produce incorrect images) on our generated inputs (Section 5.1). We then show how the same prompt can produce failures on many state-of-the-art systems, and include qualitative examples of failures using state-of-the-art text-to-image, text-to-video, and text-to-3d models (Section 5.2).
5.1 Manually evaluating generated images
We check that the inputs generated by MULTIMON produce errors in downstream systems by manually labeling whether the output images match the generated input text. We also plot the error rate against CLIP similarity, and use this to justify the CLIP similarity threshold chosen in Section 4.
To measure whether MULTIMON produces errors in downstream systems, we test the candidate pairs generated from systematic failures in Section 4.1. We say a candidate pair is a successful downstream failure if at least one input in the pair produces an incorrect image. To measure this, we create an annotation UI (Appendix D.1) where annotators are shown one generated image from the pair along with both text inputs, and asked whether the image corresponds to input 1, input 2, or neither input. The annotators also report whether the text inputs describe the same set of images; e.g., "A nice house" and "A lovely house". An input pair is a downstream failure if at least one image is labeled with an incorrect input or with "neither".
When evaluating MULTIMON, we want to ensure the failures found are nontrivial, since models may be brittle on any out-of-distribution input rather than the specific ones found by our system. To test this, we introduce a baseline system that ablates MULTIMON's scraping stage. Specifically, the baseline scrapes random pairs from MS-COCO (without ensuring high CLIP similarity), then categorizes these into systematic failures and generates new individual failures normally. Since the categorization and generation stages are fixed, the pairs we produce seem plausible; e.g., "A woman painting a beautiful landscape", and "A beautiful landscape painting on a wall".
In total, we generate 100 input pairs with MULTIMON and 100 pairs with the baseline. For each pair, we randomly select one of four text-to-image systems (Stable Diffusion XL, Stable Diffusion 2.1, Stable Diffusion 1.5, Midjourney 5.1) to generate images, label each image in the annotation UI, then combine the annotations to classify whether the pair is a failure. Annotations were performed by two authors, who were blinded to whether image pairs came from the baseline or from MULTIMON.
We find that MULTIMON produces far more downstream failures than the baseline;  $80\%$  of the pairs that MULTIMON generates are downstream failures, compared to only  $20\%$  of the baseline pairs. We then use these results to calibrate the CLIP similarity threshold from Section 4, which aims to capture when outputted images are visually indistinguishable. To do so, we histogram the ratio of downstream failures versus the CLIP similarity (Figure 10 in Appendix D.2). We find that the ratio grows roughly monotonically, and set the threshold at a jump at 0.88 where  $65\%$  of pairs are failures. We include the user-interface, additional details, and additional results in Appendix D.1.
Prompt: "a wind turbine at rest"
Figure 5: Top. Example of a 3d-scene Shape-E generates with 8 chairs instead of 4, rotated at different angles. Bottom. Example of a video VideoFusion generates of a wind turbine spinning, instead of at rest, captured at different frames.
5.2 Qualitative examples on state-of-the-art multimodal models
We next showcase how MULTIMON produces compelling qualitative examples of failures on state-of-the-art text-to-image, text-to-video, and text-to-3d systems, including examples steered towards self-driving. These examples are easy to obtain using MULTIMON; we simply take the pairs from Section 4, run both inputs through the model, and select one incorrect output.
Text-to-image models. MULTIMON produces failures on all state-of-the-art text-to-image models: Stable Diffusion XL [Stability.ai, 2023], Stable Diffusion 2.1 [Rombach et al., 2022], Midjourney 5.1 [Midjourney, 2023a] and DALL-E [Ramesh et al., 2022]. We access Stable Diffusion XL via DreamFusion, Stable Diffusion 2.1 via Huggingface [von Platen et al., 2022], Midjourney via Discord fast mode, and DALL-E via New Bing. We present examples in Figure 4, and in Appendix D.3.
These results demonstrate how state-of-the-art diffusion models cannot overcome the failures of CLIP embeddings: the same inputs produce failures across all tested text-to-image systems. They also show that MULTIMON can quickly find failures of new systems as they are released: two models that we test were released within two weeks of the writing of this paper, and three within a month.
Text-to-3D models. MULTIMON produces failures on a state-of-the-art text-to-3D system, Shap-E [Jun and Nichol, 2023]. We access Shap-E via Huggingface. In Figure 5, we present an example where Shap-E ignores numerical quantities (by including too many chairs at a dining room table), and include more examples in Appendix D.4.
Text-to-video models. MULTIMON also produces failures in dynamic scenes: we show that the pairs that MULTIMON generates produce failures on the best open-source text-to-video system, VideoFusion [Luo et al., 2023]. We access VideoFusion via Huggingface. In Figure 5, we present an example where VideoFusion struggles to capture differences in action states: "a wind turbine at rest" generates a video where the turbine is moving. Note that "a wind turbine at rest" and "a wind turbine in motion" might have been visually identical in static scenes, but are semantically distinct in video.
Steering Towards Applications. We next show that MULTIMON can be steered to produce specific kinds of downstream failures. Using the pairs generated in in Section 4.2, we exhibit self-driving-related failures in text-to-image, text-to-3d, and text-to-video systems (Figure 6). These include image examples where a car is in the incorrect lane, a 3d-scene example where a stop sign is mixed up with a yield sign, and a video of a car erroneously running through a red light. These examples could be salient to multimodal systems deployed in self-driving settings, but would have been challenging to uncover without explicitly steering MULTIMON towards the target subdomain.
5.3 Downstream failures beyond CLIP
We additionally show that MULTIMON can find failures in downstream systems that do not use CLIP. Specifically, we show that MULTIMON can be used to find failures in text-to-image systems that do not use CLIP (Appendix E), and that MULTIMON can circumvent safety filters on CLIP-based
Figure 6: Examples of failures that are relevant to "self-driving". These include images (top left, showing incorrect positions and colors), a 3d-scene (top right, depicting stop instead of yield sign), and a video (bottom, showing a car in the background erroneously not stopping for a light).
"a car stops for red light"
systems (Appendix F). These results underscore how MULTIMON is general purpose, and can find failures with deployed systems in high-stakes settings.
In this work, we produce failures of text-guided multimodal systems by scraping failures using erroneous agreement, then categorizing and generating new failures with language models. Our resulting system, MULTIMON, automatically finds failures that generalize across state-of-the-art text-to-image, text-to-video, and text-to-3d systems.
There is room for improvement at each stage of the MULTIMON pipeline. For example, we could find ways to scrape individual failures that erroneous agreement does not catch, or use better prompts at the categorization and generation steps. However, MULTIMON will naturally improve as language models do, since better language models can seamlessly plug into our pipeline. Subsequent work could even use MULTIMON to improve other systems, e.g., via fine-tuning on failures.
Our pipeline can in principle find failures with any system (e.g., large language models), since erroneous agreement is agnostic to the system architecture, input, or output type. MULTIMON is especially well-suited to multimodal systems, since erroneous agreement can be efficiently computed between embeddings; we thus find failures without ever generating outputs, which can be expensive (over one minute per output) for some of the models that we test. Subsequent work could design methods to efficiently approximate erroneous agreement for other systems, like language models or classifiers, by studying when inputs produce similar outputs but should not.
Our work demonstrates how recycling the same components across systems (such as CLIP) may inadvertently add new risks; the inputs that MULTIMON generates produce failures across all of the multimodal systems that we test, since they all (likely) rely on CLIP to encode text. These failures are also hard to fix post-hoc: repairing the CLIP embeddings would not be enough, since most downstream models would have to be retrained on the new embeddings. This is related to the issue of algorithmic monoculture, where models that use similar algorithms [Kleinberg and Raghavan, 2021], or that are trained with similar data [Bommasani et al., 2022], produce homogeneous errors. Components that are likely to be recycled across many models, like CLIP or GPT-4, should undergo more rigorous testing and updates before deployment.
More broadly, to address the robustness problems of the future, we need scalable evaluation systems: evaluation systems that (i) improve naturally via existing scaling trends, and (ii) and are not bottlenecked by human ingenuity. Model outputs like videos, proteins, and code are challenging and time-consuming for humans to evaluate, and can be incorrect in ways that are difficult to predict a priori. Developing scalable evaluation systems is critical as models improve, as models may reach the point where only machines can anticipate, detect, and repair their failures.
We thank Ruiqi Zhong, Lisa Dunlap, and Jean-Stanislas Denain for helpful feedback and discussions. E.J. was supported by a Vitalik Buterin Ph.D. Fellowship in AI Existential Safety.
A.1 Pseudo code of scraping and categorizing for MULTIMON
We provide pseudocode for MULTIMON in Algorithm 1. The algorithm also contains steps to steer scraping discussed in Section 4.2.
Algorithm 1 Pseudocode for scraping and categorizing in MULTIMON
1: procedure FINDFAILURES(corpus, threshold,  $k$ , steeredirection = None)  
2: pairs  $\leftarrow$  emptylist  
3: for each  $(\mathbf{x_1},\mathbf{x_2})$  in corpus do  
4: if cosine_similarity(encysemantic  $(\mathbf{x_1})$ , encsemantic  $(\mathbf{x_2}))\leq$  threshold then  
5: if steeredirection = None or  $\mathbf{x_1},\mathbf{x_2}$  related to steeredirection then  
6: pairs.append(( $\mathbf{x_1},\mathbf{x_2}$ )  
7: end if  
8: end if  
9: end for  
10: failures  $\leftarrow$  Categorizer(pairs,  $k$ )  
11: return failures  
12: end procedure
B Prompts Used in MULTIMON
In this section, we provide the prompt used in MULTIMON for categorizing systematic failures in Appendix B.1 and generating individual failures in Appendix B.2.
B.1 Prompt for categorizing systematic failures
We use the following prompts for categorizing. We first use this prompt to ask LLM remember scraped individual failures, provide the individual failures, then categorize them into examples:
I will provide a series of data for you to remember. Subsequently, I will ask you some questions to test your performance! Here are some pairs of prompts for you to memorize.   
[   
the cat chases the dog, the dog chases the cat   
a sky with one balloon, a sky with two balloons   
...(k Failure Instances)   
]   
I'm trying to find failures with an embedding model. The above are some pairs of sentences that it encodes very similarly, even though they're conveying different concepts. Using these specific examples, are there any general types of failures you notice the embedding is making, or any common features that the embedding fails to encode? Try to give failures that are specific enough that someone could reliably produce examples that the embedding would encode similarly, even though it shouldn't. Please try to give as many general failures as possible. Please focus on differences that are important visually, as these embeddings are later used to generate images, or videos. In your failure modes, please explain clearly why the failure would lead to problems for future tasks related to visual generation.Please summarize as many as you can and stick to the examples..
B.2 Prompt for generating individual instances
Given a systematic failure categorized, we prompt a language model to generate arbitrarily many new individual failures with the following prompt:
Write down 41 additional pairs of prompts that an embedding model with the following failure mode might encode similarly, even though they would correspond to different images if used as captions. Use the following format:
("prompt1", "prompt2"),
You will be evaluated on how well you actually perform. Your sentence structure and length can be creative; extrapolate based on the failure mode you've summarized. Be both creative and cautious.
Failure Mode:
[Systematic Failure (with full description)]
We can continue to generate subsequent instances by asking the LLM to generate more in the same session.
C Additional Quantitative Results on CLIP
C.1 The number of erroneous agreements in each corpus
While we only use 150 pairs of erroneous agreement in the prompt (due to the context window), we scrape 33922 pairs of erroneous agreements from SNLI (using 157351 examples to make pairs), and 2131440 pairs of erroneous agreement from MS-COCO (using 616767 examples to make pairs). Intuitively, even relatively small corpora may produce many examples of erroneous agreement, since the number of possible pairs scales quadratically with the size of the corpus.
C.2 Description of systematic failures
Systematic failures categorized by GPT-4 We provide the descriptions of the 14 systematic failures categorized by MULTIMON using MS-COCO and SNLI as the corpus and GPT-4 as categorizer.
1. Negation: Embedding models may not correctly capture the negative context in a sentence, leading to similarities between sentences with and without negation. This can result in incorrect visual representations, as the presence or absence of an action is significant in image or video generation.
2. Temporal differences: Embedding models might not differentiate between events happening in the past, present, or future..This failure can impact visual generation tasks by incorrectly representing the timing of events in generated images or videos.
3. Quantifiers: Embedding models may fail to distinguish between sentences that use quantifiers like "few," "some," or "many." This can lead to inaccuracies in the number of objects depicted in generated images or videos.
4. Semantic Role Ambiguity (Bag-Of-Words): The models might struggle to differentiate when the semantic roles are flipped, This failure can result in visual generation tasks depicting incorrect actions or object interactions.
5. Absence Vs Presence: Embedding models may not be able to distinguish between the presence or absence of certain objects, This can lead to visual generation tasks inaccurately including or excluding objects in the scene.
6. Homonyms: The models might not be able to differentiate between sentences with homonyms or words with multiple meanings. This can cause visual generation tasks to produce incorrect or ambiguous images.
7. Subtle Differences: Embedding models may not distinguish between sentences with subtly different meanings or connotations. This can result in visual generation tasks inaccurately depicting the intended emotions or nuances.
8. Spatial Relations: Embedding models may struggle to differentiate between sentences that describe different spatial arrangements. This can cause visual generation tasks to produce images with incorrect object placements or orientations.
9. Attribute Differences: Embedding models might not capture differences in attributes like color, size, or other descriptors. This can lead to visual generation tasks producing images with incorrect object attributes.
10. Near Synonyms: Embedding models could struggle to differentiate between sentences that use near-synonyms, This can result in visual generation tasks inaccurately depicting the intended actions or scenes, due to the model's inability to recognize semantic similarity.
11. Numerical Differences: The model might not accurately capture differences in the number of people or objects mentioned in the sentences. This might lead to issues in visual generation, such as generating an incorrect number of subjects or missing important context.
12. Action State and Differences: The model might not effectively differentiate between sentences describing different actions or states. This can lead to visuals that don't accurately depict the intended action or state.
13. Subject Identity (Gender, Age): The embeddings might fail to distinguish between different subjects, such as male vs female, adult vs child, or human vs animal, which could lead to visual differences in generated images.
14. Granularity (Intensity): The embeddings may fail to distinguish between different levels of action intensity,
Systematic failures categorized by Claude v1.3 We provide the descriptions of the 11 systematic failures categorized by MULTIMON using MS-COCO and SNLI as the corpus and Claude v1.3 as categorizer.
1. Negation: The model cannot reliably represent when a concept is negated or not present. This could lead to inappropriate inclusions of negated concepts in generated visual media. For example, the model may encode "no cat" and "cat" similarly, leading to a cat appearing in the visual for "no cat".
2. Temporal Differences: Failure to encode differences in verb tense: The model does not distinguish between present, past and future tense well. This could lead to temporal mismatches in generated media.
3. Quantifier: Failing to capture subtle but important distinctions in the number of objects/people referenced. Confusing singular and plural nouns, or quantifiers like "some" vs. "many" can lead to implausible visual generations.
4. Semantic Role Ambiguity (Bag-of-Words): The embedding fails to encode specific semantic roles or relationships between people or objects. This would lead to problems generating the proper interactions and relationships between people and objects in images or videos.
5. Absence Vs Presence: Failing to encode differences in specificity or details. The embedding encodes these similarly even though one includes the additional detail of the audience. Lack of specificity could lead to vague or sparse visual generations.
6. Homonyms: Failures on metaphorical or abstract language. Sentences with metaphorical, idiomatic or abstract meanings may be embedded over-literally or inconsistently. Generating visuals for these types of language expressions would require properly encoding the intended meaning.
7. Subtle Differences: Failure to capture subtle differences. The model fails to distinguish between sentences that differ only in small words or phrases. These small differences can lead to generating very different images.
8. Spatial Relations: Failures to encode spatial relationships and locations accurately. Sentences that describe the same concept or object in different locations or with different spatial relationships to other objects may be embedded similarly. This would lead to issues generating spatially coherent images or videos.
9. Action State and Differences: Failures to encode different actions, events or temporal sequences properly. Sentences describing static scenes vs active events or different event sequences may be embedded similarly. This would lead to difficulties generating visually dynamic, temporally coherent images or videos.
10. Subject Identity: Dropping or conflating modifiers like age, gender. Failing to encode these attributes makes generated visual media much more ambiguous.
11. Granularity (Intensity): Conflating verbs that describe different types of motion or action. This can lead to inaccuracies in generated video or animation, as the type of motion and action is core to visualizing a concept.
Systematic failures categorized by GPT-3.5 We provide the descriptions of the 8 systematic failures categorized by MULTIMON using MS-COCO and SNLI as the corpus and GPT-3.5 as categorizer.