SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengbang Tong* Erik Jones* Jacob Steinhardt
{tsb, erjones, jsteinhardt}@berkeley.edu
Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MULTIMON, a system that automatically identifies systematic failures—generalizable, natural-language descriptions of patterns of model failures. To uncover systematic failures, MULTIMON scrapes a corpus for examples of erroneous agreement: inputs that produce the same output, but should not. It then prompts a language model (e.g., GPT-4) to find systematic patterns of failure and describe them in natural language. We use MULTIMON to find 14 systematic failures (e.g., “ignores quantifiers”) of the CLIP text-encoder, each comprising hundreds of distinct inputs (e.g., “a shelf with a few/many books”). Because CLIP is the backbone for most state-of-the-art multimodal models, these inputs produce failures in Midjourney 5.1, DALL-E, VideoFusion, and others. MULTIMON can also steer towards failures relevant to specific use cases, such as self-driving cars. We see MULTIMON as a step towards evaluation that autonomously explores the long tail of potential system failures.
Text-based multimodal systems, which produce images [Rombach et al., 2022], 3d scenes [Poole et al., 2022], and videos [Singer et al., 2022] from text, are extensively tested for failures during development, yet routinely fail at deployment [Rando et al., 2022]. This gap exists in part because evaluators struggle to anticipate and test for all possible failures beforehand.
To close this gap, we seek evaluation systems for multimodal models that are systematic and human-compatible. Systematic evaluations must peer into the long tail of possible model behaviors; this means that systems cannot assume a priori what behaviors to look for, or be bottlenecked by human labor. Human-compatible evaluations must be useful to the system designer; this means they should describe patterns of behavior beyond giving examples, and be steerable towards the designer's goals.
Towards satisfying these desiderata, we construct a system, MULTIMON, that uses large language models to identify failures of multimodal systems (Section 3). MULTIMON scrapes individual failures from a corpus, categorizes them into systematic failures (expressed in natural language), then flexibly generates novel instances. MULTIMON works autonomously, improves as language models scale, and produces failures that transfer across a range of multimodal systems.
To systematically scrape for individual failures, MULTIMON exploits erroneous agreement. Specifically, we observe that if two inputs produce the same output but have different semantics, at least one of them must be wrong. We can test whether two inputs produce the same output by comparing their CLIP embeddings, since many multimodal models encode inputs with CLIP before generating
$^{2}$ Code for MULTIMON is available at https://github.com/tsb0601/MultiMon
MidJourney 5.1 "an empty glass"
DALL-E (New Bing) "a family of five members"
Stable Diffusion XL "a man descending a mountain"
Figure 1: Examples failures that MULTIMON generates on state-of-the-art text-to-image systems.
Stable Diffusion 2.1
outputs. Using CLIP similarity circumvents the expensive decoding step of these models, allowing us to tractably scrape large corpora for failures.
With these scraped individual failures as a foundation, MULTIMON next uses language models to produce human-compatible explanations. Specifically, we use GPT-4 to identify systematic failures: generalizable natural-language descriptions of patterns of failures, from the scraped individual failures. These systematic failures are useful both to qualitatively understand system behavior and to generate new instances. We can even steer generation towards specific attributes, e.g. "salient to self-driving", that are missing from the original corpus but are important for downstream applications.
To evaluate MULTIMON, we measure the quantity and quality of the systematic failures. We measure quantity by counting the number of systematic failures generated, and quality by measuring what fraction of the new generated instances have high CLIP similarity.
We find that MULTIMON uncovers 14 systematic failures of the CLIP text-encoder, and from them over one thousand new individual failures (Section 4). The systematic failures include failing to encode negation, spatial differences, numerical differences, role ambiguity, quantifiers, and more. These systematic failures are high quality; 12 of the 14 systematic failures produce pairs with high CLIP similarity at least half the time, and 7 produce such pairs at least  $75\%$  of the time.
The failures of the CLIP text-encoder transfer to downstream text-to-image, text-to-video, and text-to-3d systems (Figure 1, Section 5). We assess the new individual failures that MULTIMON generates on five widely-used text-to-image systems: Stable Diffusion 1.5, Stable Diffusion 2.1, Stable Diffusion XL, DALL-E, and Midjourney 5.1, three of which were released within a month of the writing of this paper. Through a manual evaluation, we find that the systems err on  $80.0\%$  of the pairs generated by MULTIMON, compared to only  $20.5\%$  for a baseline system. We also show that MULTIMON can help evaluators identify inputs that evade commercial safety filters (Appendix F). Overall, the MULTIMON pipeline—exploiting erroneous agreement to scrape individual failures and finding patterns with language models—is simple and general, and could be a foundation for broader automatic evaluation.
Text-guided multimodal models. We study failures of text-guided multimodal models, which generate images [Rombach et al., 2022, Ramesh et al., 2022, 2021], video [Singer et al., 2022, Luo et al., 2023], and 3d-scenes [Jun and Nichol, 2023, Poole et al., 2022, Lin et al., 2022], to name a few output modalities, from textual descriptions. These models tend to first encode text with a vision-language model (VLM), which embeds text and images in a shared embedding space [Radford et al., 2021, Ramesh et al., 2022]. They then generate outputs via a guided diffusion process
Ambiguities and bias in embedding models. MULTIMON exploits failures of the CLIP embedding to produce failures of multimodal systems. This builds off of prior work documenting failures in text
embedding models [Bolukbasi et al., 2016, Caliskan et al., 2017, Gonen and Goldberg, 2019, May et al., 2019, Sun et al., 2019], including showing that BERT struggles to encode negation [Ettinger, 2020] and large numbers [Wallace et al., 2019]. Some work uncovers failures of vision-language embedding models themselves using benchmarks. For example, Thrush et al. [2022] and Yuksekgonul et al. [2023] show that vision-language-models often fail to account for different word orderings.
The closest work to ours is Song et al. [2020], which aims to adversarially construct pairs of inputs that embedding models should not encode similarly, but do. This work could potentially replace MULTIMON's scraping step by generating adversarially constructed pairs without a corpus.
Systematic failures. MULTIMON aims to automatically identify systematic failures of multimodal systems, without knowing what the failures are a priori. A related line of work automatically identifies slices of data that classifiers perform poorly on, then uses a VLM to choose a slice description [Eyuboglu et al., 2022, Jain et al., 2022, Gao et al., 2022, Wiles et al., 2022, Metzen et al., 2023, Zhang et al., 2023]. The main differences to our approach are (i) we do not make use of ground-truth labels and (ii) we generate candidate systematic failures, rather than testing predefined descriptions.
Other work uses humans to conjecture potential systematic failures of generative systems, then shows that models exhibit them. These failures include biases [Maluleke et al., 2022, Grover et al., 2019], propagated stereotypes [Sheng et al., 2019, Abid et al., 2021, Hemmatian and Varshney, 2022, Blodgett et al., 2021], and training data leaks [Carlini et al., 2021, 2023]. Liang et al. [2022] capture many language model behaviors via holistic evaluation, while other work surveys additional failures [Bender et al., 2021, Bommasani et al., 2021, Weidinger et al., 2021]. Towards making this evaluation more systematic, Jones and Steinhardt [2022] use cognitive biases to identify and test for systematic failures of code models, while Perez et al. [2022b] use language models to generate instances of conjectured systematic failures. Nushi et al. [2018] develop a system to help humans identify systematic failures, which they test on a captioning system.
Automated ways to produce individual failures. MULTIMON builds on work that uses a specification of a class of failures to find examples. Perez et al. [2022a] fine-tune a language model to find failures of a second language model, Jones et al. [2023] find language model failures directly using discrete optimization, and Wen et al. [2023] use discrete optimization to find prompts that a text-guided diffusion model generates a specific image from. Towards scraping corpora to find failures without direct supervision, Gehman et al. [2020] scrape a corpus for text that precedes toxic content, which they find often generates toxic text under a language model.
Using language models to draw conclusions from instances. MULTIMON generates systematic failures by identifying patterns in scraped instances. This builds on a recent line of work that uses large autoregressive language models [Radford et al., 2018, 2019, Brockman et al., 2023, Anthropic, 2023, OpenAI, 2023b] to draw general conclusions from individual instances. Zhong et al. [2022] describe differences in text distributions, Singh et al. [2022] try to explain prediction patterns, and Bills et al. [2023] use activation values to explain model neurons. The closest work to our categorization step is Zhong et al. [2023], which describe differences in distributions that are salient to a target goal.
3 The MULTIMON Pipeline
We now describe our system, multimodal monitor (MULTIMON), which finds failures of the CLIP text embedding model. We check that these failures transfer to downstream systems in Section 5.
3.1 Constructing MULTIMON
In this section, we describe MULTIMON's three steps, depicted in Figure 2. MULTIMON first scrapes a large corpus of sentences for individual failures, which are pairs of sentences that produce the same output, but should not (e.g., "a table with a few cups", "a table with many cups"). It then categorizes these instances into systematic failures, which are generalizable, natural-language descriptions of patterns of failure (e.g., "Quantifiers: models fail to distinguish between quantifiers like "few", "some", or "many"). It finally generates new candidate individual failures and checks their validity.
Scraping. MULTIMON first scrapes a corpus to collect an initial set of individual failures. To do this, it considers every possible pair of examples from corpus, then returns pairs that produce similar outputs, but are semantically different—this means that at least one output is incorrect.
Figure 2: The MULTIMON pipeline. Left. MULTIMON starts with a corpus of sentences (dots), then identifies individual failures: pairs that have similar CLIP embeddings but should not (circled red dots). Center. MULTIMON takes the individual failures, then categorizes them into systematic failures using a language model. Right. MULTIMON takes the systematic failures, then generates new individual failures from them using a language model, which then generate incorrect images.
To measure whether two inputs produce similar outputs, we compare their CLIP embeddings, since many multimodal models encode inputs with CLIP before generating outputs. To measure whether inputs have different semantics, we compare them under a reference embedding model (in our case, DistillRoBERTA). We return the  $n$  pairs of inputs with highest CLIP cosine similarity, such that the cosine similarity of their reference embeddings is below a threshold  $\tau$ . This process is automatic and, importantly, efficient: by exploiting the CLIP embedding bottleneck of multimodal models, we avoid ever running their decoders, which can be very expensive (e.g., generating a video or 3d-image).
Categorizing. After scraping many individual failures, MULTIMON categorizes them into general systematic failures. To do so, MULTIMON queries a language model with the prompt below ([...] indicates further text that is omitted here for space; see Appendix B.1 for the full prompt).
I will provide a series of examples for you to remember. Subsequently, I will ask you [...] [n individual failures]
The above are some pairs of sentences that an embedding model encodes very similarly.
Using these examples, are there general types of failures that the embedding model is making? Give failures that are specific enough that someone could reliably produce [...]
We choose  $n$  such that this prompt fits in the model's context window. Empirically, the language model always produces a list of systematic failures under our prompt, which can be parsed automatically. For example, the first items in the list that the language model (in this case GPT-4) generates are
Model output:
1. Negation: Embedding models may not correctly capture the negative context in a sentence, leading to similarities between sentences with and without negation,
2. Temporal Differences: Embedding models might not differentiate between events happening in the past, present, or future.
To generate more systematic failures, the language model can be queried multiple times with the same prompt, as language models often generate outputs stochastically.
Generating. MULTIMON's final step is generation, where it starts with the systematic failures from the categorization step, then queries a language model to generate arbitrarily many new individual failures. To do so, MULTIMON queries a language model with the prompt below.
Write [m] pairs of sentences that an embedding model with the following failure might encode similarly, even though they would correspond to different images if used [...] [Description of systematic failure]
See Appendix B.2 for the full prompt. We set  $m$  to be the maximum number of examples the generator can empirically produce in a single response. To generate subsequent instances, we query the language model in the same dialog session with the same prompt (but add "additional" after [m]).
3.2 Steering MULTIMON
Our construction of MULTIMON outputs systematic and individual failures that capture system behavior, but may not be relevant to specific use-cases. To remedy this, we next show how to steer MULTIMON towards failures in a specific subdomain of interest. MULTIMON can be steered during the scraping process (by choosing different individual failures to categorize), and during the generation process (by prompting language models to generate salient failures).
Steering towards systematic failures. To steer towards systematic failures that are related to a specific subdomain of interest, we edit the scraping stage of our pipeline. Specifically, we search for pairs of examples that a classifier identifies as relevant to the target subdomain, but that still have similar CLIP and different DistilRoBERTA embeddings. Intuitively, this constrains the categorizer to find only systematic failures that arise in the subdomain of interest.
Steering towards individual failures. To steer towards individual failures that are related to the target subdomain, we edit the generation stage of our pipeline. Specifically, we append "Keep in mind, your examples should be relevant to [subdomain]" to the generation prompt from Section 3.1. We generate instances using the unmodified descriptions of systematic failures from Section 3.1.
3.3 Evaluating MULTIMON
We want systems like MULTIMON to find many high-quality systematic failures. We thus care about both the quantity and quality of failures produced, and for domain-specific use cases we also care about relevance of the failures.
To evaluate quantity, we simply count the number of systematic failures each system finds in the categorization step of the pipeline.
To evaluate the quality of a systematic failure, we measure the quality of instances generated from it. Specifically, we generate  $k$  new instances (candidate pairs) from the description of a systematic failure, using the generation step in Section 3.1. We say that a candidate pair is successful if its CLIP similarity is above a threshold  $t$ , chosen such that pairs with CLIP similarity above  $t$  tend to produce visually indistinguishable images. We then define the success rate as the percentage of the  $k$  pairs that are successful. The success rate gives a quantitative metric of how useful a qualitative description is for producing new failures.
Finally, to evaluate relevance, we test whether the systematic and individual failures are relevant to the subdomain of interest. We measure this with the relevance rate: the fraction of generated individual failures that are relevant to the subdomain of interest according to a binary classifier. We measure the relevance of systematic failures by generating new instances with the unmodified generation prompt from Section 3.1, and measure the relevance of individual failures directly.
4 Automatically Finding Failures of CLIP
In this section, we use MULTIMON to produce systematic failures, and from them new individual failures (Section 4.1), using the methods described in Section 3. We then adjust MULTIMON to steer towards specific kinds of systematic and individual failures (Section 4.2).
4.1 Identifying systematic failures of CLIP with MULTIMON
We first wish to evaluate whether MULTIMON can successfully uncover failures of the CLIP text encoder. Specifically, we aim to measure whether MULTIMON manages to find many systematic
Figure 3: We report whether each LM-corpus pair uncovers each systematic failure (checkmark), along with the success rate. Both the language model and corpus influence the systematic failures that MULTIMON uncovers. We include raw success rates and error bars in Appendix C.3.
failures, and whether these failures are high-quality, as measured by their success rates. We also wish to understand how both the language model and the input corpus affect the failures we recover.
To conduct this evaluation, we test the MULTIMON system described in Section 3. During the scraping stage, we return the  $n = 150$  pairs with highest CLIP similarity, and use a semantic similarity threshold of  $\tau = 0.7$ . For the input corpus we test both SNLI [Bowman et al., 2015] and MS-COCO Captions [Lin et al., 2014]. For the language model categorizer, we consider GPT-4 [OpenAI, 2023b], Claude v1.3 [Anthropic, 2023], and GPT-3.5 [Brockman et al., 2023], and use GPT-4 as a generator unless otherwise noted.
Assessing the quantity of systematic failures. We first examine how many systematic failures MULTIMON can produce. Specifically, we prompt each language model three times, and report the aggregate list of systematic failures that it returns in Figure 3. We find that GPT-4 identifies 14 systematic failures across the two corpora, while Claude finds 11 and GPT-3.5 finds only 8. The corpus also dictates what systematic errors MULTIMON finds; for example, only COCO uncovers temporal differences as a source of failures, and the same is true for SNLI and numerical differences.
Some of the systematic failures we uncover were found in prior work using benchmarks. Yuksekgonul et al. [2023] show that CLIP embeddings act like bag-of-words models, while Ettinger [2020] find that BERT many not encode negation. MULTIMON produces these failures autonomously, and uncovers new systematic failures in addition to these known ones.
Assessing the quality of systematic failures. We next measure the quality of the generated systematic failures, as measured by the success rate (Section 3.3). To compute success rate, we use GPT-4 to generate  $k = 82$  new instances and set the CLIP similarity threshold for success to be  $t = 0.88$  (we choose 0.88 based on an empirical study; see Section 5.1 for details).
We report the success rate in Figure 3. Overall, we find that the success rate when generating new instances is usually high, but varies across models even for the same systematic failure. For systematic failures found by all three models, GPT-4 had an average success rate of  $80.2\%$ , compared to  $83.3\%$  for Claude and  $69.5\%$  for GPT-3.5. This is because the models produce different quality descriptions (i.e., GPT-4 might produce a more detailed, useful, and faithful description of a failure than GPT-3.5).
These results demonstrate that MULTIMON already produces many high-quality systematic failures, that better language models tend to improve the systematic failures generated (suggesting that MULTIMON will continue to improve in the future), and that different input corpora find different failures (suggesting that highly diverse corpora or ensembles of corpora produce the best results).
<sup>3</sup>We choose a low  $\tau$  to aggressively avoid duplicates for the scraping stage, even though many semantically different pairs have higher DistilRoBERTa similarity.
GPT-4 could generate at most 41 pairs per query, so we query twice in the same session.
MidJourney 5.1
Prompt: "a shelf with few books"
DALL-E (New Bing)
Figure 4: Examples of inputs that MULTIMON generates. Since MULTIMON uses CLIP to find failures, a single input produces the same error in many state-of-the-art text-to-image systems.
Prompt: "a cat lying outside a box"
Ablations. Language models generate high-quality systematic failures from individual ones, but might have seen the systematic failures during training. To verify this is not the case, we prompt language models to produce systematic failures without the scraped individual failures the corpus, and find that they only identify 2 of the 14 systematic failures and that the average success rate is  $29.3\%$  (Appendix C.6). This low success rate implies that even for failures that are identified without the corpus, the resulting description is low-quality.
Secondly, all of our results use GPT-4 to generate new individual failures. To isolate the role of the language model generator and check robustness, we replace GPT-4 with Claude and GPT-3.5 when generating new failures. We find that Claude tends to produce similar success rates on average, though there is variability across different failures. In contrast, GPT-3.5 is worse (Appendix C.5). This suggests that improving language models would improve generation, in addition to categorization.
4.2 Steering MULTIMON towards specific applications
In this section, we demonstrate that evaluators can steer MULTIMON towards failures in a specific subdomain of interest, using "self-driving" as an illustrative example. As we describe in Section 3.3, MULTIMON can be steered towards systematic failures (by choosing different examples to categorize), and towards individual failures (by prompting language models to generate salient failures).
Steering towards systematic failures. We first steer towards systematic failures that are related to self-driving, by editing the scraping stage of our pipeline with the method described in Section 3.3. We use a zero-shot GPT-3.5 classifier to identify instances that are relevant to self-driving (Appendix C.7), and the same classifier to compute the relevance rate.
We report the full results in Table 8 in Appendix C.7, and find that MULTIMON generates five systematic failures that are relevant to self-driving, four of which have success rates over  $95\%$ . Moreover, the systematic failures consistently generate pairs that are relevant to the subdomain of interest; all failures have relevance rates above  $90\%$ , and four out of 5 have a  $100\%$  relevance rate.