SlowGuess/ACL_2020_OCR huggingface dataset:
Beyond Accuracy: Behavioral Testing of NLP Models with CHECKLIST
<sup>1</sup>Microsoft Research
Tongshuang Wu2
Sameer Singh<sup>3</sup>
$^{2}$ University of Washington
<sup>3</sup>University of California, Irvine
marcotcr@gmail.com {wtshuang, guestrin}@cs.uw.edu sameer@uci.edu
Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.
One of the primary goals of training NLP models is generalization. Since testing "in the wild" is expensive and does not allow for fast iterations, the standard paradigm for evaluation is using train-validation-test splits to estimate the accuracy of the model, including the use of leader boards to track progress on a task (Rajpurkar et al., 2016). While performance on held-out data is a useful indicator, held-out datasets are often not comprehensive, and contain the same biases as the training data (Rajpurkar et al., 2018), such that real-world performance may be overestimated (Patel et al., 2008; Recht et al., 2019). Further, by summarizing the performance as a single aggregate statistic, it becomes difficult to figure out where the model is failing, and how to fix it (Wu et al., 2019).
A number of additional evaluation approaches have been proposed, such as evaluating robustness to noise (Belinkov and Bisk, 2018; Rychalska et al., 2019) or adversarial changes (Ribeiro et al., 2018; Iyyer et al., 2018), fairness (Prabhakaran et al., 2019), logical consistency (Ribeiro et al., 2019), explanations (Ribeiro et al., 2016), diagnostic datasets (Wang et al., 2019b), and interactive error analysis (Wu et al., 2019). However, these approaches focus either on individual tasks such as Question Answering or Natural Language Inference, or on a few capabilities (e.g. robustness), and thus do not provide comprehensive guidance on how to evaluate models. Software engineering research, on the other hand, has proposed a variety of paradigms and tools for testing complex software systems. In particular, "behavioral testing" (also known as black-box testing) is concerned with testing different capabilities of a system by validating the input-output behavior, without any knowledge of the internal structure (Beizer, 1995). While there are clear similarities, many insights from software engineering are yet to be applied to NLP models.
In this work, we propose CheckList, a new evaluation methodology and accompanying tool for comprehensive behavioral testing of NLP models. CheckList guides users in what to test, by providing a list of linguistic capabilities, which are applicable to most tasks. To break down potential capability failures into specific behaviors, CheckList introduces different test types, such as prediction invariance in the presence of certain perturbations, or performance on a set of "sanity checks." Finally, our implementation of CheckList includes multiple abstractions that help users generate large numbers of test cases easily, such as templates, lexicons, general-purpose perturbations, visualizations, and context-aware suggestions.
'https://github.com/marcotcr/checklist
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Capability	Min Func Test	INVariance	DIRECTIONAL
Vocabulary	Fail. rate=15.0%	16.2%	34.6%
NER	0.0%	B 20.8%	N/A
Negation	A 76.4%	N/A	N/A
…
Test case	Expected	Predicted	Pass?
Testing Negation with MFT
Labels: negative, positive, neutral
Template: I {NEGATION} {POS_PERB} the {THING}.
I can't say I recommend the food.	neg	pos	X
I didn't love the flight.	neg	neutral	X
Failure rate = 76.4%
Testing NER with INV Same pred. (inv) after removals / additions
@AmericanAir thank you we got on a different flight to [Chicago → Dallas].	inv	pos neutral	X
@VirginAmerica I can't lose my luggage, moving to [Brazil → Turkey] soon, ugh.	inv	neutral neg	X
Failure rate = 20.8%
Testing Vocabulary with DIR Sentiment monotonic decreasing (↓)
@AmericanAir service wasn't great. You are lame.	↓	neg neutral	X
@JetBlue why won't YOU help them?! Ugh. I dread you.	↓	neg neutral	X
Failure rate = 34.6%
Figure 1: CHECKListing a commercial sentiment analysis model  $(\mathbf{G})$ . Tests are structured as a conceptual matrix with capabilities as rows and test types as columns (examples of each type in A, B and C).
As an example, we CHECKLIST a commercial sentiment analysis model in Figure 1. Potential tests are structured as a conceptual matrix, with capabilities as rows and test types as columns. As a test of the model's Negation capability, we use a Minimum Functionality test (MFT), i.e. simple test cases designed to target a specific behavior (Figure 1A). We generate a large number of simple examples filling in a template ("I {NEGATION} {POS_VERB} the {THING}."). with pre-built lexicons, and compute the model's failure rate on such examples. Named entity recognition (NER) is another capability, tested in Figure 1B with an Invariance test (INV) - perturbations that should not change the output of the model. In this case, changing location names should not change sentiment. In Figure 1C, we test the model's Vocabulary with a Directional Expectation test (DIR) - perturbations to the input with known expected results - adding negative phrases and checking that sentiment does not become more positive. As these examples indicate, the matrix works as a guide, prompting users to test each capability with different test types.
We demonstrate the usefulness and generality of CheckList via instantiation on three NLP tasks: sentiment analysis (Sentiment), duplicate question
detection (QQP; Wang et al., 2019b), and machine comprehension (MC; Rajpurkar et al., 2016). While traditional benchmarks indicate that models on these tasks are as accurate as humans, CheckList reveals a variety of severe bugs, where commercial and research models do not effectively handle basic linguistic phenomena such as negation, named entities, coreferences, semantic role labeling, etc, as they pertain to each task. Further, CheckList is easy to use and provides immediate value – in a user study, the team responsible for a commercial sentiment analysis model discovered many new and actionable bugs in their own model, even though it had been extensively tested and used by customers. In an additional user study, we found that NLP practitioners with CheckList generated more than twice as many tests (each test containing an order of magnitude more examples), and uncovered almost three times as many bugs, compared to users without CheckList.
2 CHECKLIST
Conceptually, users "CHECKLIST" a model by filling out cells in a matrix (Figure 1), each cell potentially containing multiple tests. In this section, we go into more detail on the rows (capabilities), columns (test types), and how to fill the cells (tests). CHECKLIST applies the behavioral testing principle of "decoupling testing from implementation" by treating the model as a black box, which allows for comparison of different models trained on different data, or third-party models where access to training data or model structure is not granted.
2.1 Capabilities
While testing individual components is a common practice in software engineering, modern NLP models are rarely built one component at a time. Instead, CheckList encourages users to consider how different natural language capabilities are manifested on the task at hand, and to create tests to evaluate the model on each of these capabilities. For example, the Vocabulary  $+POS$  capability pertains to whether a model has the necessary vocabulary, and whether it can appropriately handle the impact of words with different parts of speech on the task. For Sentiment, we may want to check if the model is able to identify words that carry positive, negative, or neutral sentiment, by verifying how it behaves on examples like "This was a good flight." For QQP, we might want the model to
understand when modifiers differentiate questions, e.g. accredited in ("Is John a teacher?", "Is John an accredited teacher?"). For  $MC$ , the model should be able to relate comparatives and superlatives, e.g. (Context: "Mary is smarter than John", Q: "Who is the smartest kid?", A: "Mary").
We suggest that users consider at least the following capabilities: Vocabulary  $^+$  POS (important words or word types for the task), Taxonomy (synonyms, antonyms, etc), Robustness (to typos, irrelevant changes, etc), NER (appropriately understanding named entities), Fairness, Temporal (understanding order of events), Negation, Coreference, Semantic Role Labeling (understanding roles such as agent, object, etc), and Logic (ability to handle symmetry, consistency, and conjunctions). We will provide examples of how these capabilities can be tested in Section 3 (Tables 1, 2, and 3). This listing of capabilities is not exhaustive, but a starting point for users, who should also come up with additional capabilities that are specific to their task or domain.
2.2 Test Types
We prompt users to evaluate each capability with three different test types (when possible): Minimum Functionality tests, Invariance, and Directional Expectation tests (the columns in the matrix).
A Minimum Functionality test (MFT), inspired by unit tests in software engineering, is a collection of simple examples (and labels) to check a behavior within a capability. MFTs are similar to creating small and focused testing datasets, and are particularly useful for detecting when models use shortcuts to handle complex inputs without actually mastering the capability. The Vocabulary+POS examples in the previous section are all MFTs.
We also introduce two additional test types inspired by software metamorphic tests (Segura et al., 2016). An Invariance test (INV) is when we apply label-preserving perturbations to inputs and expect the model prediction to remain the same. Different perturbation functions are needed for different capabilities, e.g. changing location names for the NER capability for Sentiment (Figure 1B), or introducing typos to test the Robustness capability. A Directional Expectation test (DIR) is similar, except that the label is expected to change in a certain way. For example, we expect that sentiment will not become more positive if we add "You are lame." to the end of tweets directed at an airline (Figure 1C). The expectation may also be a target
label, e.g. replacing locations in only one of the questions in  $QQP$ , such as ("How many people are there in England?", "What is the population of England  $\rightarrow$  Turkey?"), ensures that the questions are not duplicates. INVs and DIRs allow us to test models on unlabeled data – they test behaviors that do not rely on ground truth labels, but rather on relationships between predictions after perturbations are applied (invariance, monotonicity, etc).
2.3 Generating Test Cases at Scale
Users can create test cases from scratch, or by perturbing an existing dataset. Starting from scratch makes it easier to create a small number of high-quality test cases for specific phenomena that may be underrepresented or confounded in the original dataset. Writing from scratch, however, requires significant creativity and effort, often leading to tests that have low coverage or are expensive and time-consuming to produce. Perturbation functions are harder to craft, but generate many test cases at once. To support both these cases, we provide a variety of abstractions that scale up test creation from scratch and make perturbations easier to craft.
Templates Test cases and perturbations can often be generalized into a template, to test the model on a more diverse set of inputs. In Figure 1 we generalized "I didn't love the food." with the template "I {NEGATION} {POS_VERB} the {THING}.", where {NEGATION} = {didn't, can't say I, ...}, {POS_VERB} = {love, like, ...}, {THING} = {food, flight, service, ...}, and generated all test cases with a Cartesian product. A more diverse set of inputs is particularly helpful when a small set of test cases could miss a failure, e.g. if a model works for some forms of negation but not others.
Expanding Templates While templates help scale up test case generation, they still rely on the user's creativity to create fill-in values for each
Figure 2: Templating with masked language models. "I really {mask} the flight." yields verbs that the user can interactively filter into positive, negative, and neutral fill-in lists.
Table 1: A selection of tests for sentiment analysis. All examples (right) are failures of at least one model.
We provide users with an abstraction where they mask part of a template and get masked language model (RoBERTa (Liu et al., 2019) in our case) suggestions for fill-ins, e.g. "I really {mask} the flight." yields {enjoyed, liked, loved, regret, ...}, which the user can filter into positive, negative, and neutral fill-in lists and later reuse across multiple tests (Figure 2). Sometimes RoBERTa suggestions can be used without filtering, e.g. "This is a good {mask}" yields multiple nouns that don't need filtering. They can also be used in perturbations, e.g. replacing neutral words like that or the for other words in context (Vocabulary+POS INV examples in Table 1). RoBERTa suggestions can be combined with WordNet categories (synonyms, antonyms, etc), e.g. such that only context-appropriate synonyms get selected in a perturbation. We also provide additional common fill-ins for general-purpose categories, such as Named Entities (common male and female first/last names, cities, countries) and protected group adjectives (nationalities, religions, gender and sexuality, etc).
Open source We release an implementation of CheckList at https://github.com/marcotcr/checklist. In addition to templating features and mask language model suggestions, it contains various visualizations, abstractions for writing test expectations (e.g. monotonicity) and perturbations, saving/sharing tests and test suites such that tests can be reused with different models and by different teams, and general-purpose perturbations such as char swaps (simulating typos), contractions, name and location changes (for NER tests), etc.
3 Testing SOTA models with CHECKLIST
We CHECKLIST the following commercial Sentiment analysis models via their paid APIs: Microsoft's Text Analytics (■), Google Cloud's Natural Language (G), and Amazon's Comprehend (▲). We also CHECKLIST BERT-base (▲) and RoBERT-base (RoB) (Liu et al., 2019) finetuned on SST-2³ (acc: 92.7% and 94.8%) and on the QQP dataset
2From 11/2019, but obtained similar results from 04/2020. 3Predictions with probability of positive sentiment in the  $(1 / 3,2 / 3)$  range are considered neutral.
Table 2: A selection of tests for Quora Question Pair. All examples (right) are failures of at least one model.
(acc:  $91.1\%$  and  $91.3\%$ ). For MC, we use a pretrained BERT-large finetuned on SQuAD (Wolf et al., 2019), achieving 93.2 F1. All the tests presented here are part of the open-source release, and can be easily replicated and applied to new models.
Sentiment Analysis Since social media is listed as a use case for these commercial models, we test on that domain and use a dataset of unlabeled airline tweets for  $\mathrm{INV}^4$  and DIR perturbation tests. We create tests for a broad range of capabilities, and present subset with high failure rates in Table 1. The Vocab.+POS MFTs are sanity checks, where we expect models to appropriately handle common neutral or sentiment-laden words. and RoB do poorly on neutral predictions (they were trained on binary labels only). Surprisingly, G and fail  $7.6\%$  and  $4.8\%$  ) on sentences that are clearly neutral, with G also failing  $(15\%)$  on nonneutral sanity checks (e.g. "I like this seat.). In the DIR tests, the sentiment scores predicted by and G frequently  $(12.6\%$  and  $12.4\%$  ) go down con
siderably when clearly positive phrases (e.g. "You are extraordinary.") are added, or up (G:  $34.6\%$  for negative phrases (e.g. "You are lame").
All models are sensitive to addition of random (not adversarial) shortened URLs or Twitter handles (e.g.  $24.8\%$  of  $\underline{\mathbf{a}}$  predictions change), and to name changes, such as locations ( $\mathbf{G}$ :  $20.8\%$ ,  $\underline{\mathbf{a}}$ :  $14.8\%$ ) or person names ( $\mathbf{G}$ :  $15.1\%$ ,  $\underline{\mathbf{a}}$ :  $9.1\%$ ). None of the models do well in tests for the Temporal, Negation, and SRL capabilities. Failures on negations as simple as "The food is not poor" are particularly notable, e.g.  $\mathbf{G}$  ( $54.2\%$ ) and  $\underline{\mathbf{a}}$ , ( $29.4\%$ ). The failure rate is near  $100\%$  for all commercial models when the negation comes at the end of the sentence (e.g. "I thought the plane would be awful, but it wasn't"), or with neutral content between the negation and the sentiment-laden word.
Commercial models do not fail simple Fairness sanity checks such as "I am a black woman." (template: "I am a {PROTECTED} {NOUN}."). always predicting them as neutral. Similar to software engineering, absence of test failure does not imply that these models are fair - just that they are not unfair enough to fail these simple tests. On
4For all the INV tests, models fail whenever their prediction changes and the probability changes by more than 0.1.
Test TYPE and Description	Failure Rate (♂)	Example Test cases (with expected behavior and prediction)
Vocab	MFT: comparisons	20.0	C: Victoria is younger than Dylan.Q: Who is less young? A: Dylan♂: Victoria
MFT: intensifiers to superlative: most/least	91.3	C: Anna is worried about the project. Matthew is extremely worried about the project.Q: Who is least worried about the project? A: Anna♂: Matthew
Taxonomy	MFT: match properties to categories	82.4	C: There is a tiny purple box in the room. Q: What size is the box? A: tiny♂: purple
MFT: nationality vs job	49.4	C: Stephanie is an Indian accountant.Q: What is Stephanie's job? A: accountant♂: Indian accountant
MFT: animal vs vehicles	26.2	C: Jonathan bought a truck. Isabella bought a hamster.Q: Who bought an animal? A: Isabella♂: Jonathan
MFT: comparison to antonym	67.3	C: Jacob is shorter than Kimberly.Q: Who is taller? A: Kimberly♂: Jacob
MFT: more/less in context, more/less antonym in question	100.0	C: Jeremy is more optimistic than Taylor.Q: Who is more pessimistic? A: Taylor♂: Jeremy
Robust.	INV: Swap adjacent characters in Q (typ)	11.6	C: ...Newcomen designs had a duty of about 7 million, but most were closer to 5 million...Q: What was the ideal duty → udty of a Newcomen engine? A: INV♂: 7 million → 5 million
INV: add irrelevant sentence to C	9.8	(no example)
Temporal	MFT: change in one person only	41.5	C: Both Luke and Abigail were writers, but there was a change in Abigail, who is now a model.Q: Who is a model? A: Abigail♂: Abigail were writers, but there was a change in Abigail
MFT: Understanding before/after, last/first	82.9	C: Logan became a farmer before Danielle did.Q: Who became a farmer last? A: Danielle♂: Logan
Neg.	MFT: Context has negation	67.5	C: Aaron is not a writer. Rebecca is. Q: Who is a writer? A: Rebecca♂: Aaron
MFT: Q has negation, C does not	100.0	C: Aaron is an editor. Mark is an actor. Q: Who is not an actor? A: Aaron♂: Mark
Coref.	MFT: Simple coreference, he/she.	100.0	C: Melissa and Antonio are friends. He is a journalist, and she is an adviser.Q: Who is a journalist? A: Antonio♂: Melissa
MFT: Simple coreference, his/her.	100.0	C: Victoria and Alex are friends. Her mom is an agentQ: Whose mom is an agent? A: Victoria♂: Alex
MFT: former/latter	100.0	C: Kimberly and Jennifer are friends. The former is a teacherQ: Who is a teacher? A: Kimberly♂: Jennifer
SRL	MFT: subject/object distinction	60.8	C: Richard bothers Elizabeth. Q: Who is bothered? A: Elizabeth♂: Richard
MFT: subj/obj distinction with 3 agents	95.7	C: Jose hates Lisa. Kevin is hated by Lisa. Q: Who hates Kevin? A: Lisa♂: Jose
Table 3: A selection of tests for Machine Comprehension.
the other hand, always predicts negative when {PROTECTED} is black, atheist, gay, and lesbian, while predicting positive for Asian, straight, etc.
With the exception of tests that depend on predicting "neutral", and RoB did better than all commercial models on almost every other test. This is a surprising result, since the commercial models list social media as a use case, and are under regular testing and improvement with customer feedback, while and RoB are research models trained on the SST-2 dataset (movie reviews). Finally, and RoB fail simple negation MFTs, even though they are fairly accurate (91.5%, 93.9%, respectively) on the subset of the SST-2 validation set that contains negation in some form (18% of instances). By isolating behaviors like this, our tests are thus able to evaluate capabilities more precisely, whereas performance on the original dataset can be misleading.
Quora Question Pair While  $\hat{\mathbf{Q}}$  and RoB surpass human accuracy on QQP in benchmarks (Wang et al., 2019a), the subset of tests in Table 2 indicate that these models are far from solving the ques
tion paraphrase problem, and are likely relying on shortcuts for their high accuracy.
Both models lack what seems to be crucial skills for the task: ignoring important modifiers on the Vocab. test, and lacking basic Taxonomy understanding, e.g. synonyms and antonyms of common words. Further, neither is robust to typos or simple paraphrases. The failure rates for the NER tests indicate that these models are relying on shortcuts such as anchoring on named entities too strongly instead of understanding named entities and their impact on whether questions are duplicates.
Surprisingly, the models often fail to make simple Temporal distinctions (e.g. is≠used to be and before≠after), and to distinguish between simple Coreferences (he≠she). In SRL tests, neither model is able to handle agent/predicate changes, or active/passive swaps. Finally,  $\hat{\mathbf{Q}}$  and RoB change predictions  $4.4\%$  and  $2.2\%$  of the time when the question order is flipped, failing a basic task requirement (if  $q_{1}$  is a duplicate of  $q_{2}$ , so is  $q_{2}$  of  $q_{1}$ ). They are also not consistent with Logical implications of their predictions, such as transitivity.
Machine Comprehension  $\text{Vocab} + \text{POS}$  tests in Table 3 show that  $\hat{\mathbf{u}}$  often fails to properly grasp intensity modifiers and comparisons/superlatives. It also fails on simple Taxonomy tests, such as matching properties (size, color, shape) to adjectives, distinguishing between animals-vehicles or jobs-nationalities, or comparisons involving antonyms.