Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs
Authors: Chenyang Yang
Rishabh Rustogi
Rachel Brower-Sinning
Grace Lewis
Christian Kaestner
Tongshuang Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: model testing
knowledge base
large language models
human-centered NLP

Abstract: Current model testing work has mostly focused on creating test cases.
Identifying what to test is a step that is largely ignored and poorly supported.
We propose Weaver, an interactive tool that supports requirements elicitation for guiding model testing.
Weaver uses large language models to generate knowledge bases and recommends concepts from them interactively, allowing testers to elicit requirements for further testing.
Weaver provides rich external knowledge to testers and encourages testers to systematically explore diverse concepts beyond their own biases. 
In a user study, we show that both NLP experts and non-experts identified more, as well as more diverse concepts worth testing when using Weaver.
Collectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT.
Our case studies further show that Weaver can help practitioners test models in real-world settings,
where developers define more nuanced application scenarios (e.g., code understanding and transcript summarization) using LLMs.

Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMs
Chenyang Yang'
Rishabh Rustogi Christian Kästner
Rachel Brower-Sinning' Grace A. Lewis?
Tongshuang Wu Carnegie Mellon Software Engineering Institute
!Carnegie Mellon University
Abstract
Requirements analysis
Acceptance test plan
Acceptance testing
Current model testing work has mostly focused on creating test cases. Identifying what to test is step that is largely ignored and poorly sup- ported. We propose WEAVER, an interactive tool that supports requirements elicitation for guiding model testing.' WEAVER language models to generate knowledge bases and recommends concepts from them interac- tively, allowing testers to elicit requirements for further testing. WEAVER provides rich ex- ternal knowledge testers, and encourages testers to systematically explore diverse con- cepts beyond their own biases. a user study, we show that both NL. experts and non-experts identified more, as well as more diverse con- cepts worth testing when using WEAVER. Col- lectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT, Our case studies further show that WEAVER help practitioners test models in real-world settings, where developers define more nuanced application scenarios (e.g.. code understanding and transcript summarization) using LLMs.
Integration test plan
Integration testing
decomposition
Model requirements
Unit test plan
Unit testing
large
Implementation
Figure 1: The V-model (Sommerville, 2015), widely used development process in many fields of engineering, adapted for testing models within system. This lay- ering and planning is also compatible with more agile development approaches, where the different activities may be iterated in different ways but are still linked.
Such model testing of nuanced behaviors usually requires translating behavior expectations into test cases (input-outpy output pairs). To enable such test case creation, prior work has taken inspiration from the long-established software testing approaches: For example, in their CheckList framework, Ribeiro et al. (2020) used templates to form minimal func- tionality test cases, which was inspired by unit tests. Morris et al. (2020)'s 's work on editing imputs (e.g.. synonym swap) for testing model invariances is akin metamorphic testing. To enable testing gen- erative models, Ribeiro (2023) have also explored specifying properties that any correct output should follow, similar to property-based testing-
1 Introduction
Despite being increasingly deployed in real-world products, ML models still suffer from false- hoods (Maynez al., 2020), biases (Shah et al., 2020), and shortcuts (Geirhos et al., 2020), lead- ing to usability, fairness, and safety issues in those products (Liang et al., 2023; Nahar et al., 2023). For example, toxicity detection models are used by social media platforms to flag or remove harm- ful content, but their biases amplify harm against minority groups (Sap et al., 2019). As standard benchmarks are often too coarse to expose these issues, recent work has proposed to test nuanced behaviors of ML models (Ribeiro et al., 2020; Goel et al., 2021; Ribeiro and Lundberg, 2022).
However, prior work has focused on how write test cases, not what tests to write. In software engineering, tests are fundamentally grounded in requirements and design work, as commonly ex- pressed in the V-model (Sommerville, 2015, Fig- ure 1). Ideally, each test can be traced back to a requirement and all requirements are tested. Soft- ware engincering research has long established the importance of requirements for development and testing. and studied many approaches for require- ments elicitation (Van Lamsweerde, 2009).
In comparison, little work has explicitly sup- ported identifying what to test in model testing. Researchers and practitioners seem to rely mostly
!Weaver is availahle open-source at https: //gi thub. hom//mallusamayo/Heaaeee

---

on intuition and generic domain knowledge (Mc- Coy et al., 2019; Dhole al., 2021). or debug a small set of issues initially identified through er- vor analysis (Naik et al., 2018; Wu et al., 2019b). Such approaches are often shaped heavily by indi- vidual knowledge and biases (Rastogi et al., 2023; Lam et al., 2023), driving practitioners to focus on local areas of a few related concepts where they find problems, while neglecting the larger space (Ribeiro and Lundberg, 2022), as exempli- fied in Figure 2. For example, to test toxicity de- tection models, practitioners may identify and test (I) racism with handcrafted test cases and (2) ro- bustness to paraphrasing with CheckList. How- ever, they are likely to miss many concepts in the space (e.g., spreading misinformation as way to spread toxicity) if they have never seen worked on similar aspects, as often observed in the fairness literature (Holstein et al., 2019).
Figure 2: Local exploitation (left) vs. global exploration (right). Most model testing is opportunistic in terms of what to test, often "hill-climbing" by searching near existing problems, with the risk of getting stuck in local areas. In contrast, WEAVER supports exploring concepts globally across the entire space.
test models in real-world settings, from transcript summarization to code understanding.
2 WEAVER
Our key idea is to provide a knowledge base to support systematic exploration of domain cepts to guide model testing. This allows testers to consider requirements broadly, mitigating their biases as testers otherwise tend to opportunisti- cally explore concepts in local areas (Figure 2). Our tool, WEAVER, has three primary building blocks: (1) An LLM-generated knowledge base for a given testing task, for encouraging more sys- tematic and diverse requirement elicitation beyond individual biases; (2) graph-inspired recommen- dation paradigm that prioritizes diverse yet relevant concepts for user inspection; and (3) an intuitive interface that can be casily paired with any test case creation methods, for supporting users to in- teractively navigate the knowledge base(s) for their own purposes. Below, we walk through the design choices and corresponding rationales.
In this work, we contribute the concept and method of requirements engineering ('what to "est") for improving model testing. We connect testing to requirements with WEAVER, an inter- active tool that supports requirements elicitation for guiding model testing. Our goal is to provide comprehensive external knowledge, encouraging testers to systematically explore diverse concepts beyond their biases (Figure 2), while balaneing the completeness of requirements collection and the effort of practitioners exploring these require- ments. WEAVER systematically generates knowl- edge bases (KB) by querying large language mod- els (LLMs) with well-defined relations from Con- ceptNet, allowing testers to elicit requirements for any use cases (as opposed to being limited by pre- built KBs). For example, in Figure 3, from seed concept "online toxicity",, the LLM generates KBB showing "spreading misinformation" with relation "done via." To not overwhelm users and encourage iterative exploration, WEAVER shows only a subset of the concepts from the KBs, balancing relevance and diversity, while allowing users to still explore more concepts on demand.
2.1 LLM-generated Knowledge Base
To support testers to explore different concepts relevant to the problem, WEAVER needs to pro- vide comprehensive knowledge beyond individ- ual testers' biases. As such, we choose to power WFAVER using knowledge bases (KBs) generated by LLMs. As LLMs store diverse world knowledge that can be easily extracted through prompting (Co- hen et al., 2023), they empower WEAVER to sup- port a wide range of domains, tasks, and topics.
We demonstrate the usefulness and generality of WFaVeR through a user study and case studies. In the user study ($4), users identified more, well more diverse concepts worth testing when using WEAYER. Collectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT (gpt-turbo-3.0 5..5). Our case studies ($5) further show that WEAVER can help practitioners
We start knowledge base construction with seed concept, i.e., a user-provided high-level term that can represent their tasks well. These seeds can be as simple as the task name and description, or can be more customized depending on user needs. Using the seed, we will then automatically query an

---

LLM build a partial knowledge base of related concepts (Wang et al., 2020; Cohen et al., 2023). Specifically, we iteratively prompt LLMs for en- tities or concepts that have different relations to the queried concept, using fluent zero-shot prompts paraphrased from well-established relations used in knowledge bases. For example. prompting LLMs with "List some types of online toxicity." can help us extract specific Of online toxicity. By default, we use 25 relations from ConceptNet' (Speer et al., 2017), e.g., MotivatedBy and LocatedAt, and man- ually curated corresponding zero-shot templates. These relations prove to be reusable across many different domains in our studies, but users can also specify custom relations they want to explore (a complete list of prompts in Appendix A).
online toxicity !
subtype
speech
morivated racism
done
people with different political beliefs
has auosype
religious discrimination
subcancepts for "hate spaach"
via cyber-bullying
has subtyse
misogyny
Q
t
O dane via
spreading misinformation
O done to
disabled people
O done by
celebrity
O located online geming
Show more subcomccppss for "online toxicily"
Figure 3: WEAVER interface, where users can interac- tively explore concepts in the LLM-generated KB to elicit requirements for model testing. @: seed concept, @: recommended children concepts with relations (tran-- lated to user-friendly form), @: options for users to expand a concept or & get more recommendations on any concepts, (B: editing, creating, or removing concepts manually, @: selecting important concepts for testing.
As the KB used to support exploration (ex- plained more below), we initially pre-generate two layers of the KB, and iteratively expand the knowl- edge base based on user interactions.
between each other, while they are still relevant in the sense that they occur frequently in the context of the queried concept. Essentially, we aim to find subgraph G' of size k: that maximizes weighted (a > 0) sum of edge weights wE (diversity) and node weights wv (relevance):
2.2 Recommend Diverse & Relevant Concepts
A comprehensive KB comes at a higher cost of exploration- a single tester can easily get over- whelmed if presented with the entire KB. To assist users to navigate through the KB more efficiently, we employ the "overview first, taxonomy (Shneiderman, 1996): We provide initial recommendations starting points to user explo- rations. In particular, we strive to make recommen- dations that are diverse (so as to bring diverse perspectives to users) but still relevant (so users do not feel disconnected from their goals).
arg max C'CG, =k "E((') a wV
To build the graph, we measure concept dif- ferences with cosine distance between concept embeddings using SentenceBERT (Reimers and Gurevych, 2019), and measure relevance with the perplexity of sentence "(concept) often occurs in the context of [queried_concceeee (Radford et al., 2019). Since finding the opti- mal subgraph is computationally expensive, we ap- ply the classic greedy peeling algorithm (Wormald, 1995) to approximate it in linear time. That is, we greedily remove nodes with the smallest weights (sum of node and all edge weights) one time until the graph size reaches (= 10 for initial rec- ommendation, but grows with user expansion). We empirically show that the recommended concepts are of high quality and diverse in our evaluation.
The trade-off between relevant and diverse onates with the common peploration-exllitattiin trade-off in information retrieval (Athukorala et al., 2016), and can be naturally translated to a graph problem: If we have all candidate concepts to form a fully-connected graph, where edge weights are distances between different concepts and node weights are the concepts' relevance to the queried concept, then our goal becomes to recommend diverse subset where concepts have large distances
using GPT-
"W used OpenAl's text-davinci-e00 (Ouyang al., 2022), but LLMS with similar instruction following capa- bilities should all he useful (see additional experiments on
2.3 Interactive Interface for Exploration
11ama-2-13b- 3b-cchat Appendix F).
Besides the recommended starting points, we al- low users iteratively and interactively locate their concepts of interest. WEAVER visualizes the knowledge base in a tree structure (Figure 3), a rep- resentation that is familiar to most ML practitioners.
We used ConceptNet relations because they represent generic semantic relations, which we expect to be more or less generalizable-aa an assumption that is validated by our user study and case studies. In contrast, alternative KBs (c.g.. WikiData, DBpedia) tend to focus on more specific types of semantic relations that are biased towards certain domains.

---

The knowledge base starts with I the seed concept users specify, with each recommended concept rep- resented as @ a node the tree, accompanied by its relation to the parent concept. The interface allows users to identify concepts by diving deeper and @ exploring broadly before @ selecting concept to test. Alternatively, users can also distill personal knowledge by @ creating concepts manually.
We evaluate the comprehensiveness of the gen- crated knowledge using recall, i.c., the fraction of existing concepts that also appear the KB. Since there are many phrasing variations of the same con- cept, we decide that a concept is in the KB if it appears exactly the same in the KB, or our manual check decides that it matches one of the 10 most similar concepts from the KB, as measured by the cosine distance (ef. $2.1). We established that the manual process is reliable by evaluating inter-rater reliability where two authors independently labeled a random sample of 50 concepts, finding substan- tial agreement (ı: 69.4%).
To assist users in creating concrete test cases, WEAVER incorporates AdaTest (Ribeiro and Lund- berg, 2022) as the default test case creation method, which uses LLMs to suggest test cases. However, the design of WEAVER is compatible with any other techniques to test models once requirements are identified ge.g., Zeno, Cabrera et al., 2023). The full interface including the AdaTest integration can be seen in Appendix B.
We also evaluate the validity of the generated knowledge using precision, i.e., the fraction of KB edges that are valid. Note that because our ground truths incomplete by nature (collected from dataset analysis and user study), KB edges that are not in the ground truths can still be valid. Following prior work (Cohen et al., 2023), we performed man- ual validation on sampled KB edges. We sampled 50 edges from each of the four generated KBs.
3 Intrinsic Evaluation
As the primary goal of WEAVER is to provide ex- ternal knowledge to guide testing, it is important that the knowledge provided is comprehensive in the first place. Here, we quantitatively evaluate: Q.I How comprehensive are the knowledge bases generated by WEAVER?
Results Overall, our KBs cover 91% of ground- truth concepts on average (Table 1), with 81% of sampled generated edges being valid. Qualitatively, we found that there are two distinct types of con- cepts the KB failed to cover: First, there are some very specific concepts (e.g., old photo in hateful meme detection). Although the 2-layer KB does not cover them, it does often cover their hyper- nyms (e.g.. photo). Therefore, these concepts be discovered if users choose to explore deeper. Second, some concepts are interactions of two con- cepts fossil fuel uses in developing countries in climate change stance detection). These can be identified by users manually, as both of their com- ponents (fossil fuel uses and developing countries) usually already exist in the KB.
Tasks, data, and metrics. We select four tasks for the evaluation: Hateful meme detection, Pedes- trian detection, Stance detection for feminism, and Stance detection for climate change (task descrip- tions in Appendix C). These tasks cover diverse domains and modalities, and importantly, provide with gold concepts that can be used to evaluate our L.LM-generatee been studied in prior work, and we directly use their ground-truth truth concepts collected from existing documents (Barzamini et al., 2022b) and user stud- ies (Lam et al., 2023). For the last two tasks, we aggregate all concepts identified by 20 participants without using WEAVER as part of our user study (discussed later in $4), which we consider our ground truth. Intuitively, such aggregation should help represent what concepts are generally deemed important. As shown in Table 1, the tasks have on average 144 ground-truth concepts.*
KB. The first two tasks have
4 User Study
Does WEAVER support effective requirements clic- itation? We conduct a user study to evaluate:
Q.2 To what degree does WEAVER help users ex- plore concepts faster?
Independently, we generated a knowledge base for each task using WEAVER with default relations.
Q.3 To what degree does WRAVER help users ex- plore concepts broadly?
We derived the seed concepts directly from the task names: (1) "hateful meme", (2) "pedestrian", (3) "Seminism", and (4) "climate change."
Q.4 How much does WEAVER miligate user biases when exploring concepts?
We expect that our interaction design (52.3) sup- ports faster exploration (Q.2) and that the recom- mendations ($2.2) support broader and less biased
'All ground-truth concepts are shared at https:// 111/4488699FF5 a69 b36dd766888.
fi Figshare.com com/s/481t

---

Task
Recall
Precision
# Concept
Hateful meme detection
93.1% %
88.0%
101
Pedestrian detection
91.8%
74.0%
146
Stance detection for feminism
86.9%
84.0%
145
Stance det. for climate change
91.4%
76.0%
185
Average
90.6%
80.5%
144
Table I: Knowledge bases generated by WEAVER cover 90.6% of existing concepts on average.
exploration (Q.3 and Q.4).
of model testing is designed ground participants in what concepts are worth testing. The second phase of concept exploration is designed approx- imate longer time of model testing. This final de- sign was derived from an earlier pilot study, where we observed that writing test cases for each con- cept took more time than identifying interesting concepts (WEAVER's objective). In the end, par- ticipants filled out post-study survey (details in Appendix D.2). Participants were compensated for their time.
4.1
Study Design
Conditions. We design IRB-approved user study within-subject controlled experiment, where participants test models in two conditions: A treatment condition, where users use WEAVER to find concepts for testing, and a control condition, where users add the concepts manually while they explore test cases. In both conditions, users have access to AdaTest's LLM-based test case sugges- tions (cf. $2.3). In essence, the control interface is a re-implementation of AdaTest with WEAVER's interface and interaction experience.
Metrics and analysis. We use two measurements to approximate participants' exploration procedure: (1) the number of concepts they explore (represent- ing exploration speed, Q.2), and (2) the number of distinct concepts they explore (Q.3).
Tasks and models. We select two tasks of sim- ilar difficulty for user study: Stance detec- tion for feminism, and stance detection for climate change. They are accessible participants from different backgrounds. We had participants test the performance zero-shot ChatGPT (OpenAI, 2022) for both tasks, as we observed that it easily outperformed any available fine-tuned models on Huggingface- the latter failed at simple test cases (full prompts in Appendix A).
Specifically, for distinctiveness, we want to distinguish the local vs. global exploration patterns (cf. Figure 2), which requires us locate clusters of similar concepts, or concepts that only differ in granularity. Quantitative, this is reflected through inter-relevance between concepts, c.g., rising sea level should be considered close to sea surface temperature increase but distinct from waste management. To find a set of distinct concept clusters, we again measure the concept distance using SentenceBERT, and run Hierarchical Clus- tering (Ward Jr, 1963) on all available concepts collectively selected or created by our 20 user study participants, which, as argued $3, forms a representative set of what end users may care about for a given task. Note that we do not use all concepts from our KB for clustering as it would influence the ground truth. Hierarchical clustering allows to choose concept clusters that have sim- ilar granularities using a single distance threshold. Empirically, we use the threshold of 0.7, which pro- duces reasonably distinct clusters for both tasks (41 and 46 clusters for feminism and climate change respectively with an average size of 6.1 concepts).
Procedure. We recruited 20 participants (gradu- ate students with varying ML/NLP experience, de- tails in Appendix D.1) for 90-minute experiment session. We started by walking through the study instructions and asked them try WEAVER in an interactive tutorial session. Then participants tested the two aforementioned stance detection models for 30 minutes each, one in the treatment condition and one in control condition. To mitigate learning effects, we use Latin square design (Box, 2009) with four groups, counterbalancing (1) which con- dition participant encounters first, and (2) which model is tested first. Within each session, they were first asked to perform model testing for 25 min- utes, and then identify (select or create) concepts worth future testing for 5 minutes. The first phase

---

control
treatment
control treatment
loss of polar bear habitat
education
climate change
loss of arctic seu ice
feminism
* efeects on see animals
20
identified concepts
hit clusters
climate change
effects on animals
humans
Figure 4: Participants with WEAVER identified more concepts and hit more clusters.
suspended particulate metter
air pollution
ozone layer carbon monoxides
forcs
feminism
climate change
# hit clusters
control treatment
fossil fuel
20
industrialization
mining
10-
10-
Figure 6: We project the SentenceBERT embeddings of concepts explored by two participants (P15 and P16) into a 2D space, using t-SNE (van der Maaten and Hin- ton, 2008). Without WEAVER, participants explore less space, they performed more local exploitation than glohal exploration.
# identified concepts
identified concepts
Figure 5: Participants without WEAVER converged to hitting previously identified concept clusters as they identified more concepts. The gap between the two groups widened over the process.
WEAVER helps users more concept clus- ters (Q.3). More interestingly, observe that with WEAVER, participants not only found more concepts but also covered 47.7% (p < 0.01) more clusters in the space, i.c., they also explored more diverse concepts (Figure 4). This aligns with the survey responses, where 80% of participants agree that WEAVER helps them test the model more holis- tically and 76% of participants agree that WEAVER helps them find more diverse model bugs. We con- jecture that this because users with WEAVER explore more concepts not only in quantity (Q.2) but also in diversity, which is confirmed by many participants, e.g., "... (KB) encouraged me to ex- plore more areas of the domain than would have otherwise considered" (P10).
As such, distinctiveness is represented by the number of hit cluster in each user's exploration.
We analyze both measurements with repeated measures ANOVA analysis, which highlights the significance of our test condition (whether partici- pants use WEAVER) while considering the poten- tial impact from other independent variables. In our analysis, we test to what degree our tool, the task, and the order (tool first tool last) explain vari- ance in the outcome variables. Detailed analysis results can be found in Appendix D.3.
4.2 Results
WEAVER helps users identify more concepts (Q.2. We first observe that with WEAVER, partic- ipants identified 57.6% (p < 0.01) more concepts (Figure 4). This is likely because users can more easily explore a wider range of concepts with the external knowledge base, as confirmed by partici- pant P6: "... (KB) gives ideas beyond what I had in mind so I explored wider base of concepts."
Looking at their exploration trajectory (Figure 5), see evidence indicating that WEAVER enables users to continuously discover distinct concepts. In contrast, participants the control condition converged hitting previously identified concept clusters as they identified more concepts. We ob- served that without WEAVER, participants tended to refine existing concepts later in the exploration, as exploring new distinct areas becomes increas- ingly difficult.
We also observe that mug-finding is relatively independent of concept exploration. On average, participants found around 11 failing test cases (0.44 per minute), regardless of whether they used WEAVER or not. Since testing is orthogonal, partic- ipants who explore more concepts will find more failing test cases. We expect that if participants test longer, those with WEAVER will find more bugs while others will run out of concepts to test.
These contrasting trajectories eventually lead to different exploration results. As reflected in Fig- ure 6, the participant without WEAVER performed noticeably more local exploitation, finding highly related concepts in local arcas, whereas WEAVER helped the other participant explore more diverse

---

violence, harassment, racism...
integrating WEAVER into their natural evaluation environment (Jupyter Notebook), joining each user to explore their models for approximately three hours, and offering feedback and discussions when- ever necessary (e.g., as practitioners brainstormed their seed concepts).
# users hit the cluster
mode
control
treatment
5 0
5
cluster
Case selection. We approached researchers our contact network to identify projects that were actively developing and improving models to be in- tegrated into software products, such that (1) model testing has real stakes for them, and (2) the model needs to meet real- world requirements of a product, not just succeed
STEM, admission, workplace...
|Violencel: Verbal abuse of women not bad physical violence.
none against |Sexual Harassment]: Sexual harassment in the workpllace against should not tolerated. Sexual harassment against lavor against
not
big deal. MMMToo
[SEMM) Women are underreoresented in STEM clue to of peae
access education and resources.
standard benchmark.
AAdmissonn: It's not important to ensure that admissions proceszes free gender and discrimination.
We ended up recruiting practitioners working on two projects that matched our scope to try WEAVER. The first practitioner (CI) is building a pipeline for knowledge transfer. where they prompt an LLM to summarize content from transcripts into instructions. The second practitioner (C2) is build- ing IDE plugin help developers understand unfamiliar code, developing LLM prompts gen- crate text summary for code segments. While these are two distinct scenarios, their shared challenge is that for practitioners working on novel tasks, it is often non-trivial to perform prompt engineering, especially because they do not have appropriate datasets for evaluating their prompts.
favor against
Figure 7: Visualizing the identified concepts for femi- nism by their corresponding clusters, we see participants in different conditions had different focuses in model testing. This shows that WEAVER suggests concepts complementary human intuitions. In each test, the red trike-throughh labels are the wrong model predictions and the green ones are user-specified ground-truths.
concepts globally, without losing the ability to dive deeper into few local areas.
WEAVER shifts users towards different con- cept distríbution (Q.4). We also observe that participants collectively explored different con- cepts with WEAVER, as shown in Figure 7. Some concepts (c.g., violence) are much more explored by participants with WEAVER. This suggests that WEAVER helps mitigate participants' own biases and explore new concepts, supported by par- ticipant P13: "... (KB) gives some inspiration in concepts would not have otherwise thought of..."
The case studies were IRB-approved and partici- pants were compensated for their time.
WEAVER supports quick and effective iterations. Both C1 and C2 started with seed concept and then refined the seed concept least once based their observations. For example, C2 first tried seed concepts "challenges for summarizing a code script" and "reasons why people look for code sum- mary", finding the recommended concepts generic and not their major concerns. They self-reffee reflerefle through the process and identified the key scenario their plugin wants to support: (1) it targets novice programmers and (2) the most important applica- tion domain is data visualization. After this, they tried the seed concept "specific challenges that novice programmers might have in comprehending data visualization code" and found recommended concepts much more helpful.
That said, WEAVER also brings in its own biases, c.g. participants with WEAVER rarely explored con- cepts like STEM compared to those without, pos- sibly because they were too heavily anchored by the suggested concepts. This indicates that humans and knowledge bases complementary future work can support humans to better exploit their prior knowledge while exploring diverse concepts.
5 Case Studies
Using two case studics, we demonstrate that WEAVER can help practitioners test their mod- els and find various bugs in real-world settings, and has the potential to provide support beyond host-hoc model testing. In the studies, we provided sufficient supports to the practitioners, including
WEAVER helps practitioners find new model bugs by augmenting their existing analyses. While we did not have participants rate each con- cept they explored, based on their think-aloud reflection, we note that they were able to find

---

many helpful concepts in a short amount of time. Even though practitioners have been working on their (LLM-backed) models for while, they both obtained new insights into their models. First, WEAVER helped them observe issues they did not consider before. For example, in the seven concepts C1 tested, they found that the resulting instructions are always chronological even when there are de- tours in the input and steps reordering is desired. Second, WEAVER also helped them turn their prior, often fuzzy knowledge problems or requirements into concrete testable concepts. For example, C1 turned their vague notion "useful summaries should not take transcripts literal" into concrete theories, including "behind the transcript, there is hidden thought process important for identifying key ac- tion steps." Third, they were able to confirm model deficiencies they already suspected through sys- tematic tests (c.g.. "transcript summaries are often too verbose")). Similarly, C2 tested seven concepts and found "different parameters for customization" " and "when to use different data visualization APIs" particularly novel and insightful.
6 Related Work
Requirements elicitation. Requirements engi- neering has been extensively studied (Van Lam- sweerde, 2009). Despite many calls for the impor- tance of requirements in ML (e.g., Rahimi et al., 2019; Vogelsang and Borg, 2019), requirements in ML projects are often poorly understood and documented (Nahar et al., 2022), which means that testers can rarely rely on existing requirements to guide their testing. Requirements elicitation is usu- ally a manual and laborious process (c.g., inter- views, focus groups, document analysis, prototyp- ing), but the community has long been interested in automating parts of the process (Meth et al., 2013), e.g.. by automatically extracting domain concepts from unstructured text (Shen and Breaux, 2022; Barzamini et al., 2022a). We rely on the insight that LLMs contain knowledge for many domains that can be extracted as KBs (Wang et al., 2020; Cohen et al., 2023), and apply this idea to require- ments elicitation.
Model evaluation, testing, and auditing. Re- cent work ML model evaluation (c.g., Ribeiro et al., 2020; Goel et al., 2021; Röttger et al., 2021; Yang et al., 2022) has pivoted from i.idd tradi- tional accuracy evaluation to nuanced evaluation of model behaviors. As surveyed in our prior work (Yang et al., 2023), this line of research uses vari- ous test creation techniques, including slicing, per- turbations, and template-based generation. While providing many useful tools, these approaches of- ten assume an existing list of requirements and rarely engage with the question of what test. Ex- iting research relied mostly on the knowledge of particular researchers, resulting in incomplete and biased requirements. For example, Ribeiro et al. (2020) explicitly state that their list of requirements in CheckList is not exhaustive and should be aug- mented by users with additional ones that are task- specific. Through LLM-assisted requirements clic- itation, WEAVER helps users identify what to test systematically.
Notably, while C1 used AdaTTes for testing mod- els on different concepts, C2 reused test cases from their existing datasets, showing WFAVER'S flexibil- ity with different test case creation techniques. That C2 still discovered new insights within their own dataset demonstrates WEAVER's ' capability for en- couraging nuanced testing following requirements.
WEAVER is useful beyond testing models after- the-fact. While we mostly position WEAVER as a model testing tool, we find that its support for requirements elicitation supports the entire model development cycle (cf. the V-model, Fig. 1).
Although practitioners sometimes found it ini- tially challenging to define seed concepts, they found the process itself valuable. For example, C2 eventually settled on "specific challenges that novice programmers might have in comprehending [domain] code": ; they self-reflected how finding a good seed nudged them to state their goal explic- itly for the first time. For them, this reflection hap- pened too late to radically redesign their product, but it shows that WRAVER has the potential to sup- port carly-stage requirements engineering both for products and models. Meanwhile, C1 was inspired by concepts identified with WEAVER on model improvement. They experimented with different changes to prompts, encoding context for concepts they found challenging (e.g., step ordering).
Various alternative methods have been proposed for identifying what to test. For example, error analysis (c.g., Naik al., 2018; Wu et al., 2019a) and slice discovery (Eyuboglu et al., 2022) can help identify issues in existing datasets, but datasets are often incomplete and biased (Rogers, 2021), and can even be missing for emerging LLM applications where no dataset has been

---

pre-collected. Dataset-agnostic approaches like adaptive testing (Ribeiro and Lundberg, 2022; Gao et al., 2022) help users iteratively ideate concepts abstracted from generated test cases, but, as we confirmed, users tend to explore only local areas. These approaches engage in bottom-up style elicitation, which is reactive and may fare poorly with distribution shift. In contrast. WEAVER engages in top-down style elicitation, a more proactive process grounded in an understanding of the problem and domain.
them find pre-existing collections on important concepts. As such, enabling each individual to iden- tify what-to-tes/ is essential. We hope WEAVER can be used for democratizing rigorous testing, just as LLMs democratized access to powerful models. Still, currently WEAVER relies purely on practition- ers to identify requirements worth testing, which may result in mis-matched requirement granularity (cf. $3). Future work can explore more complex structures that can represent knowledge (e.g.. from KBs to knowledge graphs), and advanced recom- mendation mechanisms for practitioners to find the best requirements to explore first.
Furthermore, algorithmic auditing (Metaxa et al., 2021) elicits concerns from people with different backgrounds, usually on fairness issues, to avoid being limited by the ideas of single tester. However, it can be challenging to recruit, incentivize, and scaffold the auditors (Deng et al., 2023). In way, WEAVER might complement such work by providing diverse requirements for individual testers or crowd auditors.
Rethinking requirements for ML development. Though position WEAVER to ground model testing in requirements, we expect it be use- ful also in other development stages (cf. $5). For example, we expect that it can help developers think about high-level goals and success mea- sures for their products (Rahimi et al., 2019; Passi and Barocas, 2019), to guide development early on. For example, building on the observation that requirement-basee testing may help practitioners perform prompt engineering, we envision that future practitioners can use WEAVER for rapid prototyping, where they identify unique require- ments, pair them with corresponding test cases, and achieve better overall performance either through ensembled prompts (Pitis et al.. 2023) or prompt pipelines (Wu et al., 2022). Moreover, elicited model requirements themselves scriptions and documentation, which foster collaboration and coordination in interdisciplinary teamwork (Nahar al., 2022; Subramonyam et al., 2022). Notably, we believe WEAVER can support such iterations because it is built to be lightweight. In prior research, requirements engineering has sometimes been criticized to be too slow and bu- reaucratic, making developers less willing to ded- icate time to this step. In contrast, WEAVER al- lows developers easily adjust their exploration directions (through seeds and interactions), which makes it feasible be integrated into more agile and iterative development of ML products where requirements are evolving quickly.
7 Discussion and Conclusion
In this work, propose WEAVER, tool that uses knowledge bases to guide model testing, help- ing testers consider requirements broadly. Thor- ough user studies and case studies, we show that WEAVER supports users to identify more, as well as more diverse concepts worth testing, can suc- cessfully mitigating users' biases, and can support real-world applications. Beyond being a useful test- ing tool, the underlying concept of WEAVER have interesting implications on ML model testing and development, which we detail below.
serve de-
Model testing in the era of LLMs. Through- out our user studies and case studies, we focused on testing "models" achieved by prompting LLMs. Here, we would like to highlight the importance of requirements in such cases. LLMs are increas- ingly deployed in different applications. and tradi- tional model evaluations are becoming less indica- tive. With these models trained on massive web text, it is unclear what should be considered as "in-distribution evaluation data." Instead, the eval- uation objectives heavily depend on what practi- tioners need, which should be reflected through well-documented requirements.
Limitations
Meanwhile, as most practitioners are not NLP experts, they face challenges articulating how and what they should test about their prompted mod- cls (Zamfirescu- Caneffrescu-Periiraa et al., 2023). As their use cases become more nuanced, it is also less likely for
Availability of domain knowledge in LLMs. LLMs encode a vast amount of knowledge, but may not include very domain-specific knowledge for specialized tasks, very new tasks, or tasks where

---

relevant information is confidential. Our technical implementation fundamentally relies on extracting knowledge from LLMs and will provide subpar guidance the model has not captured relevant domain knowledge. Conceptually our approach to guide testing with domain knowledge would also work with other sources of the knowledge base, whether manually created, extracted from text corpus (Shen and Breaux, 2022; Barzamini et al., 2022a), or crowdsourced (Metaxa et al., 2021).
align with individual subjective judgment. Second, we conducted case studies in real-world settings with practitioners (high external validity) but can naturally not control the setting or conduct repeated independent observations (limited internal validity). With only two case studies, generaliza- tions must be made with care.
This tradeoff of external and internal validity is well understood (Siegmund et al., 2015). Con- ducting both forms of studies allows us to perform some limited form of triangulation, increasing con- fidence as we see similar positive results regarding WEAVER's 's usefulness for discovering diverse con- cepts.
Impacts from biases in LLMs. WEAVER uses LLMs to build knowledge bases such that users can elicit diverse requirements. However, LLMs themselves are found to be biased, sometimes un- truthful, and can cause harm (Nadeem et al., 2021; Kumar et al., 2023). Therefore, users should care- fully interpret results from WEAVER in high-stake applications.
Subjectivity in human judgments. All model testing requires judgment whether model's pre- diction for a test example is correct. We noticed that user study participants and sometimes also case study practitioners struggled with determining whether model output for a specific test example was a problem, and multiple raters may sometimes disagree. For our purposes, we assume it is the tester's responsibility of identifying which model outputs they consider problematic, and we do not question any provided labels. This, however, re- minds us that like data annotation (Santy et al.. 2023), any model testing process will likely bring in testers' biases, as they get to decide what is right and what is wrong. In practice, broader discussion among multiple stakeholders may be required to identify what model behavior is actually expected and decomposition of model testing using require- ments might be helpful foster such engagement.
Threats to validity in human-subject evalua- tions. Every study design has tradeoffs and lim- itations. In our evaluation, we intentionally com- bined multiple different kinds of user studies to triangulate results.
First, we conducted a user study as a controlled experiment. While the results are very specific and created in somewhat artificial settings and must be generalized with care (limited external validity). the study design can enact high level of control to ensure high confidence in the reliability of the find- ings in the given context with statistical techniques (high internal validity). For example, regarding external validity, results may not generalize casily to other tasks that require different amounts of domain understanding or are differently supported by the chosen test case creation technique, and our participant population drawn from graduate students with technical background may not equally generalize to all ML practitioners. There are also some threats to internal validity that remain, for example, despite careful control for ordering and learning effects with a Latin square design and assuring that the four groups were balanced in experience ('years of ML experience' and 'NLP expertise" asked in the recruitment sur- vey before assignment), we cannot control for all possible confounding factors such as prior domain knowledge, gender, and motivation. In addition, we rely on clustering and similarity measures among concepts for our dependent variables, which build on well-established concepts but may not always
Ethies Statement
Research Reproducibility. While our experi- ments are mostly conducted on the closed API (lext-davinci-003) provided by OpenAI, none of the conceptual contributions of our paper relate to specific models or APIs. The concrete evalua- tion results depend on how humans interact with specific models, but the approach can be used with other models. Indeed, our extra experiments with 1lama-22 1-ama-2-13bbl 2-b3b-chat on the climate change task show that the generated concepts from open-source models achieve substantial levels of recall (83% vs. 91 % originally). This supports that the idea be- hind WEAVER is reproducible. While users may see somewhat different KBs through different runs of the same/different LLMs, they get similar chances of seeing useful concepts, receive a similar level of

---

support on requirement elicitation (our core con- tribution), and will be able to yield similar model testing effectiveness.
In Proceedings of the 2023 CHI Conference on Hu- man Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machin- ery.
Human-subject Experiments. Our studies had been approved by our IRB before it was conducted, as is standard practice for human-subjer experi- ments. We recruited all participants through emails, and all of them are graduate students with varying ML/NLP experience (see details in Appendix D. The participants were compensated for their time ($20 per hour). As part of testing, they may write review text that is abusive, dangerous, hateful, offensive-they were made aware of this fact and could end participation at any time.
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge- base of language models.
Wesley Hanwen Deng, Boyuan Guo, Alicia Devrio, Hong Shen, Motahhare Eslami, and Kenneth Hol- stein. 2023. Understanding practices, challenges, and opportunities for user-engaged algorithm audit- ing in industry practice. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abi- naya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al. 2021. Nl-augmenter: A frame- work for task-sensitive natural language augmenta- tion. arXiv preprint a2x02:2112.02221.
A Acknowledgements
This work was supported in part by the National Science Foundation (#2131477) and gift funds from Adobe, Oracle, and Google. Work by Brower- Sinning and Lewis was funded and supported by the Department of Defense under Contract No. 15-70-1----0002 with Carnegie Mellon Univer- sity for the operation of the Software Engineering Institute, federally funded research and develop- ment center (DM23-2019).
Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Re 2022. Domino: Discovering systematic errors with cross-modal embeddings. In International Confer- ence on Learning Representations.
Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. 2022. Adaptive testing of com- puter vision models.
References
Kumaripaba Athukorala, Alan Medlar, Anti Oulasvirta, Giulio Jacucei, and Dorota Glowacka. 2016. Be- yond relevance: Adapting explorationlexploitationn in information retrieval. In Proceedings of the 21 International Conference on Intelligent User Inter- faces, IUI 16. page 559-369,10 New York, NY, USA. Association for Computing Machinery.
Robert Geirhos. Henrik Jacobsen, Claudio Michaelis. Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 166656673
Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. 2021. Robustness gym: Unifying the NLP evaluation land- scape. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies: Demonstrations, pages 42-55, Online. As- sociation for Computational Linguistics.
Hamed Barzamini, Mona Rahimi, Murteza Shahzad, and Hamed Alhoori. 2022a. Improving generaliz- ability of ml-enabled software through domain spec- ification. In Proceedings of the 1st International Conference on Al Engineering: Software Engineer- ing for AI, CAIN '22, page 181-192, New York, NY, USA. Association for Computing Machinery.
Irtiza Hasan, Shengeai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao. 2021. Generalizable pedes- trian detection: The elephant in the room. In 2021 IEEE/CVF Conference on Computer Vision and Par- tern Recognition (CVPR), pages
Hamed Barzamini, Murtuza Shahzad, Hamed Alhoori, and Mona Rahimi. 2022b. A multi-level semantic web for to-d-cc-- domain concept, pedestrian in ml-hased software. Requir. Eng., 1612)26
George E. P. Box. 2009. Statistics for experimenters: design, innovation, and discovery. Wiley- -iakweB
Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Im- proving fairness machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference Human Factors in Com- puting Systems, CHI '19, page 1-16, New York, NY, USA. Association for Computing Machinery.
Ángel Alexander Cabrera, Erica Fu, Donald Bertucci, Kenneth Holstein, Ameet Talwalkar, Jason I. Hong, and Adam Perer 2023. Zeno: An interactive frame- work for behavioral evaluation of machine learning.

---

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh. Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes chal- lenge: Detecting hate speech multimodal memes. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA. Curran Associates Inc.
119-126. Online. Association for Computational Lin- guistics.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 5356-5371, Online, Association for Computational Linguistics.
Sachin Kumar, Vidhisha Balachandran, Lucille Njoo. Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. Language generation models can cause harm: So what can we do about it? an actionable survey. In
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 3299-3321, Dubrovník, Croatia, As- sociation for Computational Linguistics.
Nadia Nahar, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian Kästner. 2023. A meta-summary of challenges building products with ml components collecting experiences from 4758+ practitioners. In Proceedings of the International Conference on Al Engineering Software Engineering for Al (CAIN).
Michelle S Lam, Zixian Ma. Anne Li, Izequiel Freitas, Dakuo Wang. James A Landay, and Michael Bern- stein. 2023. Model sketching: Centering concepts in carly-stage machine learning model design. arXiv preprint arXiv:2303.02884
Nadia Nahar, Shurui Zhou, Grace Lewis, and Christian Kästner. 2022. Collaboration challenges in build- ing ml-enabled systems: Communication, documen- tation, engineering, and process. In Proceedings of the 44th International Conference on Software En- gineering (ICSE), pages 413-425, New York, NY. ACM Press. ** Distinguished Paper Award**.
Jenny T Liang, Chenyang Yang, and Brad A Myers. 2023. Understanding the usability of ai programming assistants. arXiv preprint arXiv:2303.17225
Aakanksha Naik, Abhilasha Ravichander. Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Com- putational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingnistics, pages 1906-1915 On- line. Association for Computational Linguistics.
R Thomas MeCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntac- tic heuristics in natural language inference. arXiv prepriat urXiv:1902.01007
OpenAL. 2022. Introducing chatgpt.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Danaë Metaxa, Joon Sung Park, Ronald E. Robert- son, Karrie Karahalios, Christo Wilson, Jeff Han- cock, and Christian Sandvig. 2021. Auditing algo- rithms: Understanding algorithmic systems from the outside in. Found. Trends Hlu.--Comput, Interact., 14(4):272 344.
Samir Passi and Solon Barocas. 2019. Problem formu- lation and fairness. In Proceedingy of the Conference Fairness, Accountabiliy, and Transparency, FAT '19, page 39-48, New York, NY, USA. Association for Computing Machinery.
Hendrik Meth, Manuel Brhel, and Alexander Maedche. 2013. The state of the art in automated requirements elicitation. Information and Software Technology. 55(10):1695-1709
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEvall-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) pages 31-
Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.05977
41, San Diego, California. Association for Computa- tional Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: frame- work for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages
Mona Rahimi, Jin LC Guo, Sahar Kokaly, and Marsha Chechik. 2019. Toward requirements specification for machine-learned components. In IEEE 27th In- ternational Requirements Engineering Conference Workshops, pages 241-244. IEBE

---

Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King. and Saleema Amershi. 2023. Supporting human- ai collaboration in auditing Ilms with llms, arXiv preprint arXiv:2304.09990
pages 5248-5264 Online, Association for Computa- tional Linguistics.
Yuchen Shen and Travis Breaux. 2022. Domain model extraction from user-authored scenarios and word em- beddings. In 2022 IEEE 30th International Require- ments Engineering Conference Workshops (REW), pages 143-151.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP) pages 3982-3992, Hong Kong, China. Association for Com- putational I Linguisties.
Ben Shneiderman. 1996. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings 1996 IEEE symposium on visual lan- guages, pages 336-343. IEEE
Janet Siegmund, Norbert Siegmund, and Sven Apel. 2015. Views internal and external validity in em- pirical software engineering. In Proc. IEEE/ACM 37th International Conference on Software Engineer- ing, volume 1, pages 99, IEEE
Marco Tulio Ribeiro. 2023. Testing language models (and prompts) like we test software. Medium.
Marco Tulio Ribeiro and Scott Lundberg. 2022, Adap- tive testing and debugging of NLP models. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Pupers), pages 3253-3267, Dublin, Ireland. Associa- tion for Computational Linguistics.
Ian Sommerville. 2015. Software Engineering, 10th edition. Pearson.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, A77''17, page 4444-4451. AAAI Press.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behav- ioral testing of NLP models with CheckList. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
Hariharan Subramonyam, Jane Im, Colleen Seifert, and Eytan Adar. 2022. Solving separation-of of-conerns problems in collaborative design of human-ai sys- tems through leaky abstractions. In Proceedings of the 2022 CHI Conference on Human Factors in Com- puting Systems.
Anna Rogers. 2021. Changing the world by changing the data. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 2182-2194, Online. Association for Computa- tional Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of Machine Learning Research, 976))2557-2
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional tests for hate speech detection models. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 41-58, Online. Association for Computational Linguistics.
Axel Van Lamsweerde. 2009. Requirements engineer- ing: From system goals to UML models to software, volume 10. Chichester, UK: John Wiley & Sons.
Andreas Vogelsang and Markus Borg. 2019. Require- ments engineering for machine learning: Perspec- tives from data scientists. In IEEE 27th International Requirements Engineering Conference Workshops. pages 245 251. IEEE
Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs. arXiv preprint arXiv:2010. /
Sebastin Santy, Jenny T. Liang, Ronan Le Bras, Katha- rina Reinecke, and Maarten Sap. 2023. Niposition- ality: Characterizing design biases of datasets and models. In ACL.
Joe H Ward Jr. 1963. Hicrarchical grouping to opti- mize an objective function. Journal of the American statistical association, 58(301):236224:
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias hate speech detection. Proceedings of the 57th Annual Meeting of the Association for forrCoon Computational Linguistics, pages 1668-16785 Florence, Italy. Asso- ciation for Computational Linguistics.
Nicholas C. Worrmadd 1995. Differential Equations for Random Processes and Random Graphs. The Annals of Applied Probability, 5(4): 1217 1235.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019a. Errudite: Scalable, repro- ducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747-763, Florence, Italy. Association for Computational Linguistics.
Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview, In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics,

---

Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2019b. Errudite: Scalable, repro- ducible, and testable enoor analysis. In Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pages 747-763.
LOCATEDNEAR: List (N) things that often locates near {concept).
CAPABLEOF: List (N) things that (concept) is
capable of.
HASSUBEVENT: List (N) subevents {concept).
HASPREREQUISITE: List (N} things that happen before {concept).
Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference Human Factors in Computing Sys- tems, pages 1-22.
DESIRES: List (N) things that {concept) desires. CREATEDBY: List (N) creators of {concept). SYMBOLOF; List (N) symbols of {concept).
CAUSESOESIRE: List (N) desires caused by (
concept),
MADEOF: List (N) materials of [concept).
RECEIVESACTION: List (N) actions that can be done to {concept).
Chenyang Yang, Rachel A Brower-Sinning, Grace Lewis, Christian Kässtnr, and Tongshuang Wu. 2023. Capabilities for better ml engineering. Proceed- ings of the AAAT-23 Workshop on Artificial Intelli- gence Safety (SafeAl).
DESIREDBY: List (N) entities or people that desire {concept).
CREATES: List (N) things that {concept) creates.
CAUSEDBY: List {N) things that cause {concept). DONEBY: List (N) entities or people that can do [concept).
Guanqun Yang, Mirazul Haque, Qiaochu Song, Wei Yang, and Xueqing Liu. 2022. TestAug: frame- work for augmenting capability-basee NLP tests. In Proceedings of the 29th International Conference on Computational Linguistics, pages -480-3495, Gyeongju, Republic of Korea. International Com- mittee Computational Linguistics.
DESIRECAUSEDBY: List (N) things that cause desire of {concept).
DONETO: List (N) entities people that { concept) can be done to.
RELATEDTO: List (N) concepts related {concept }.
## Example contexts prompts
J.D. Samfirescu-Pereing Richmond Wong, Bjoern Hartmann, and Qian Yang. 2023, Why johnny can't prompt: How non-ai experts try (and fail) design IIm prompts. Proceedings of the 2023 CHI Confer- ence on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
TYPEOF: {concept) type of (parent_concept) PARTOF:
{concept)
of [parent_concept]
HASPROPERTY: (parent_concept) is described concept).
USEDFOR: pparent_concept) used for (concept).
ATLOCATION: gparent_concept, locates {concept
CAUSES: searent_concenn causes (concept),
MOTIVATEDBY: (parent_concept) is motivated by {
Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele. 2017. Citypersons: A diverse dataset for pedestrian detection. In 2017 IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pages 4457- 4465.
concept),
OBSTRUCTEDBY: (parent_concepl is obstructed by [concept).
MANNEROF: (concept] is way {
varent_concepti)
LOCATEDNEAR: {concept) locates near {
saarent_concept):
A Complete List of Used Prompts
CAPABLEOF: {parent_concepp) is capable of { concept).
# Prompts for expanding knowledge bases
HASSUBEVENT: (concept) happens during { parrent_concept).
[cantext)
HASPREREQUISITE: {concept) happens before (tarent_concept)
(list_prompt) Pay attention the context above. Summarize JSON list.
DESIRES: (parent_conceeee concept] desires (concept).
CREATEDBY: {concept) creates (parent_concept).
"'json
SYMBOLOF:
{concept) is symbol of parent_connnnc concept).
CAUSESDESIRE: (parent_concept) causes desire of [concept).
Example list prompts
TYPEOF: List (N) types [concept).
MADEOF:
is made of (concept). RECEIVESACTION: (parent_concept) receives action of (concept).
PARTOF: List (N) parts or aspects [concept).
HASPROPERTY: List (N) descriptions of [concept). USEDFOR:
List (N) things (concept) could be used
DESIREDBY: (concept) desires (parent_concept).
for.
CREATES:
creates (concept).
ATLOCATION: List (N) locations (concept) could appear in.
CAUSEDBY: (concept) causes ).anept_conccpt).
DONEBY: {concept) does (parent_concept).
CAUSES:
List (N) consequences of (concept).
DESIRECAUSEDBY: (concept) causes desire of parent_concept).
MOTIVATEDBY: List (N) motivations behind { concept).
DONETO: {parent_concep) is done to (concept).
OBSTRUCTEDBY: List (N) things, entities, or people against (cancept).
RELATEDTO: [concept) related { parent_concept).
MANNEROF: List (N) ways to do (concept).

---

Stance detection. Stance detection requires clas- sifying texts as either being in favor of, against, neutral toward the given target (Mohammad et al., 2016). The task is crucial for understanding the public's perception of given targets We selected two targets for our evaluation: feminism and cli- mate change, which are previously explored for Tweets (Mohammad et al., 2016).
Prompts for the tested models in the user study
Stance detection on feminism
Carefully classify a sentence's stance on feminism. The labels are \"favor\", \o
against\" or \"none\.. Only reply with the label.
Sentence:
(example)
# Stance detection on climate change Carefully classify sentence's 's stance on combating climate change. The labels are V" favor\", \"against\" or \"none\". Only reply with the label. Sentence: (example)
D User Study
D.1 Study Design Details
Participants. We recruited 20 participants (grad- uate students with varying ML/NLP experience). Among them, 70% have worked MI for more than three years; 80% rate themselves as least somewhat familiar with NLP; 60% are working on a project using NLP at the time of the study. We randomly assigned participants to the four experi- mental groups. The four groups are comparable in terms of ML/NLP experience and familiarity. The participants were compensated $20 per hour.
The path contexts in our full prompt are used to mitigate the issue of polysemy and deviation when users explore deeper the graph.
SLRVEY QUESTIONS
Rate whether you agree with the statement: Concept knowledge graph helps find more diverse model
bugs. Write justification for your rating.
Rate whether you agree with the statement: Concept knowledge graph helps me find more important model bugs. Write a justification for your rating.
D.2 Post-study Survey
Rate whether you agree with the statement: Concept knowledge graph helps me test the model more holisti- cally. Write a justification for your rating.
We share our survey questions (Figure 8) and users' responses (Table 2).
Rate whether you agree with the statement: want the model
tool with knowledge graph build/use the future.. Write a justification for your rating.
D.3 Quantitative Analysis
We show the ANOVA analysis results in Table 3.
you want use the the future, what the model and task you want use for?
Feedback on how improve tool in future.
E Additional Data on Running WEAVER
It takes 30 seconds average to generate default KB from scratch (with around 500 concepts) on machine (Precision 3650 workstation, with In- tel(R) Xeon(R) W-1350 CPU and 32GB memory). When users explore the KB interactively and ex- pand a node, the query takes seconds on average. The wait can be greatly reduced via pre-fetching (i.e., expanding the node on display in the back- ground before the actual query), which has been implemented in WEAVER.
Figure 8: Post-study survey questions.
B User Interface
Figure 9, we show the complete user interface.
C Task Descriptions
Hateful meme detection. Hateful meme detec- tion (Kiela et al., 2020) requires classifying an image (meme with text) as hateful or non-hateful. This task is challenging in that it requires multi- modal reasoning in order to classify the original meme and its confounders correctly
F WEAVER with Open-source LLMs
We conducted an extra experiment to evaluate whether open-source LLMs can also generate com- prehensive KBs. We generated another KB on the climate change task, using 1lama-2-13b-ch with 4-bit quantization. We found that the gener- ated KB achieved comparable levels of recall (83% 91 % with text-davinci-003)
Pedestrian detection. Pedestrian detec- tion (Zhang et al., 2017) requires detecting and localizing pedestrians in images. Though it one of the longest-standing problems in computer vision, people still observe generalizability issucs in existing detectors (Hasan et al., 2021).

---

online toxicity ①
Example Panel
has subtype
hate speech 2
motivated by
racism
Concept: misogyny
done people with different political balicfs
Path: Misogyny is type of online toxicity.
suptype
religious discrimination
Show subconcepts for "hate speach"
Suggestions
Add Examples
via cyber-bullinng
Ask for suggestions..
has subtype
misogyny
done via
spreading misinformation
Input
Predicted
Label
Pass Fail
disabled people
against
against
done by
celebrity
to workplaces.
located online gaming
Show subconcepts for "online toxicily"
4
Figure 9: Complete user interface for WEAVER. On the right is re-implemented version of AdaTest to assist users to create test cases.
Statement
Distribution
Q1
Knowledge hase helps me find more diverse model
75%
10%
bugs,
Q2
Knowledge hase helps me find more important model bugs.
45%
15%
Q3 Knowledge base helps test the model more holis- tically.
80%
0%
Q4
want to use WEAVER to test the model build/use
95%
5%
the future.
Strongly agree
Agrec
Neutral
Disagree
Strongly disagree
Table 2: Participants' responses in the post-study survey.
Number of found concepts
ges
Interv.: Used WEAVER?
0.307
Order: Tool in first task?
0.008
Task number
0.000
Number of hit clusters
ges
Interv.: Used WEAVER?
0.408
Order: Tool in first task?
0.039
Task number
0.006
0.01
N 40
Table 3: User study ANOVA analysis results. We mea- sure effect size with ges (generalized Eta-Squared mea- of effect size).


Reviews:
[{"review_id": "UINn31ENBa", "reviewer": "Reviewer_rtMG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contribution to the model testing procedure!\n\n**Technical implementation details**\n\n1. Indeed, Weaver will need to rebuild the KB each time a new seed is queried, which is a fundamental strength supporting Weaver’s flexibility and generalizability. Generating a default KB from scratch (with ~500 concepts) takes ~30 seconds on our machine (Precision 3650 workstation, with Intel(R) Xeon(R) W-1350 CPU and 32GB memory). When users explore the KB interactively and expand a node, the query takes ~8 seconds. The wait can be greatly reduced via pre-fetching (i.e., expanding the node on display in the background before the actual query), which has been implemented in Weaver. In practice (in our case studies), users did not complain about the usability of Weaver.\n\n2. The ground-truth concepts in Section 3 are collected from previous work and user studies (L251-259), we shared the full list of the ground-truth concepts in the link here: https://figshare.com/s/481a69fa1b36dbd76088 \n\nWe plan to extend our write-up on the technical implementation upon acceptance, and open source both the Weaver KB builder and user interface.\n\n**Case study details**\n\nWe are glad that the reviewer found the case study insightful! We are happy to add more concrete evidence on practitioner usage. \n\n1. *Practical problem*: As described in Section 5, C1 is building a pipeline for knowledge transfer, where they prompt an LLM to summarize content from transcripts into instructions. C2 is building an IDE plugin to help developers understand unfamiliar code, developing LLM prompts to generate text summaries for code segments. While these are two distinct scenarios, their shared challenge is that for practitioners working on novel tasks, it is often non-trivial to perform prompt engineering, especially because they don’t have appropriate datasets for evaluating their prompts.\n\n2. *Concrete exploration procedure*: Both C1 and C2 started with a seed concept of their own wording, then refined the seed concept at least once based on their observations. For example, C2 first tried seed concepts “*challenges for summarizing a code script*” and “*reasons why people look for code summary*”, finding the recommended concepts generic and not their major concerns. They self-reflected through the process and identified the key scenario their plugin wants to support: 1) it targets novice programmers and 2) the most important application domain is data visualization. After this, they tried the seed concept “*specific challenges that novice programmers might have in comprehending data visualization code*” and found many helpful concepts.\n\n3. *Useful concepts*: While we did not have participants rate each concept they explored, based on their think-aloud reflection, we note that they were able to find many helpful concepts in a short amount of time. At a high level, concrete concepts that help practitioners explicate existing requirements or discover new requirements tend to be more helpful. For example, C1 tested 7 concepts and they (a) re-confirmed their concerns by finding multiple examples where the summaries are too verbose and (b) found several other issues by testing concepts “*right tone and voice for audience*” and “*understanding of thought process*”. C2 tested 7 concepts and found “*different parameters for customization*” and “*when to use different data visualization APIs*” particularly helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes that it is important to identify \"what to test\" in model testing. They introduced an interactive testing tool that uses large language models to generate concepts for model testing.\n\n2. The user study and case studies provide empirical evidence of the effectiveness of the tool to guide model testing. The results show that human users identified more, as well as more diverse concepts worth testing when using WEAVER.\n", "reasons_to_reject": "1. The paper is a bit difficult to follow. It lacks some details on the technical implementations. It would be beneficial for readers if the paper provided more descriptions, e.g., in Section 2.2, does each query requires building a new graph (involving SentenceBERT and GPT-2)? How long does this process take? Does the user need to wait for this each time the seed query is changed? In Section 3, line 262, what are the 144 ground-truth concepts?\n\n\n2. The case studies in Section 5 are insightful but too brief. Readers can benefit from more details, e.g., what are the practical problems, what are the seed queries practitioners use, what concepts are most helpful, etc?\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces an interactive tool, WEAVER, for guiding model testing. WEAVER uses large language models to generate concepts by prompting with pre-defined relations from ConceptNet. It applies a greedy algorithm to select the concepts that are diverse but still relevant to the seed queries. Then the generated concepts are expected to help developers or researchers to come up with better testing examples. The paper presents a user study on two tasks, stance detection for feminism and stance detection for climate change. The experiments show that both NLP experts and non-experts identified more, as well as more diverse concepts worth testing when using WEAVER. The paper also provides two case studies to demonstrate how WEAVER can help practitioners test models in real-world applications.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LnEQdCXhZH", "reviewer": "Reviewer_hoyj", "summary": "", "questions": "A: Besides recall, have you evaluated the concepts generated by weaver that are not present in the ground-truth concepts? I mean, some kind of evaluation taking into account precision. This is important because these additional concepts could mislead users", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Relation to traditional benchmarking and model testing approaches**\n\nThank you for acknowledging that the concepts Weaver suggests are important! Better contextualizing Weaver based on existing model analysis literature is indeed important, and we will add these discussions to the paper. \n\nSimilar to existing literature on SE-inspired NLP model testing, our work is *complementary to traditional benchmarking*, in that it offers more fine-grained information about concrete model behaviors beyond standard metrics. The testing schema is becoming more important as LLM+prompting is applied to various novel tasks where traditional benchmarking is nearly impossible due to the lack of datasets (as in our case studies).\n\nAmong the related work in model testing (CheckList, AdaTest), as we have reviewed in Section 6 and as the reviewer has noticed, “prior work has focused on how to write test cases, not what tests to write.” In contrast, the core contribution of Weaver is *on systematically revealing “what to test”*, which precisely makes it “a recommender system of *diverse but relevant* concepts” – this is the key novelty of our approach over the state of the art. Similarly, we note that the goal of Weaver is not “quick testing / error analysis,” but more towards supporting *thorough* testing of *all relevant requirements* for a task even though it might not make testing itself faster. We will clarify that Weaver’s testing power comes from combining Weaver’s LLM+ConceptNet-based KB with existing test generators like AdaTest.\n\n\n**Precision vs. recall in Weaver concept evaluation**\n\nThank you for the suggestion. We carefully considered how to evaluate the quality of presented concepts, but are limited by available ground truth data. As described in Section 3, we have two sources of *partial* ground truth: dataset analysis and user study in prior papers, and concepts identified in our user study. Both sources are incomplete. For example, the Hateful Meme Detection ground truth is collected from a user study, where users iterate over their models and find important concepts. As a result, these ground truths only reflect concepts that are in-distribution and are known to be issues, but do not contain other relevant concepts that would be important for practical model deployment. With this information, we can only compute recall based on known concepts, but cannot compute precision.\n\nWe did perform a follow-up, manual validity check to evaluate precision. Following prior work [1], we sampled 200 edges from our 4 KBs evaluated in Section 3, 50 each, and manually judged their correctness. We found 161 out of 200 edges valid, achieving a precision of 80.5\\%. We will add these justifications and results to the paper revision. \n\nAs a proxy for practical usefulness (i.e., practically acceptable precision), consider that participants in our studies could navigate the KG without being overwhelmed by useless proposed concepts. Indeed, we did not observe users deleting invalid concepts or making relevant complaints in the post-study survey.\n\n\n**References**\n\n[1] Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge base of language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method for creating the KB using LLMs and show it to the users.", "reasons_to_reject": "- Although the concepts suggested are important for a developer/researcher, I miss how this work is connected to traditional benchmarking of systems for improving testing or deeper analysis. The proposed system can be connected to a test generator but I see the proposed interface as a recommender system of some concepts instead of a tool for a quick testing of error analysis, or something similar, in new developments.\n", "questions_to_authors": "A: Besides recall, have you evaluated the concepts generated by weaver that are not present in the ground-truth concepts? I mean, some kind of evaluation taking into account precision. This is important because these additional concepts could mislead users", "paper_topic_and_main_contributions": "The authors propose a system for suggesting different concepts to be evaluated related to the same task. The concepts are suggested using KB and shown using a tree structure. The KB is automatically created using the knowledge contained in LLMs.\nThe main contributions are the system presented for suggesting additional tests. I think the main contribution is clear from the following excerpt in the paper \"However, prior work has focused on how to write test cases, not what tests to write.\"", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 370, \"To to\" -> \"To\""}, "extra_text": {}}, {"review_id": "3WiU1Ym7og", "reviewer": "Reviewer_Mdj9", "summary": "", "questions": "1. In software engineering, the requirement is specified before the developers even write the software, but the authors are testing existing models. Do authors believe the way of requirement engineering is different for traditional software and machine learning models?\n\n2. The Weaver tries to build and recommend fine-grained topics based on a knowledge graph (KG). However, there are a few issues in this process:\n   1. In Line 160, the generated topics are not checked for their validity.\n   2. The authors use the relations provided by ConecptNet as-is without considering its bias (for example, its coverage may be insufficient). Even though this choice seems reasonable, I expect the authors to discuss the bias of using existing relations in ConceptNet.\n   3. In Line 199, the authors propose to maximize diversity and relevance at the same time. However, it would be better to formulate this as a multi-objective optimization problem since we will likely land on a solution with very high diversity but low relevance.\n3. Is your user study single-blind or even double-blind? I do not see whether the users mentioned in Section 4 know which group (control vs. treatment) they are in, and I don't know if the authors could unconsciously influence the users.\n4. The design of Figure 5 is quite interesting. However, I do not understand the role of gray lines.\n5. Are the distances between concepts in Figure 6 drawn to scale? How do the authors place the clusters? These plotting details are important as the authors are trying to use this graphic as part of the evidence that their proposed approach (i.e., orange clusters) are better than the control group (i.e., blue clusters) as it generates clusters that are more spread out.\n6. Missing minor details\n   1. L327 - L330: What exact models do you use?\n   2. L371 - L386: What are the configurations of your hierarchical clustering?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the value of requirement elicitation for NLP model testing! We do plan to open source both the Weaver KB builder and user interface upon paper acceptance.\n\n**Contribution**\n\nWe would like to emphasize that our core contribution is the **concept and method of requirements engineering to improve testing**. It is purely out of convenience that we used AdaTest as our primary test generator, but Weaver and AdaTest make contributions on completely different parts of the larger problems, i.e., what to test vs. how to test. As mentioned in our case study, AdaTest is not necessarily the only test generator that can be used. Weaver’s interface can be paired with other test generation methods as well.\n\n\n**Dependence on closed-source API**\n\nWe acknowledge that we use a closed-source API (`text-davinci-003`) for building the KB in Weaver. To see whether we can use an open-source model alternative, we performed an extra experiment with `llama-2-13b-chat` on the climate change task (in addition to what has been conducted in the paper). We found that the generated concepts from open-source models achieve comparable levels of recall (82.7\\% vs. 91.4\\% originally). Again, we want to emphasize that none of the conceptual contributions of this paper relate to specific models or APIs. The model is important for creating the KB but multiple papers have now robustly shown that this is possible using different models (`text-davinci-002`, gpt2, BERT) [1, 2], so we expect this to replicate conceptually even when an exact reproduction of data may not be possible.\n\n> “As a side note, the authors claim to use ChatGPT (`gpt-turbo-3.5`) throughout the text, but they use `text-davinci-003`.”\n\nTo clarify, we did evaluate Weaver on ChatGPT (`gpt-turbo-3.5`) in our user study, while we built Weaver using `text-davinci-003`, the most powerful LLM API accessible at the time of our system development.\n\n\n**User study design**\n\n> User study completeness\n\nThe reviewer seems to misinterpret reports from a pilot study in L351 - L355 as an indication of an incomplete study. In fact, a pilot study is a common approach in human-subject studies to validate and iterate on study designs. All the results were based on observations in the final study.\n\nHere, we illustrate more on our rationale for *designing the study to be 25 min model testing + 5 min additional exploration, instead of making the study longer*, according to our pilot study observations: The primary goal of our user study is to see whether Weaver *better supports the concept exploration step in the context of model testing* – In other words, we are more interested in seeing what concepts would catch participants’ attention, and less interested in seeing them create test cases for every concept. In the *pilot study*, we noticed the latter (writing test cases) takes much longer time and effort – completely testing all the concepts users find interesting requires ~2 hours. To ensure that participants can focus on the primary goal, we chose to only have them test a subset of concepts to verify the error rate per concept and get grounded in what concepts are worth testing, such that they can spend a larger proportion of time collecting concepts to be tested later. \n\n> Single / double-blindness of our study\n\nThe discussion of single-blind vs. double-blind does not apply to our within-subject study design. All participants perform tasks in the control and the experimental conditions (the order and tasks are intentionally varied and counterbalanced between participants). It is obvious to participants that they perform the task in different conditions, though they are not informed which condition is the treatment condition. The experimenter also can identify the conditions. Within-subject designs are very common for controlled experiments and have tradeoffs with between-subject designs (where double-blind studies are possible) [3]. In our context, we picked within-subject for reducing noise while keeping the number of participants manageable.\n\n\n**Is requirements engineering different for traditional software and ML models?**\n\nThat’s a very thoughtful question! As argued in our paper, the *state of practice* between requirements engineering in traditional software and ML models is certainly very different. More importantly, we also expect there to be substantial differences *conceptually* due to a lack of a clear specification for ML models: In traditional software, requirements engineers create the specifications. In machine learning, we expect that requirements are discovered more incrementally during testing rather than specified upfront.\n\nFurther, to clarify, the V-model emphasizes the mapping between requirements and tests, but the process does not necessarily need to be sequential.\n\n\n**The validity and creation of KB**\n\n> Invalid Concepts in the KB\n\nWe did not filter out invalid concepts as we empirically observed that concepts suggested by the LLM are mostly valid. In practice, users can always delete invalid concepts or simply ignore them. However, we did not observe users deleting invalid concepts or making relevant complaints in the post-study survey. In an extra experiment to evaluate precision, we sampled 200 edges from our 4 KBs evaluated in Section 3, 50 each, and manually judge their correctness. We found 161 out of 200 edges valid, achieving a precision of 80.5\\%.\n\n> Dependence on ConceptNet\n\nWe picked ConceptNet because it's more about common sense and has generic semantic relations between concepts. Though knowledge tends to have bias, we expect generic semantic relations to be more or less generalizable – an assumption that’s validated by our user study and case studies. We will discuss alternative KBs in the revision, such as WikiData, DBpedia, etc., that tend to focus on more specific types of semantic relations.\n\n\n\n**Other technical details**\n\n> What is the role of gray lines in Figure 5?\n\nThe grey line is the identical line that plots the ideal situation, where each identified concept is distinct and falls into a different cluster.\n\n> Are the distances between concepts in Figure 6 drawn to scale? How do the authors place the clusters?\n\nYes. We use t-SNE to project all concepts’ SentenceBERT embeddings into a 2d space. We then filter out concepts explored by other participants and draw the clusters such that their relative positions are minimally changed.\n\n> What exact models do you use?\n\nWe started from the open-source huggingface models (https://huggingface.co/cardiffnlp/twitter-roberta-base-stance-climate, https://huggingface.co/cardiffnlp/twitter-roberta-base-stance-feminist) fine-tuned on the TweetEval dataset. We ended up just testing `gpt-3.5-turbo` as it achieves better performance.\n\n> What are the configurations of your hierarchical clustering?\n\nWe use the following setting: `sklearn.cluster.AgglomerativeClustering(n_clusters=None, affinity=’cosine’, memory=None, connectivity=None, compute_full_tree='auto', linkage='average', distance_threshold=0.7, compute_distances=False)`\n\n\n**References**\n\n[1] Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967.\n\n[2] Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge base of language models.\n\n[3] George E. P. Box. 2009. Statistics for experimenters: design, innovation, and discovery. Wiley-Blackwell.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is trying to solve a real and useful problem - requirement elicitation for NLP models; it is also one of the first few papers trying to do so, making it timely and relevant to the NLP testing community.\n- The authors' method is sound overall, and their tool seems user-friendly, although the authors have not released a demo version for reviewers to try.", "reasons_to_reject": "- This paper tries to improve one component of the previous AdaTest system, which looks incremental. Furthermore, the author's approach does not improve the error rates (as the authors acknowledge in L405 - L408) of the tested models, even though it helps explore more topics. \n\n- The authors rely on the closed-source API - `text-davinci-003` from OpenAI to generate the specific list of topics under an abstract topic; the system the authors try to improve (i.e., AdaTest) also uses `text-davinci-003` or similar APIs. However, this service is going to discontinue in January 2024. I expect to see authors rebuild their system (including both Weaver and AdaTest) on open-source models and reproduce the experiments; the proposed approach should show that it can work with and without a strong base LM (i.e., text-davinci-003).\n\n  As a side note, the authors claim to use ChatGPT (`gpt-turbo-3.5`) throughout the text, but they use text-davinci-003. It will also be nice to see authors reproduce results on `gpt-turbo-3.5` to make the experiment results and claims in the paper match each other.\n\n- The authors' user study seems incomplete from what I can see on L351 - L355. The authors may want to spend more time on the user study to make the experiments more complete before making a submission.", "questions_to_authors": "1. In software engineering, the requirement is specified before the developers even write the software, but the authors are testing existing models. Do authors believe the way of requirement engineering is different for traditional software and machine learning models?\n\n2. The Weaver tries to build and recommend fine-grained topics based on a knowledge graph (KG). However, there are a few issues in this process:\n   1. In Line 160, the generated topics are not checked for their validity.\n   2. The authors use the relations provided by ConecptNet as-is without considering its bias (for example, its coverage may be insufficient). Even though this choice seems reasonable, I expect the authors to discuss the bias of using existing relations in ConceptNet.\n   3. In Line 199, the authors propose to maximize diversity and relevance at the same time. However, it would be better to formulate this as a multi-objective optimization problem since we will likely land on a solution with very high diversity but low relevance.\n3. Is your user study single-blind or even double-blind? I do not see whether the users mentioned in Section 4 know which group (control vs. treatment) they are in, and I don't know if the authors could unconsciously influence the users.\n4. The design of Figure 5 is quite interesting. However, I do not understand the role of gray lines.\n5. Are the distances between concepts in Figure 6 drawn to scale? How do the authors place the clusters? These plotting details are important as the authors are trying to use this graphic as part of the evidence that their proposed approach (i.e., orange clusters) are better than the control group (i.e., blue clusters) as it generates clusters that are more spread out.\n6. Missing minor details\n   1. L327 - L330: What exact models do you use?\n   2. L371 - L386: What are the configurations of your hierarchical clustering?", "paper_topic_and_main_contributions": "This paper proposes a tool (Weaver) to help NLP model testers automatically develop specifications under an initial abstract and less well-defined topic, such as \"online toxicity.\" The tool addresses the limitations of a previous tool called AdaTest by mitigating the burden of testers of summarizing and organizing LM-generated test cases into a test tree. The controlled experiments show that Weaver can help testers explore more diverse topics, which benefits NLP testing.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 1: The `# Conecpt` should be average rather than sum.\n\n- L370: Redundant \"to\"."}, "extra_text": {"Justification_for_Ethical_Concerns": "- I do not see if an IRB approves your user study and how you compensate for your users (I have scoured the manuscript and the supplementary materials section on OpenReview, but I could not find anything). This approval and subsequent discussions are essential as some of the datasets you use (e.g., hateful memes and stance detection for feminism) could impose harm on the testers."}}]

Author rebuttal:

Meta-review: This paper proposes a new interactive system, WEAVER, to help/guide NLP model testers. The system generates concepts from a large language model (LLM) that are diverse but relevant to a task. The idea is that the generated concepts can provide additional tests for modeling testing. The authors' experiments show that WEAVER can help testers explore similar topics.

[Paper Clarity]: Both reviewers rtMG and Mdj9 raise concerns about the paper's clarity. In particular, reviewers rtMG have issues with the lack of technical detail around how the proposed system works. Whereas reviewer Mdj9 concerns are with regard to details not mentioned in the paper around the validity of new concepts, the authors's choice to use ConceptNet, and the authors' optimization objective. 

[Traditional Software Engineering Testing]:
Both reviewers hoyj and Mdj9 raise concerns about how the proposed approach is related to traditional software engineering testing that provides quick testing/error analysis.
Decision: Accept-Findings