SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruiqi Zhong* Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, Jacob Steinhardt
Exploring large corpora can generate useful discoveries but is time-consuming for humans. We formulate a new task, D5, that automatically discovers differences between two large corpora in a goal-driven way. The task input is a problem comprising a user-specified exploration goal ("comparing the side effects of drug A and drug B") and a corpus pair (collections of patients' self-reported reactions after taking each drug). The output is a goal-relevant description (discovery) of how these corpora differ (patients taking drug A "mention feelings of paranoia" more often). We build a D5 system, and to quantitatively evaluate its performance, we 1) build a diagnostic benchmark, SYND5, to test whether it can recover known differences between two synthetic corpora, and 2) contribute a meta-dataset, OPEND5, aggregating 675 open-ended problems ranging across business, social sciences, humanities, machine learning, and health. With both synthetic and real datasets, we confirm that language models can leverage user-specified goals to propose more relevant candidate discoveries, and they sometimes produce discoveries previously unknown to the authors, including demographic differences in discussion topics, political stances in speech, insights in commercial reviews, and error patterns in NLP models. Finally, we discuss the limitations of our D5 system, which discovers correlation rather than causation and potentially reinforces biases when misused; therefore, practitioners should treat the outputs of our system with caution.
Exploring large corpora and generating discoveries from them can be ad hoc and laborious. For example, to compare the side effects of drug A and drug B, doctors might inspect two large corpora of patients' self-reported reactions after taking each drug; based on ad hoc insights, they hypothesize that patients taking drug A more often "mentions feelings of paranoia", and then validate this hypothesis by laboriously inspecting the two corpora. Since machines can automatically process a large amount of texts, we might hope for ML systems to facilitate exploratory analyses like the one above.
However, an ML task requires a unified input-output space and evaluation metric so that it can be automated, benchmarked, learned, and analyzed. To this end, we formalize one type of exploratory analysis problem as a natural language generation task: goal driven discovery of differences between text distributions via language descriptions (D5). As shown in Figure 1, the input to the D5 task is a "problem" comprising a description of a user-specified exploration goal (understanding side effects) and a corpus pair (text samples from the distributions of self-reported reactions after taking each drug). The output is a "discovery" represented as a natural language predicate ("mentions feelings of paranoia"). We evaluate a discovery with two criteria (Section 3): (1) validity: it should describe a true difference (Zhong et al., 2022); and (2) relevance to the goal (McGarry, 2005).
*University of California, Berkeley, EECS Department. Email: ruiqi-zhong@berkeley.edu
Figure 1: Each problem in OPEND5 contains 1) a corpus pair, which has  $\sim 17\mathrm{K}$  samples on average and is partitioned into two halves called "exploration split" and "validation split", and 2) a natural language description of the exploration goal, which also contains information about how the corpus pair was collected. A D5 system takes the goal and the exploration split as inputs and generates valid and relevant discoveries in natural language as outputs. The underlined texts in the exploration goal vary across problems, while the rest are templates.
Since D5 is open-ended and aims at discovering unknowns, the most popular benchmark practice—comparing system-generated outputs with human-written references on a test set—is infeasible. We therefore design two evaluation strategies.
- Diagnostic: we synthesized a dataset of D5 problems with known solutions, SYND5, to diagnose whether a D5 system can recover known differences between two synthetic corpora. This strategy is cheap and automated but might not reflect user utility in real applications.
- Open-ended: we collected a dataset, OPEND5, by aggregating 675 open-ended D5 problems ranging across business, social sciences, humanities, health, and machine learning (Figure 2), comprising 4.4 million text samples in total across problem corpora. We then manually evaluated a subset of the output discoveries. This strategy is subjective and expensive, but useful for obtaining qualitative insights on more realistic applications.
These two strategies allow us to quantitatively evaluate and compare D5 systems. For example, we compared 1) the system from Zhong et al. (2022) designed to describe corpus-level differences without goals, and 2) a goal-conditioned variant that we develop in Section 4. We found language models successfully use the specified goal: the goal-conditioned variant is correct  $12\%$  more often on SYND5, and it produces relevant candidate discoveries  $31\%$  more often on OPEND5.
We envision OPEND5 to be a growing, diverse repository of open-ended D5 problems. They will not only help us evaluate D5 systems more reliably, but also allow the following operations:
Facilitate exploratory analysis. Every time we build a better D5 system, we can apply it to a repository of open problems and send the discoveries to researchers who posed them. We show this paradigm is plausible by using our system to automatically produce useful discoveries on OPEND5 (Section 6.1), including insights from commercial reviews, temporal and demographic differences in discussion topics, political stances and stereotypes in speeches, differences in lyric styles, and error patterns in NLP systems. We anticipate future systems to produce more discoveries.
Analyze the limitations of our evaluation. Using concrete examples from OPEND5, we show that our current evaluation metrics do not encourage diverse findings, do not always produce causal conclusions, and cannot evaluate discoveries involving heavy expert knowledge (Section 6.2). More D5 problems can help us identify more limitations, which inform areas for future improvement.
Train better D5 systems. Like other ML tasks, we can train a system once we have a dataset. We describe a self-supervised learning algorithm that uses a repository of problems (without reference solutions) to train LMs to propose more valid hypotheses (Section 4.3). As a proof-of-concept, we show that it can make LMs better describe the differences between small groups of text samples.
To conclude, we show that D5 can be quantitatively evaluated, automated, analyzed, and learned. Like other ML tasks, it would benefit from a more diverse, authentic, and larger dataset. We hope future works can gather feedback from domain experts and curate an ever-larger dataset of D5 problems, thus accelerating exploratory analyses and facilitating scientific discoveries.
2 Datasets: SYND5 and OPEND5
We first introduce how each input problem is formatted. Then we discuss 1) how we synthesized SYND5, which is used for automatic diagnostic evaluation, and 2) how we collected OPEND5, which is used to investigate the practical value of D5 systems in open-ended applications.
2.1 Task Format
Each D5 problem is represented by a corpus pair (Corpus A/B) and a description of the exploration goal. For example, Corpus A/B might be self-reported reactions after taking drug A/B, and the goal description would be “comparing the side effects of drug A and drug B”. The desired output is valid and relevant discoveries in the form of natural language predicates (Figure 1), e.g. Corpus A has more samples that “mentions feelings of paranoia”.
2.2 SYND5, a Diagnostic Benchmark with Reference Solutions and an Automatic Metric
To automatically diagnose a D5 system, we synthesized SYND5, a dataset of D5 problem with reference solutions. To synthesize Corpus A and Corpus B for each input problem, we used a language model (LM) to generate two corpora that simultaneously differ on two dimensions, one of which is goal-relevant and one of which is a distractor. For instance, suppose the goal is to "understand how Corpus A differs from Corpus B in terms of topic". Then we would synthesize an example where Corpus A is more sports-related while B is more art-related (goal-relevant: varying topic), while additionally Corpus A is in English while B is in French (distractor: varying language). The reference solution is the difference on the goal-relevant dimension, e.g. "is sports-related".
In more detail, to synthesize an example in SYND5, we first picked one goal-relevant and one distractor dimension from the set {topic, genre, language}, and sampled a value for each corpus and dimension (e.g. Corpus A: [sports, English]; Corpus B: [art, French]). We then synthesized Corpus A/B such that all its samples are in English/French (i.e. completely different on the distractor dimension) while  $V$  percent of them are sports-related/art-related, where we varied  $V$  from 0.6 to 1. Since the distractor difference is more salient, SYND5 penalizes D5 systems that ignore the goal and output the incorrect distractor difference "is in English". We synthesized 300 problems in total to create SYND5; see Appendix 9 for a detailed description of the pipeline.
To compute a D5 system's accuracy, we prompted Claude-v1.3 (Bai et al., 2022b) to judge how often the output discovery is semantically equivalent to the reference. We construct the prompt by using 6 pairs of predicates with the labels of "equivalent", "similar", or "irrelevant" as few-shot examples, and ask Claude-v1.3 to judge whether the output discovery and the reference are "equivalent". As a result, we can automatically diagnose a D5 system. See Appendix 12 for the prompt for equivalence judgement and Appendix 10 for two robustness checks, which (a) consider "similar" discoveries to be correct as well, and (b) use other LMs for equivalence judgement.
2.3 OPEND5, a Realistic Open-Ended Dataset without Reference Solutions
To evaluate a D5 system's utility under realistic applications, we also gathered OPEND5, a realistic dataset of 675 open-ended D5 problems. These problems range across business, social sciences, humanities, health, and machine learning; see Figure 2 for a few examples. To build OPEND5, two of the authors performed an extensive literature review on problems that could potentially benefit from our system, e.g., reading survey papers (Nguyen et al., 2020) and courses on computational social sciences, and skimming through the ACL proceedings from the past decade and datasets from Kaggle that have an NLP tag; we then annotated the exploration goals, scraped/generated the
$^{2}$ Our code is released at https://github.com/ruiqi-zhong/D5 and our code to download OPEND5 is released at https://github.com/petezh/OpenD5. Given the limitations of our system, practitioners should interpret its outputs with caution and not use it to fully automate scientific discoveries.
Figure 2: OPEND5 contains 675 problems. See citations in Appendix 23.
corresponding corpora, and post-processed them over nine months (see complete list of citations in Appendix 23). As shown in Figure 1, each goal describes the original dataset, how the two given corpora are generated, who is using the system, and what property of the two corpora does the user want to understand. Each OPEND5 problem is reviewed by at least two of the authors to reduce grammatical mistakes and ambiguous interpretations of the goal.
Each corpus contains around 17K text samples on average, and OPEND5 in total comprises 4.4 million distinct text samples. We use  $50\%$  of each corpus as the "exploration" split and  $50\%$  as the "validation" split. The system can only access the exploration split, while the validation split is reserved for the evaluators to validate the discovery. A validation split prevents overfitting the discoveries to the given samples and is analogous to the train-test split in machine learning.
Since we hope to build systems that can tackle challenging open-ended problems, we did not avoid cases where we do not know the ground truth answer. This is different from standard benchmarking practices, where humans can provide a reference solution to evaluate an AI system. However, even though we do not know the ground truth, once a system produces a discovery, we can still evaluate it. We present our evaluation metrics in the next section.
3 Evaluation Metrics for Open-End D5 problems
For the goal of comparing the side effects of drug A and drug B, how do we evaluate a system-generated discovery that Corpus A "mention feelings of paranoia" more often? First, it needs to be valid, such that indeed more samples from Corpus A satisfy this predicate, which can be evaluated (approximately) objectively. Second, it needs to be relevant to the goal of understanding side effects, which depends on the user's subjective judgement. We define validity and relevance below.
Validity. Similar to Zhong et al. (2022), we require an output discovery  $h$  to be a truth predicate on a text sample. For example, if  $h =$  "mentions about family", then  $h$  is true on the string  $x_{1} =$  "My daughter loves me" and false on the string  $x_{2} =$  "I'm going to school". Define  $T(h,x)\in [0,1]$  as "the certainty that  $h$  is true on  $x$ ", e.g.,  $T(h,x_1)\approx 1$  and  $T(h,x_2)\approx 0$ . We approximate  $T(h,x)$  by asking three Turkers how certain they are and averaging their responses (see Appendix 11 for details).
Let  $\mathcal{D}_A^{\mathrm{val}}$  and  $\mathcal{D}_B^{\mathrm{val}}$  denote the validation sets for Corpus A and B. We define the "validity"  $V$  as
$$
V (h) := \mathbb {E} _ {x \sim \mathcal {D} _ {A} ^ {\text {v a l}}} [ T (h, x) ] - \mathbb {E} _ {x \sim \mathcal {D} _ {B} ^ {\text {v a l}}} [ T (h, x) ]. \tag {1}
$$
Computing  $V(h)$  is expensive since it requires human annotations  $T(h, x)$  on a set of text samples even to evaluate a single discovery  $h$ . In practice, we do not have the budget to compute  $V(h)$  on the entire validation split; therefore, we approximate this quantity by randomly sampling from Corpus  $A$  and Corpus  $B$ . We use these samples to compute an empirical estimate of  $V$ , as well as a  $p$ -value for the null hypothesis that  $V \leq 0$  using a one-sided t-test.
Relevance. A discovery may be irrelevant even if  $V = 1$ . For example, if the goal is to understand the writing style differences between higher-scoring essays (Corpus A) and lower-scoring ones (Corpus B), the discovery that Corpus A "achieves higher scores" has high validity score by definition but irrelevant to the goal of understanding stylistic differences.
Therefore, we designed a procedure to evaluate relevance, where human or language model evaluators would score each discovery with  $2 / 1 / 0$ . The evaluators used the rubric below, which illustrates the meaning of each score with the essay example above:
- ②, relevant; e.g. the discovery "write in first person" is directly related to the writing style.
-  $①$ , indirectly relevant; e.g. the discovery "use the word "I", is not exactly a writing style, but can still inform the relevant underlying principle of "write in first person".
- ①, irrelevant; e.g. the discovery "argue for abortion" is unrelated to the writing style.
To minimize biases while comparing two systems, the evaluators are blind to which system generates which discoveries.
To conclude, an ideal discovery would have a high  $V$  value with a small  $p$ -value and achieve ratings of ② in relevance. In the next section, we will build a D5 system that addresses these criteria by first proposing goal-relevant candidate discoveries (hypotheses) and then automatically validate them.
Other metrics. We also explored two other subjective metrics, novelty (how difficult it is to generate the discovery) and significance (how beneficial it is to learn about the discovery). Due to space limit, we present their rubrics and related results in Appendix 14.
4 Methods: Building a D5 System
We describe our D5 system, which maps from a corpus pair and an exploration goal to a set of natural language predicates. Our system is inspired by a two-stage model of how humans discover patterns in data: creatively brainstorming hypotheses and then rigorously validating them on the data (Ludwig & Mullainathan, 2022). Analogously, we first propose hypotheses conditioned on the exploration goal and a subset of samples from the corpus pair (Section 4.1). We then use a language model to approximately compute the validity of each hypothesis, and output the valid ones as the final discoveries (Section 4.2). Our system closely mirrors that of Zhong et al. (2022), except that we leverage the goal to propose more relevant hypotheses. Finally, we present a self-supervised learning algorithm to improve an LM's ability to propose more valid hypotheses (Section 4.3); however, due to API access constraint, we cannot apply it to fine-tune gpt-3, so we provide a proof of concept experiment on F1an-T5 (Chung et al., 2022).
4.1 Hypothesis Proposer
We prompt gpt-3 (Ouyang et al., 2022) to propose hypotheses. Denoting the exploration split of Corpus A/B as  $\mathcal{D}_A^{\mathrm{exp}} / \mathcal{D}_B^{\mathrm{exp}}$ , we construct the prompt by concatenating a few random samples from  $\mathcal{D}_A^{\mathrm{exp}}$  and  $\mathcal{D}_B^{\mathrm{exp}}$ , the exploration goal, and an instruction to output a list of hypotheses. Figure 3 (left) depicts an example of the resulting prompt, together with a typical language model output.
Since the entire corpus pair might not fit into one prompt, we construct multiple prompts with different sets of samples so that gpt-3 can "see" as many different samples as possible in our pipeline. We continue sampling hypotheses with different prompts until obtaining a set of 60 hypotheses, which we call  $H_{\mathrm{init}}$ . Appendix 15 includes more details on selecting the sets of samples for different prompts.
4.2 Hypothesis Validator
Many hypotheses in  $H_{\mathrm{init}}$  have low validity: they are not more often true on  $\mathcal{D}_A$  than on  $\mathcal{D}_B$  (i.e.  $V(h) \leq 0$ ). To automatically filter them out, we use a language model  $T'$  to simulate the Turkers'
Figure 3: All underlined content in the prompt differs across problems, while the other content in the prompt is templated. Left: proposer prompt. The generated hypotheses are in blue. All content with colored background is excluded for brevity. For the baseline of not using the exploration goal, we removed the "exploration goal" block from the prompt. Right: the validator prompt.
judgement  $T$  and hence approximate the validity score  $V(h)$  with the function  $V^{\prime}(h)$ , defined as
$$
V ^ {\prime} (h) := \mathbb {E} _ {x \sim \mathcal {D} _ {A} ^ {\exp}} [ T ^ {\prime} (h, x) ] - \mathbb {E} _ {x \sim \mathcal {D} _ {B} ^ {\exp}} [ T ^ {\prime} (h, x) ]. \tag {2}
$$
To compute  $T'$ , we ask F1an-T5 whether  $x$  satisfies  $h$  with the prompt shown in Figure 3 (right). To better simulate Turker's judgment, we collected additional Turker annotations to fine-tune FLAN-T5 (see Appendix 16 for details about the data collection process). We then obtain a significance value  $p'$  by performing a t-test to compare the mean value of  $V'(h, x)$  on the exploration split of Corpus A to that of Corpus B, rule out the hypotheses with  $p'$  greater than 0.001, and output the remainder as discoveries. Finally, we obtain additional discoveries by repeating the same process but asking our system to propose and validate hypotheses about Corpus B rather than Corpus A. Appendix Figure 5 visualizes our entire pipeline and Appendix 8 discusses the computational resources we used.
4.3 Self-Supervised Learning with Open-Ended Problems: A Proof of Concept
Since D5 problems are open-ended, future systems could potentially produce discoveries with higher validity scores than any known discovery. Therefore, we design a self-supervised learning algorithm to improve an LM's ability to propose more valid hypotheses, using the principle that it is easier to validate a discovery than to generate one.
Algorithm. Suppose we are given a set of problems for training and an initial language model  $m_0$ . Our goal is to automatically generate a set of prompt-completion pairs to fine-tune  $m_0$  so that it can propose hypotheses that are more valid. To generate a prompt, we randomly sample a problem and create a proposer prompt following the procedure in Section 4.1. To generate the desired completion given a prompt, we sample multiple hypotheses from  $m_{\mathrm{init}}$ , approximate their  $V'$  score on the samples in the proposer prompt with the same language model  $m_{\mathrm{init}}$  (Section 4.2), and select the highest scoring hypothesis. Finally, we use the prompt-completion pairs to fine-tune  $m_0$ .
A Proof of Concept Experiment. Since we cannot fine-tune text-davinci-003, we can only experiment with Flan-T5-xx1 (Chung et al., 2022), an open-sourced instruction-tuned model that might only work well for easier "mini-problems". As a proof of concept, we tested the above self-supervised learning algorithm on the task of describing groups of four samples, where each group comes from a text cluster.
We computed both the automated "self-evaluation" validity score  $V'$  and the "true" validity score  $V$  according to Turker evaluation for evaluation. After self-training,  $V'$  improves substantially from 0.22 to 0.37, and the  $V$  improves from 0.07 to 0.10, with a  $p$ -value of 0.02. This result provides preliminary evidence that self-training could be applied to a large set of problems to improve the
text-davinci-003	w/ goal	wo/ goal	gpt-4	w/ goal	wo/ goal
w/ validator	12%	2%	w/ validator	27%	15%
wo/ validator	4%	1%	wo/ validator	8%	5%
Table 1: The accuracy on SYND5 using different proposers, with/without incorporating goals, and with/without using placeholders. Using the validator, the goals, and gpt-4 leads to better results.
Hypothesis Relevance	②	①	①	average
Using the goal	79%	9%	12%	1.68
Not using the goal	52%	16%	32%	1.20
Table 2: How often the hypotheses proposed by text-davinci-003 are rated by the authors as  $② / 1 / 0$  in terms of relevance (Section 3). Overall, using the goal significantly increases relevance.
validity of the hypotheses; we expect future validationers to simulate human judgments better, hence decreasing the approximated gap of improvement between  $V$  and  $V'$ . We discuss more training and evaluation detail in Appendix 20.
5 Quantitative Evaluation on SYND5 and OPEND5
We show that both SYND5 and OPEND5 can be used to quantitatively evaluate D5 systems. Since SYND5 is automatic, we used it to compare a broad range of D5 systems and studied the contributions of three different factors: the quality of the proposer model (gpt-4 vs. text-davinci-003), the use of a validator, and the use of a goal. We then further investigated the effect of using goals under realistic applications through human evaluation on OPEND5.
Automatically comparing different variants with SYND5. As mentioned above, we ablated 3 factors, resulting in  $2^{3} = 8$  variants. We compared 1) using text-davinci-003 vs. gpt-4 as the hypothesis proposer; 2) using the validator to compute  $V^{\prime}$  for each hypothesis and outputting the highest-scoring hypothesis, vs. not using the validator and outputting a random hypothesis; and 3) using the goal vs. replacing it with "I want to understand how Corpus A is different from Corpus B." We then automatically calculated the accuracy for each variant as described in Section 2.2.
We report the results in Table 1. We find that using the validator and the goals significantly improve the performance, and gpt-4 outperforms text-davinci-003 with goals and the validator ( $p < 1\%$  under a t-test). We conducted two additional robustness checks in the Appendix 10: (a) using text-davinci-003 instead of Claude-v1.3 to judge predicate equivalence, and (b) considering discoveries semantically similar to the references also to be correct; our conclusions do not change.