SlowGuess/NAACL_2022_OCR huggingface dataset:
Jamar L. Sullivan Jr., Will Brackenbury, Andrew McNutt, Kevin Bryson, Kwam Byll, Yuxin Chen, Michael L. Littman†, Chenhao Tan, Blase Ur University of Chicago, † Brown University
{jlsullivan2, wbrackenbury, mcnutt, kbryson}@uchicago.edu {kbyll, chenyuxin, chenhao, blase} @uchicago.edu  $\dagger$  mlittman@cs.brown.edu
In the context of data labeling, NLP researchers are increasingly interested in having humans select rationales, a subset of input tokens relevant to the chosen label. We conducted a 332-participant online user study to understand how humans select rationales, especially how different instructions and user interface affordances impact the rationales chosen. Participants labeled ten movie reviews as positive or negative, selecting words and phrases supporting their label as rationales. We varied the instructions given, the rationale-selection task, and the user interface. Participants often selected about  $12\%$  of input tokens as rationales, but selected fewer if unable to drag over multiple tokens at once. Whereas participants were near unanimous in their data labels, they were far less consistent in their rationales. The user interface affordances and task greatly impacted the types of rationales chosen. We also observed large variance across participants.
There is a growing effort in NLP to collect human-provided justifications in addition to labels during annotation (Wiegreffe and Marasovic, 2021). These justifications can take many forms, ranging from subsets of input tokens (henceforth rationales) to natural language. They can provide additional information on how the label is derived from the input and are hypothesized to improve the robustness of models by correcting spurious correlations and enhancing the transparency of models for human-AI collaboration. Tan (2021) points out that diverse instructions can be used to collect even simple forms
Zaidan et al. (2007): "To justify why a review is positive [negative], highlight the most important words and phrases that would tell someone [not] to see the movie." Sen et al. (2020): "Select ALL words and phrases in the review that would suggest the review is positive [negative]."
Table 1: Example instructions from prior work for annotating rationales in sentiment analysis.
of rationales. Table 1 shows example instructions from prior work on sentiment analysis for movie reviews. Zaidan et al. (2007) define positive reviews as telling someone to see the movie, asking the human to annotate "the most important" words and phrases. Sen et al. (2020) instead ask the human to highlight "ALL" words and phrases that "suggest the review is positive" without defining "positive."
It remains an open question how these divergent instructions and other human-centered design decisions affect the collection of rationales. In this work, we conduct the first study of how to collect rationales by investigating how these factors affect the process and the rationales ultimately collected. We use sentiment analysis as a testbed. Rather than building a large dataset, we study how different design choices impact the collection of rationales. We thus ask all participants to label the sentiment of the same ten IMDB movie reviews and provide rationales for their selections, varying the instructions and interface in a between-subjects experiment.
Our experiment had three phases of data collection, with 332 participants total across phases. First, we assigned participants to one of six sets of instructions explaining what rationales are. Second, we tested a variation in which participants annotated rationales not just in support of the bi
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 547 - 560 July 10-15, 2022 ©2022 Association for Computational Linguistics
nary classification label they chose, but in support of the opposite label, which we term two-sided rationales. Finally, we varied both a key interface affordance—the ability to drag to select multiple words at once—and whether participants were asked to select “words and phrases” or just “words.” Superficially, one might expect the dragging affordance not to affect rationales since annotators can always click all relevant words individually.
These three phases of our between-subjects experiment let us answer four key research questions:
- RQ1: How do instructions and interfaces impact the fraction of words selected as rationales?
- RQ2: How consistently do different annotators select rationales? Do instructions and interfaces impact inter-annotator consistency?
- RQ3: How do instructions and interfaces shape which types of words are selected as rationales?
- RQ4: How long does selecting rationales take?
When instructed to select "words and phrases" as rationales and given an interface where they could click individual words or drag over multiple words, the median participant selected about  $12\%$  of words as rationales. This fraction did not vary significantly based on instructions. However, we observed high variance across participants; some selected only a few words, while others selected one-third of them. Without the dragging affordance, participants instead selected only about  $5\%$  of words. Participants (with dragging) selected  $4\%$  of words as rationales for the label not chosen.
Participants were near unanimous in the labels chosen (Krippendorff's  $\alpha$  close to 1.0). In contrast, rationales were fairly inconsistent ( $\alpha$  around 0.3) even when keeping the instructions and interface constant. As such, rationales collected from only a few annotators can be highly variable, impacting their downstream usage. The instructions and interface minimally impacted inter-annotator reliability.
Both the availability of the dragging affordance and the task (selecting "words and phrases" or just "words") greatly impacted which words were selected as rationales. In part, these factors impacted how much context participants included in rationales. Given the dragging affordance, participants tended to select full phrases like "worst movie I have seen" as rationales. Without this affordance, they tended to select only "worst movie." Asked to select "words" (vs. "words and phrases"), they tended to select single adjectives, such as "worst." In contrast, instructions had only a small impact.
A few words in each review were labeled as rationales by nearly all participants regardless of their assigned condition. However, participants varied greatly from each other in whether they labeled parts of reviews expressing ambiguous and indirect thoughts as rationales, as well as how they handled negation and complex sentences. Finally, we found that providing one-sided rationales took about  $2.5 \times$  as long as providing only labels. Thus, researchers may consider rationales worthwhile to collect if they provide greater benefit than having  $2.5 \times$  as much data with only labels. We conclude by discussing lessons for collecting rationales.
Our work is primarily related to two strands of prior work: the construction of explanation datasets and understanding the quality of annotated rationales.
Explanation Datasets: Driven by growing interest in interpretable ML (Doshi-Velez and Kim, 2017; Arrieta et al., 2020), many researchers (McDonnell et al., 2016, 2017; DeYoung et al., 2019; Sen et al., 2020; Zaidan and Eisner, 2008; Zaidan et al., 2007; Lehman et al., 2019; Thorne et al., 2018; Khashabi et al., 2018; Carton et al., 2018) have collected datasets of human explanations. See Wiegrefe and Marasović (2021) for a survey.
Despite this interest, to our knowledge we are the first researchers to focus on the HCI design space of rationale collection. In particular, we focus on a particular type of explanation: highlighting parts of the input that suggest the chosen label without further explanations of the mechanism. This approach is sometimes also termed feature feedback.
Aiming to improve the user experience of data labeling, Choi et al. (2019) used attention-based models to automatically highlight potentially important words. They also enabled crowdworkers to highlight missing important words (rationales) to improve their attention model, but did not study the design of rationale-selection interfaces.
Quality of Existing Rationales: Further motivating the study of the rationale-annotation process, Carton et al. (2020) found substantial variance in the quality of existing rationale datasets. They measured the sufficiency and comprehensiveness of human rationales based on machine learning models. Many studies have also found that learning from existing rationale datasets can hurt model performance, or that any improvement is marginal or
Figure 1: Our Baseline interface for annotating rationales. Users highlight words or phrases that suggest their chosen sentiment label by clicking on individual words (as here) or by dragging over phrases.
limited to secondary goals like explanation quality (Plumb et al., 2020; Ross et al., 2017; Zaidan et al., 2007). Recent work in the HCI community has shown that unclear or imprecise requester instructions can lead to differences between (label) annotations (Chang et al., 2017), partially inspiring our examination of the instructions provided for annotating rationales. By investigating the annotation process, our work can shed light on how the shortcomings of rationale datasets may arise.
In this section, we present the design of our experiment and its detailed procedures.
3.1 Study Structure
The broad task presented to participants was first to label the sentiment of a movie review as positive or negative, and then to choose words and phrases from the review supporting that label (the rationale). We randomly assigned each participant to a condition (Section 3.2) specifying the precise instructions given for selecting rationales, the specific rationale-selection task, and the user interface. Before proceeding, we provided a tutorial (with animated examples) adapted for each condition about how to select the label and rationales. At the end of the tutorial, we asked two quiz questions, giving the correct answer if participants answered incorrectly.
Participants were then redirected to a React web app we created. For each of ten movie reviews (Section 3.3), participants read the review and labeled it as positive or negative. On the next screen (Figure 1), they provided their rationale. The task wording, specified by their condition, appeared in
Baseline: "Select the most important words and phrases in the review that suggest the review is positive [negative]." Generalize: Baseline followed by "Select ONLY words and phrases that, if they appeared in other reviews, would also suggest positive [negative] sentiment."
%, Not Shown: Baseline followed by "Aim to select between  $10\%$  and  $20\%$  of the words in the review."
%, Shown: The same as  $\%$ , Not Shown, but with a dynamic counter at the top of the screen showing the percentage of words selected.
Zaidan: "To justify why a review is positive [negative], highlight the most important words and phrases that would tell someone [not] to see the movie."
Sen: "Select ALL words and phrases in the review that would suggest the review is positive [negative]."
Table 2: The six sets of instructions representing the conditions in Phase 1 of data collection.
blue at the top. Except when specified otherwise by their condition, participants were asked to select words and phrases as rationales either by clicking words individually or by dragging across multiple words at once. Words and phrases changed from black text on a white background to white text on a blue background when selected. They could be deselected and reselected. We required that participants select at least one word as a rationale per review to continue. Although we focus on quantitatively analyzing the process and outcomes of rationale annotation, we collected other information, including an exit survey (see Appendix B).
3.2 Conditions
Participants were randomly assigned to a condition. The set of conditions changed across phases.
Phase 1 compared the six sets of instructions shown in Table 2. Baseline aimed to be neutral and straightforward, also providing a baseline for creating subsequent conditions. Generalize tested whether prompting annotators to focus on generalizability impacted rationales.  $\%$  Not Shown and  $\%$ , Shown tested encouraging annotators to select a specified fraction of words; the latter also showed a dynamic counter of the fraction. Finally, to compare with prior work, Zaidan and Sen were instructions used in similar tasks by Zaidan and Eisner (2008) and Sen et al. (2020), respectively.
Phase 2 investigated augmenting traditional rationales for the binary classification label selected (termed regular rationales) with rationales for the label not selected (opposite rationales). We use one-sided to mean collecting only regular rationales (as in Phase 1 and Phase 3) and two-sided to mean collecting both regular and opposite rationales. Prior work has studied only one-sided ra
tionales. We repeated three of the most promising instructions from Phase 1: Baseline; Generalize; and Zaidan. We also tested two-sided analogues of each: Baseline, 2-sided; Generalize, 2-sided; and Zaidan, 2-sided. An example instruction for opposite rationales in Baseline, 2-sided was: "Even though you felt the review was positive overall, select the most important words and phrases in the review that might instead suggest the review is negative." See Appendix A for the full wordings.
Phase 3 examined the impact of varying interface affordances and the task. We used the one-sided version of Zaidan (repeated from both Phase 1 and Phase 2) as our baseline because it was most representative of both the other conditions and prior work. We hypothesized that removing the drag affordance—forcing participants to click each word in multi-word phrases individually—might make participants more intentional and careful in rationale selection. We tested a version of this limited interface (Zaidan, No Dragging) that, as in other conditions, asked participants to select "words and phrases" as rationales. In contrast, (Zaidan, No Dragging, Words Only) only asked participants to select "words" as rationales. We hypothesized this change would make participants less likely to select potentially superfluous words. Finally, to understand how much additional time rationale annotation requires, as a baseline we ran a Labels Only condition in which participants did not provide rationales, representing common practice.
3.3 Dataset and Review Selection
We used a highly cited dataset of 50,000 polarized movie reviews Maas et al. (2011) originally collected from IMDB. Because our goal was to collect rich data from many participants about a small set of reviews, we chose a set of ten reviews that we showed to all participants in the same order. The reviews in the full dataset vary greatly in length and the contents of the review. To capture these variations in our sample, we chose two highly positive reviews, two highly negative reviews, two long reviews (one positive, one negative), two short reviews (one positive, one negative), and two ambiguous reviews (in which the review text primarily summarizes the film, as opposed to justifying the rating). We chose the last category to understand what parts of a review other than subjective statements of opinion would be chosen as rationales. Appendix D contains these ten reviews' full text.
3.4 Recruitment and Participants
We recruited participants on the Prolific crowdsourcing service for a "study in which you label movie reviews." We required participants live in the USA or UK and have a  $95\%+$  approval rating over  $100+$  prior tasks. To minimize fatigue, we designed the study to take 30 minutes; it took the median participant 33 minutes. We compensated participants $10 USD. Recruitment for subsequent phases excluded participants from previous phases. Data for Phase 1 and Phase 2 was collected in August 2021, and data for Phase 3 in September 2021.
We had 332 participants (119 in Phase 1, 125 in Phase 2, and 88 in Phase 3).  $52.7\%$  were men,  $44.3\%$  were women,  $1.8\%$  were non-binary, and  $1.2\%$  preferred not to say. Regarding age ranges,  $15.4\%$  were 18-24,  $36.1\%$  were 25-34,  $25.3\%$  were 35-44,  $13.0\%$  were 45-54,  $6.9\%$  were 55-64,  $2.7\%$  were  $65+$ , and  $0.6\%$  preferred not to say.  $1.8\%$  preferred not to state their race, while  $75.6\%$  identified as white,  $7.8\%$  as Asian,  $7.5\%$  as Black,  $3.3\%$  as Hispanic/Latine,  $0.3\%$  as Native American, and  $3.6\%$  as mixed race.  $97.3\%$  spoke English natively.
At least 18 participants were randomly assigned to each condition. To facilitate visual comparisons between conditions in Section 4, we randomly selected 18 participants for each condition containing more than 18, reporting results only on those 18.
We present our key results organized by the four research questions described in Section 1.
4.1 Fraction of Words Selected (RQ1)
The fraction of words selected as rationales varied slightly across instructions, but substantially across both interfaces and individual participants. The median participant per Phase 1 condition selected between  $8.7\%$  and  $13.4\%$  of the 2,254 total words (across 10 reviews) as rationales; differences across conditions were not significant. In contrast, individual participants varied greatly. For example,  $25\%$  of Zaidan participants selected under  $8.8\%$  of words as rationales, while  $25\%$  selected over  $18.2\%$ . The standard deviation  $(\sigma)$  per Phase 1 condition ranged from  $4.1\%$  to  $6.9\%$ .
While prior work focused on rationales supporting the binary classification label selected, in Phase 2 we also investigated rationales for the label not selected, which we termed opposite rationales (vs. regular rationales). We found that the me
Figure 2: Distribution of the total fraction of words chosen as rationales (across all ten reviews) by condition.
dian participant in two-sided conditions selected between  $3.5\%$  and  $4.8\%$  of words as opposite rationales (orange boxes in Figure 2), versus between  $10.3\%$  and  $12.9\%$  as regular rationales. This suggests one-sided rationales may miss some potentially explanatory information, augmenting prior work that noted the presence of positive words in negative reviews (Aithal and Tan, 2021).
In Phase 3, we varied both the task and interface, which significantly impacted the fraction of words selected as rationales (Welch's  $t = -7.367$ ,  $p < .001$ ). Requiring each word to be clicked individually resulted in fewer words being selected as rationales than also allowing participants to drag over multiple words at once; the fractions of words selected by the median participants in Zaidan, No Dragging (5.7%) and Zaidan, No Dragging, Words Only (4.5%) were much lower than in Zaidan (14.2%). However, the range among participants asked to select "words" (1.8% to 16.1%) was smaller than for those asked to select "words and phrases" (1.3% to 27.8%), as shown in Figure 2.
4.2 Consistency of Rationale Selection (RQ2)
Participants were highly consistent in the binary classification labels they chose, but far less consistent in the rationales they chose. We calculated Krippendorff's  $\alpha$ , a common measure of inter-rater agreement applicable to any number of coders, for both labels and rationales. For six of the ten reviews, the  $\alpha$  for classification labels was 1.0 (unanimous agreement) in every condition. For the remaining four reviews (two long and two ambiguous reviews),  $\alpha$  was always at least 0.8 in every condition. We suspect long reviews are more likely to include mixed opinions, and the sentiment of reviews mostly summarizing a movie is debatable.
In contrast,  $\alpha$  for rationales ranged from 0.28 to 0.35 (mean across reviews) for each one-sided condition in Phase 1 or Phase 2. As shown in Figure 3,
Figure 3: Inter-annotator agreement (Krippendorff's  $\alpha$ ) for rationales by condition (row) and review (column).
$\alpha$  tended to be highest for short reviews and lowest for long reviews and ambiguous reviews. For the two-sided conditions in Phase 2, we found that the mean  $\alpha$  for regular rationales (those matching the chosen label) similarly ranged from 0.27 to 0.29. However,  $\alpha$  for opposite rationales ranged from 0.19 to 0.26, indicating somewhat lower agreement.
Phase 3's Zaidan, No Dragging, Words Only condition, the only one in which we told participants to select "words" (vs. "words and phrases"), had the highest mean  $\alpha$ : 0.38. In contrast, the mean  $\alpha$  was 0.29 for Zaidan, No Dragging, which differed only in asking participants to select "words and phrases." In other words, participants were somewhat more consistent when selecting only words as rationales.
4.3 The Words Chosen as Rationales (RQ3)
Figure 4 is a heatmap showing how many of the 18 participants per condition chose particular words in the Short-Negative review as rationales. Our supplementary data file contains heatmaps for the other nine reviews. This heatmap exemplifies a few important trends detailed in the remainder of this section. First, a handful of words were selected as rationales by nearly all participants regardless of their condition. However, there was a long tail of words selected by some or all participants in only certain conditions. Specifically, varying the instructions minimally impacted the rationales. Varying the interface, however, had a large impact.
Similarity of Rationales: To quantify how the words selected as rationales compared across conditions, we computed the pairwise Pearson correlation of conditions' frequency vectors (e.g., as visualized in Figure 4) concatenated across all ten reviews. We computed correlations both for sets of
Figure 4: The number of participants (of 18) who selected each word in the Short-Negative review as a rationale. All labeled the review as negative. We observed high agreement about a few words, but a long tail of disagreement.
Figure 5: Pairwise Pearson correlations of condition regarding the frequency with which participants selected words as rationales (as in Figure 4). These frequencies represent either 3 random participants per condition (averaged across 1,000 runs) or all 18. The Phase 2 table compares only regular and opposite rationales.
three randomly selected participants per condition (averaged across 1,000 runs) and for all 18.
As shown in Figure 5, Phase 1 conditions had average pairwise correlations between 0.56 and 0.65 for sets of three participants, and between 0.91 and 0.95 for all 18 participants. In short, the instructions had only a minor impact on which words were selected as rationales.
In contrast, the interface had a greater impact on the words chosen as rationales. The two Phase 3 conditions without the dragging affordance had correlations of only 0.51-0.53 (3 participants) or 0.76-0.84 (18 participants) with Zaidan.
Note that we observed large differences in correlation values between sets of 3 participants and sets
of 18 participants because a number of words were selected by only some participants, as opposed to nearly all (see Figure 4). Participants behaved similarly across conditions regarding these debatable rationales, so these differences were smoothed out in a large set of participants (e.g., 18).
Context in Rationales: A key aspect of how the interface impacted participants' approach was how much context the rationale contained. The phrase "worst movie I have seen" in Figure 4 provides a clear example. Every participant in all three relevant Phase 3 conditions selected "worst" as a rationale. However, whereas nine Zaidan participants (50%) selected the whole phrase as a rationale, the same was true of only three Zaidan, No Dragging participants (16.7%) and one Zaidan, No Dragging, Words Only participant (5.6%). Furthermore, whereas eight Zaidan, No Dragging participants (44.4%) selected "worst movie," only three Zaidan, No Dragging, Words Only participants (16.7%) did.