SlowGuess/NAACL_2022_OCR huggingface dataset:
"This movie is so bad" in Figure 4 demonstrates a related distinction. Roughly half of participants whose interface had the dragging affordance (all Phase 1 conditions and Phase 3 Zaidan) selected "movie is so bad," and the vast majority selected at least "so bad." In contrast, lacking a dragging affordance in Zaidan, No Dragging and Zaidan, No Dragging, Words Only, most participants selected only "bad." Figure 6's Phase 3 section gives more examples of similar phenomena in other reviews.
Figure 6: Illustrative examples of differences across conditions in participants' rationales from other reviews.
Figure 7: Distribution of phrase length (the number of consecutive words selected) in Phase 3 with outliers removed. Disabling dragging resulted in shorter phrases.
Changes in participants' approach based on the inclusion/absence of the dragging affordance and the selection task also manifested in the distribution of phrase length (the number of consecutive words selected as rationales). As shown in Figure 7, participants with a dragging affordance selected rationales of various lengths; two-word rationales were most common. Without the dragging affordance but still asked to select "words and phrases,"  $46.8\%$  of rationales contained just one word. However, when instead asked to select "words,"  $71.0\%$  of rationales contained one word.
Parts of Speech: The interface, and to a lesser extent the task, impacted the parts of speech (POS) selected as rationales for sentiment classification.
Removing the drag affordance, as well as asking participants to select only words, led participants to focus on adjectives. Our analysis used the Python NLTK pos_tag module.
For example, among the words selected as rationales in Zaidan (Phase 3),  $26.9\%$  were nouns,  $17.8\%$  were verbs, and  $17.1\%$  were adjectives. In stark contrast, for Zaidan, No Dragging, Words Only, adjectives were the most common  $(37.4\%)$  followed by nouns  $(22.3\%)$  and verbs  $(19.4\%)$ .
Looking at this data differently,  $31.3\%$  of appearances of adjectives were selected as rationales in Zaidan, alongside  $10.3\% - 25.1\%$  of appearances of each other POS. As previously mentioned, Zaidan, No Dragging, Words Only participants selected far fewer words overall. Nonetheless,  $22.0\%$  of appearances of adjectives were still selected, but only  $0.6\% - 6.7\%$  of appearances of each other POS.
That said, participants' focus on adjectives may be a byproduct of our choice of sentiment analysis as the annotation task. Relative to other parts of speech, adjectives are more likely to encode sentiment information. Future work should examine how the POS distribution differs for other tasks.
Handling Complexity: Participants varied in whether they tagged as rationales phrases and sentences conveying complex thoughts and relationships. For instance, a few participants in each condition generally selected most of the long final sen
tence in Figure 4 as a rationale: "the only reason why you would watch it is if all the rest of the movies on earth as well as t.v. had been destroyed." This phrase only indirectly comments on the movie being reviewed, which appears to have divided participants. In contrast, the majority of participants in conditions without the drag affordance selected just the final word: "destroyed."
In part, this phenomenon seemed to relate to the generalizability of the thought being expressed. For instance, in "you'll be glad I didn't say too much," (Figure 6, Phase 1), most Generalize participants selected "you'll be glad." In any case, about half of participants across Phase 1 conditions selected "glad." This oblique reference does not comment directly on the movie being reviewed, yet one can imagine parts of this phrase indicating positive sentiment in other contexts. The opposite approach to generalization is visible in Figure 4. Seven participants each in Baseline, 2-sided and Zaidan, 2-sided selected "Madsen fans" as opposite rationales. This phrase references a specific actor and is unlikely to generalize, so only two Generalize, 2-sided participants selected this phrase.
Bifacial Words: In some cases in two-sided conditions (Phase 2), particularly when the review was expressing a complex thought, the same word was selected as both a regular and an opposite rationale. For example, in the Generalize, 2-sided and Zaidan, 2-sided conditions,  $72.2\%$  and  $77.8\%$  of participants, respectively, chose the same word for both sides of their rationale at least once across the ten reviews. For example, as in Figure 6, this phenomenon appears for negation ("not very interesting" vs. "interesting") and concepts that are absent ("instead of going for the usual happy ending" vs. "happy ending"). This observation shows that modeling contextual information is critical for leveraging rationales. Such complex rationales may have limited use in bag-of-words approaches.
4.4 Time Taken to Provide Rationales (RQ4)
While rationales hold promise for supporting downstream natural language processing tasks, collecting rationales alongside classification labels takes longer than collecting labels alone. Quantifying how much longer, and thus how much less data would be labeled with rationales, can inform whether this richer rationale information justifies labeling less data. While some prior work (McDonnell et al., 2017; Zaidan et al., 2007) has timed
rationale selection informally, it did not examine variations among a large, diverse set of participants.
Compared to only collecting labels, also collecting one-sided rationales took roughly  $2.5 \times$  as long (comparing Labels Only to Zaidan in Phase 3). This finding suggests that rationales must provide as much benefit as having  $2.5 \times$  as much data with only labels to be worthwhile. Providing two-sided rationales took roughly  $3.8 \times$  as long.
Comparing one-sided conditions in Phase 1 and Phase 2, the instructions did not significantly impact the time taken. In Phase 3, Zaidan, No Dragging and Zaidan, No Dragging, Words Only participants took significantly less time than Zaidan participants ( $t = -5.254$ ,  $p < 0.001$ ), though participants in those conditions also selected significantly fewer words as rationales (see Section 4.1).
5 Concluding Discussion
We conducted the first study of how to collect rationales. Our results have implications for UI design for this task, as well as for how rationales are used. Affordances: Interface affordances mattered more than instruction wordings, suggesting that HCI factors overlooked in prior work critically impact the characteristics of rationales collected from humans. It is critical that future work carefully consider, and also carefully report, these design factors to allow for replicable studies and effective comparisons. We also encourage researchers to open-source interfaces used for data annotation.
Low Inter-annotator Agreement: The low inter-annotator agreement we observed draws into question approaches in past work (DeYoung et al., 2019) of using rationale annotation as ground truth to evaluate machine-generated rationales. It also suggests the need for more nuanced ways to combine rationale annotations from humans, rather than simply treating rationales as another binary classification. To this end, it may be important to embrace these exhibited ambiguities and develop novel algorithms that can learn from the distributions of rationale annotations. The role of phrases highlights the importance of moving beyond bag-of-words approaches in incorporating rationales. In contrast, if researchers want to emphasize collecting consistent (if semantically limited) rationales from annotators, it may be best not to provide a drag affordance.
Two-sided Rationales: While prior work focuses on one-sided rationales, our results show that a non-trivial fraction of words imply the opposite
sentiment. This finding is especially important for sentiment analysis, and it echoes recent studies showing that negative reviews nonetheless contain positive words (Aithal and Tan, 2021). However, two-sided rationale collection took our participants far more time than one-sided, suggesting that future work investigate UX strategies to reduce this time.
Limitations: Our work focused on a single task, sentiment analysis, and was based on ten curated reviews. Furthermore, we explored only one of many possible workflows (e.g., selecting rationales at the same time as selecting the sentiment label). Experiments with diverse datasets and tasks, as well as alternative workflows, can verify the robustness of our findings. However, we believe our work represents a valuable contribution toward understanding how HCI factors impact rationale annotation.
Risks: Annotation seeking to maximize rationale consistency may exclude diverse opinions.
Availability: We have made data and code available in both the online data supplement and on GitHub (Sullivan Jr. et al., 2022). Our data supplement contains heatmaps analogous to Figure 4 for all ten reviews, as well as the full data (routines, detailed timing/click information, survey responses) for the  $92\%$  of participants who granted us optional permission to release their data. Our open-source code consists of our rationale-annotation UI, which was written in the React framework.
Ethics: Our study was determined to be exempt by UChicago's IRB. We built on our institution's model consent form for online studies. Participants checked boxes to affirm consent. We did not collect any PII beyond Prolific IDs. Our compensation ($10 USD) for a study that took a median of 33 minutes corresponds to an $18 USD hourly wage.
This material is based upon work supported by the National Science Foundation in collaboration with Amazon under Grant No. 1939728, as well as work supported by the National Science Foundation under Grant No. 2126602.
Madhusudhan Aithal and Chenhao Tan. 2021. On positivity bias in negative reviews. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).
Alejandro Barredo Arrieta, Natalia Diaz-Rodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, et al. 2020. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58:82-115.
Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2018. Extractive adversarial networks: High-recall explanations for identifying personal attacks in social media posts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Samuel Carton, Anirudh Rathore, and Chenhao Tan. 2020. Evaluating and characterizing human rationales. arXiv:2010.04736.
Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017. Revolt: Collaborative crowdsourcing for labeling machine learning datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
Minsuk Choi, Cheonbok Park, Soyoung Yang, Yonggyu Kim, Jaegul Choo, and Sungsoo Ray Hong. 2019. AILA: Attentive interactive labeling assistant for document classification through attention-based deep neural networks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2019. Eraser: A benchmark to evaluate rationalized NLP models. arXiv:1911.03429.
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv:1702.08608.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.
Tyler McDonnell, Micahid Kutlu, Tamer Elsayed, and Matthew Lease. 2017. The many benefits of annotator rationales for relevance judgments. In Proceedings of the 26th International Joint Conference on Artificial Intelligence.
Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? collecting annotator rationales for relevance judgments. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing.
Gregory Plumb, Maruan Al-Shedivat, Angel Alexander Cabrera, Adam Perer, Eric Xing, and Ameet Talwalkar. 2020. Regularizing black-box models for improved interpretability. arXiv:1902.06787.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv:1703.03717.
Cansu Sen, Thomas Hartvigsen, Biao Yin, Xiangnan Kong, and Elke Rundensteiner. 2020. Human attention maps for text classification: Do humans and neural networks focus on the same words? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Jamar L. Sullivan Jr., Will Brackenbury, Andrew McNutt, Kevin Bryson, Kwam Byll, Yuxin Chen, Michael L. Littman, Chen-hao Tan, and Blase Ur. 2022. Open-source code release. GitHub repository. https://github.com/UChicagoSUPERgroup/rationales-naac122.
Chenhao Tan. 2021. On the diversity and limits of human explanations. arXiv:2106.11988.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for Fact Extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Sarah Wiegrefe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable NLP. arXiv:2102.12060.
Omar Zaidan and Jason Eisner. 2008. Modeling annotators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing.
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using "annotator rationales" to improve machine learning for text categorization. In Proceedings of Human Language Technologies 2007: The Conference of the North American chapter of the Association for Computational Linguistics.
A Wordings for Two-sided Conditions
Baseline, 2-sided: "Even though you felt the review was positive overall, select the most important words and phrases in the review that might instead suggest the review is negative."
Generalize, 2-sided: "Even though you felt the review was positive overall, select the most important words and phrases in the review that might instead suggest the review is negative. Select ONLY words and phrases that, if they appeared in other reviews, would also suggest negative sentiment."
Zaidan, 2-sided: "Even though you felt the review was positive overall, highlight words and phrases that would instead tell someone not to see the movie."
Table 3: Example instructions for opposite rationales ("positive" labels) in Phase 2's two-sided conditions.
B Additional Information Collected
Participants were asked to rate their confidence in their labeling and rationales on five-point Likert scales before proceeding to the next review. Following these tasks, participants completed a survey. We first asked Likert-scale and open-ended questions about the process of labeling a review as positive or negative: the difficulty of doing so, the participant's approach, what they focused on, and what could have been improved about the instructions and UI. We then asked about selecting rationales, the order in which participants selected words or phrases, and whether they chose "words or phrases that, if they appeared in other reviews, would also suggest the same positive/negative sentiment." Finally, we asked demographic questions. Appendix C contains our survey instrument.
C Survey Instrument
Almost done! We have a few final pages of questions about your experience across all 10 reviews.
C.0.1 Page 1
This first page asks about your experience determining whether a review was positive or negative.
1. Please respond to the following statement: Determining whether a review was positive or negative was difficult.
Strongly agree Agree Somewhat agree Neither agree or disagree Somewhat disagree Disagree Strongly disagree
2. Why?
3. Please briefly describe your process for determining whether a review was positive or negative.
4. What parts of the review did you focus on in determining whether a review was positive or negative?
5. What, if anything, could have been more clear about the instructions you were given for labeling movie reviews as positive or negative?
6. What could have been improved about the user interfaces for any parts of today's task (other than the surveys)? If you encountered any issues or errors, please include them here.
C.0.2 Page 2
This page was excluded for Labels Only participants, and questions were reworded to replace "words or phrases" with "words" for Zaidan, No Dragging, Words Only participants.
This page asks about your experience selecting the words or phrases you considered important.
1. Please respond to the following statement: Selecting the words or phrases I considered important was difficult.
Strongly agree Agree Somewhat agree Neither agree or disagree Somewhat disagree Disagree Strongly disagree
2. Why?
3. Please briefly describe your process for selecting the words or phrases you considered important.
4. What characteristics of a word or phrase made you select a particular word or phrase as important?
5. Think about a case where you considered selecting a particular word or phrase as important, but chose not to do so.
Please describe this situation, as well as why you ultimately chose not to select those words/phrases as important.
6. What, if anything, could have been more clear about the instructions you were given for selecting words or phrases as important?
C.0.3 Page 3
1. Please briefly describe the order in which you selected the words or phrases you considered important.
2. Please respond to the following statement: When selecting words or phrases that I considered important, I selected only words or phrases that, if they appeared in other reviews, would also suggest the same positive/negative sentiment.
3. Why?
4. What could have been improved about the user interfaces for any parts of today's task (other than the surveys)? If you encountered any issues or errors, please include them here.
C.0.4 Page 4
This final page of the survey asks about your demographics.
1. What is your age?
$18 - 24$  25-34 35-44 45-54
55-64  $\bigcirc$  65+  $\bigcirc$  Prefer not to say
2. What is your gender?
○ Woman ○ Man ○ Non-binary ○ Prefer to self-describe _______ ○ Prefer not to say
3. Please specify your ethnicity (select all that apply).
$\bigcirc$  Asian  $\bigcirc$  Black or African American  $\bigcirc$  Hispanic or Latinx  $\bigcirc$  Native American or Alaskan Native  $\bigcirc$  Native Hawaiian or Other Pacific Islander  $\bigcirc$  White or Caucasian  $\bigcirc$  Prefer to self-describe  $\bigcirc$  Prefer not to say
4. What is the highest degree or level of education you have completed?
No formal education Some high school, no diploma  $\bigcirc$  High school or equivalent  $\bigcirc$  Some college credit, no degree
Trade/technical/vocational training  $\bigcirc$  Associate's degree  $\bigcirc$  Bachelor's degree  $\bigcirc$  Master's degree  $\bigcirc$  Professional degree (JD, MD)  $\bigcirc$  Doctorate degree  $\bigcirc$  Prefer to self-describe  $\bigcirc$  Prefer not to say
5. Do you, or have you, held a degree or job related to computer science, IT, or similar?
Yes No I don't know Prefer not to say
6. Do you consider yourself to have expertise in computer security?  $\bigcirc$  Yes  $\bigcirc$  No  $\bigcirc$  I don't know  $\bigcirc$  Prefer not to say
7. Are you a native speaker of the English language?  $\bigcirc$  Yes  $\bigcirc$  No  $\bigcirc$  Prefer not to say
8. (Optional) Is there anything else you'd like to tell us before submitting the survey?
D Full Text of Reviews
Short-Negative: This has to be the worst movie I have seen. Madsen fans don't be drawn into this like I was. He is only in it for a maximum of five minutes. This movie is so bad that the only reason why you would watch it is if all the rest of the movies on earth as well as t.v. had been destroyed.
Short-Positive: I first saw this film on hbo around 1983 and I loved it! I scoured all of the auction web sites to buy the vhs copy. This is a very good suspense movie with a few twists that make it more interesting. I don't want to say too much else because if you ever get a chance to see it, you'll be glad I didn't say too much!
Long-Negative: Granting the budget and time constraints of serial production, BATMAN AND ROBIN nonetheless earns a place near the bottom of any "cliffhanger" list, utterly lacking the style, imagination, and atmosphere of its 1943 predecessor, BATMAN. The producer, Sam Katzman, was known as "King of the Quickies" and, like his director, Spencer Bennett, seemed more concerned with speed and efficiency than with generating excitement. (Unfortunately, this team also produced the two Superman serials, starring Kirk Alyn, with their tacky flying animation, canned music, and dull supporting players.) The opening of each chapter offers a taste of things to come: thoroughly inane titles ("Robin Rescues Batman," "Batman vs Wizard"), mechanical music droning on, and our