SlowGuess/NAACL_2022_OCR huggingface dataset:
For our experimental setup, we finetune a pretrained RoBERTa-base model plus a linear layer on the stance classification sub-task in the Perspectrum dataset. We follow the same experimental setup in Chen et al. (2019b). Our stance classifier achieves  $91.6\%$  accuracy on the test split of the Perspectrum dataset. Given a user query and a generated perspective as input to the stance classification model, we use the last hidden layer output from the backbone RoBERTa model as the representation of the perspective conditioned on the user query. For all perspectives generated, with their corresponding articles for a user query, we use k-means clustering with  $k = 3$  on the encoded representations. Conceptually, as we use a stance classification model as our encoder, the clusters resemble the different stances that responses can take towards the user query.
4.2.4 Identifying Evidence from Document
To corroborate the argument presented in the generated perspective, we extract up to three sentences from the article as supporting evidence. We formulate this as a ranking task. Given a generated perspective and all sentences from the corresponding document, we train a regression model to predict the probability of each sentence being evidence towards the perspective. We choose Evidence Sentences dataset (Ein-Dor et al., 2020), an English argument mining dataset that most closely resembles our task. We finetune a pretrained BERT-base model with 768 hidden size plus a linear layer to predict the probability of a sentence being evidence
4https://newspaper.readthedocs.io/en/ latest/
System	Organization	Comprehensiveness	Informativeness	Overall Preference
Google	92 (31%)	78 (26%)	135 (45%)	110 (37%)
Multi-Perspective Search	208 (69%)	222 (74%)	165 (55%)	190 (63%)
Table 2: Number of total votes from 10 participants comparing Google's search results to our prototype's on 30 different controversial queries. For each query, we ask participants to vote for the preferred system (i.e. Google vs. ours) from four criteria: (i) Organization of search results, (ii) Comprehensiveness of search results as potential answers to the query, (iii) Informativeness of search results, and (iv) Overall preference. See Appendix A for statistical significance testing and more details on the result.
towards the perspective. The training takes less than 30 minutes on a GeForce 1080Ti GPU and we conduct our experiments in a train/dev/test setting of  $80\% / 10\% / 10\%$ .
For each document and its perspective, we extract a ranked list of sentences with top- $k$  highest output probabilities from the model. As the training objective and data does not pose any constraint on whether the evidence supports or opposes the input perspective, we pipe it with a stance classifier (Chen et al., 2019b) to select only the sentences that support the perspective, and use a relevance classifier (Chen et al., 2019b) to rank them and present the three most relevant evidence paragraphs to the user. We conduct our evaluation on the test split of Evidence Sentences dataset, and our evidence mining system achieved a mean squared error of  $3.97\%$ .
5 A Study on User Information Needs
With the prototype search engine, we revisit the discussions in § 3. Our goal for the user study is to validate our hypothesis on user information need, and discover new insights from users' experience with an actual prototype model.
We collect and construct 30 queries of 6 controversial topics from 2020 United States presidential debate<sup>5</sup>. The 6 topics include COVID-19, Race & Justice, Immigration, Healthcare, Climate change, and Policy. We conduct a survey that compares the search results of these queries returned by Google search with the results returned by our prototype. The complete list of queries can be found in Appendix C.
We use a Google Programmable Search Engine to retrieve the same list of ten articles for each query. Our prototype system would further process the articles and present users with extracted
responses, evidence paragraph in different order, i.e. organized based on the predicted stance. For each controversial query, we show ten participants five different questions that compare the retrieved results from Google with those from our prototype. For each query and retrieved results from two systems, we ask the participants to choose which system (i) provides a better organization of results; (ii) shows a more comprehensive view of the topic; and (iii) is more informative in the context of the topic. We then ask for their the overall preferences between the system, and their explanations, expressed in free form text, for their choices.
To eliminate user interface (UI) as a confounding factor for the study, we construct a web interface for the survey that presents two sets of results with nearly identical UI. A screenshot of the survey interface is shown in Appendix D. We hire participants from Amazon Mechanical Turk to collect responses. Participants are located in the United States and are required to have master qualifications, i.e. top performers recognized by MTurk among all workers. We compensate $1 for answering 25 questions in total for five queries. The compensation rates are determined by estimating the average completion time for the survey. The participants are anonymized using the Amazon Mechanical Turk ID. The data collection protocol is determined exempt by an ethics review board.
5.2 Quantitative Analysis
The statistics from our survey responses are shown in Table 2.  $63\%$  of the responses indicate that they generally prefer our prototype to Google Search on the query.  $69\%$  and  $74\%$  of the responses state that our prototype has a better organization of the results and offers a more comprehensive view of the query. The noticeable advantages of these two aspects align with our assumptions. We believe that users prefer seeing search results in different clusters of perspectives instead of in a list ranked by
<sup>5</sup>https://2020election.procon.org/
relevance, and are convinced that such organization presents users a broader and more diverse set of views of the topic. On the other hand, only  $55\%$  of the responses indicate that our prototype presents more information on the topic. This meets our expectation as well given that the two systems are presenting the same set of articles for each query.
We perform hypothesis testing to validate that users prefer our prototype over Google Search on all of the aspects. We resample from the distribution of the results of each question by bootstrapping and compute the mean and standard deviation of each distribution. We repeat this process for 1000 times and use the average mean and standard deviation as the sample mean and sample standard deviation for each question. We then perform one-tailed z-test for each of the sample distribution. The p-values of all aspects except Informativeness are smaller 0.01, giving us  $99\%$  confidence to reject the null hypothesis that less than half of the participants prefer our prototype under these aspects, and the p-value of Informativeness is 0.043, showing a  $95\%$  confidence level in improvements over Google Search.
To understand the information needs of participants, we look into the explanations they provided for their general preferences on the system. We find that some participants prefer the search results provided by our prototype because they seek for direct answers to the query: The search is asking a question, and the results of System 1 (Google) offer only a collection of articles without any real answers. I like that System 2 (Our prototype) filters the results by organizing the "yes" articles together; the "no" articles together; and even the "maybe" articles together. This is a more convenient and helpful way for the user to find what they're really looking for. This finding confirms our hypothesis that some users expect to see direct answers when searching for a controversial query.
Some other participants prefer the way we present the information by showing bullet points instead of snippets: I prefer system 2 because each result has bullets of information which makes it easier for me to decide if it is worth clicking. Also, it is separated by results that answer yes/no to the question.. It shows that some users prefer to see an organized list of key information in an article and use it to decide whether or not the article is
worthwhile reading.
Despite our intent of providing a diverse set of perspectives to alleviate selection bias, a small portion of participants express concern that the automation provided by our prototype would in turn create biases comparing to Google's. I prefer system 1 because it gives a more balanced set of articles and gives the impression of being more unbiased, whereas System 2 has a statement right at the beginning that lends one to think of bias. Even when given the exactly same set of articles, a participant thinks that our prototype gives an impression of being more biased because our system explicitly states the perspective of an article, whereas Google Search results don't show that explicitly and require the users to look into the articles to find out. This finding shows that the information needs of users may vary, and explicitly displaying the stances of articles may have biased connotations.
The results from the user study offer validation our main hypothesis for the paper, that users typically wish to see direct and diverse responses when issuing a controversial, debate-worthy query to the search engine. At the same time, we also see concerns over potential model mistakes and intrinsic biases, which lead to decreased trust compared to existing document retrieval services.
Surrounding the three design factors introduced in § 3, we discuss a few existing trade-offs in our conceptual paradigm through the lens of technical challenges involved in developing a multiperspective search system.
Direct Response vs. Trust As the responses act as an extra layer of abstraction over the documents, the question on the trustworthiness of the machine-identified response naturally rises as a concern.
Such concerns involve two conflicting factors of research and design considerations. First, explanation-driven design of natural language understanding tools is beneficial, not only for improvements on model performance, but allows for more efficient design iteration of models. For this reason, explanation-driven task and data design has drawn more attention for information retrieval applications (Lamm et al., 2020).
However, from the user experience point of view, the harm of displaying the wrong explanation might overwhelm the benefit of increased model performance by incorporating explanation. In the case
of controversial queries, users are expected to be more sensitive to such trade-offs.
Diverse Response vs. Trust An unfortunate yet existing factor that influences user trust is confirmation bias. For example, Meppelink et al. (2019) discovers in the domain of health information retrieval, the user tend to only agree with results that conform with their prior belief. This raises a question about whether displaying a diverse set of perspectives can influence or correct the potential biases within users' prior belief.
Assessing such hypothesis is out of scope of this current project, and we will defer a study of such to the future. However, as empirical evidence show that such confirmation bias can be attributed in part to the users' access pattern and consumption of news information (Knobloch-Westerwick et al., 2015), we tend to believe that improving how search engines present controversial information will have a positive impact on alleviating confirmation bias among consumers of news information.
This study aims to understand the user information needs for controversial and debate-worthy queries to search engines. We argue for and examine the need of delivering direct and diverse response to such queries. To demonstrate the utility of the paradigm, we develop a prototype multiperspective search engine by synthesizing a suite of relevant NLP and IR tasks. We use the tool to facilitate a user study that confirms the benefit of serving direct and diverse responses to user queries. Through the positive and negative feedback we receive on the prototype, we discuss a few trade-off between technical design and user experience. We hope the findings in the paper offers insights, guidance and opportunities for future development on web document retrieval systems.
As the data and models used to develop the multiperspective search engine inevitably contain artifacts and inaccuracy, the returned results should only serve as speculation, as opposed to consolidated evidence for user queries, and should not be taken as granted without further reasoning and validation. The participants of the user study are informed of the potential risks of such before the annotation task.
The authors would like to thank Disha Jindal, Hegler Tissot for helpful conversations throughout the development of the prototype search engine, and the anonymous reviewers for their valuable feedback on the draft. In addition, the authors are grateful for the collaboration with 10 clouds to research, design and develop the front-end interface for the prototype. This work was supported in part by a Focused Award from Google, and in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong. 2009. Diversifying search results. In Proceedings of the second ACM international conference on web search and data mining, pages 5-14.
Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Rudy Rinott, Dan Gutfreund, and Noam Slonim. 2014. A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In Proceedings of the first workshop on argumentation mining, pages 64-68.
Roy Bar-Haim, Indrajit Bhattacharya, Francesco Dinuzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251-261, Valencia, Spain. Association for Computational Linguistics.
Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav, and Noam Slonim. 2020. From arguments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4029-4039.
Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. 2018. Give me more feedback: Annotating argument persuasiveness and related attributes
$^{6}$ https://10clouds.com/
in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621-631.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879.
Sihao Chen, Daniel Khashabi, Chris Callison-Burch, and Dan Roth. 2019a. Perspectroscope: A window to the world of diverse perspectives. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 129-134.
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019b. Seeing things from a different angle: discovering diverse perspectives about claims. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 542-557, Minneapolis, Minnesota. Association for Computational Linguistics.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461.
Charles LA Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the trec 2009 web track.
Charles LA Clarke, Maheedhar Kolla, Gordon V Cormack, Olga Vechtomova, Azin Ashkan, Stefan Böttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 659-666.
Van Dang and Bruce W Croft. 2013. Term level search result diversification. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 603-612.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2018. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179.
Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, et al. 2020. Corpus wide argument mining-a working solution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7683-7691.
Robert M Entman. 2007. Framing bias: Media in the distribution of power. Journal of communication, 57(1):163-173.
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758-764.
William Goffman. 1964. A searching procedure for information retrieval. Information Storage and Retrieval, 2(2):73-78.
Sha Hu, Zhicheng Dou, Xiaojie Wang, Tetsuya Sakai, and Ji-Rong Wen. 2015. Search result diversification based on hierarchical intents. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 63-72.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262.
Silvia Knobloch-Westerwick, Benjamin K Johnson, and Axel Westerwick. 2015. Confirmation bias in online searches: Impacts of selective exposure before an election on political attitude strength and shifts. Journal of Computer-Mediated Communication, 20(2):171-187.
Oleksandr Kolomiyets and Marie-Francine Moens. 2011. A survey on question answering technology from an information retrieval perspective. Information Sciences, 181(24):5412-5434.
Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2020. Qed: A framework and dataset for explanations in question answering.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Siyi Liu, Sihao Chen, Xander Uytendaele, and Dan Roth. 2021. MultiOpEd: A corpus of multiperspective news editorials. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4345-4361, Online. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Corine S Meppelink, Edith G Smit, Marieke L Fransen, and Nicola Diviani. 2019. "i was right about vaccination": Confirmation bias and health literacy in online health information seeking. Journal of health communication, 24(2):129-140.
Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. 2021. Rethinking search: making domain experts out of dilettantes. In ACM SIGIR Forum, volume 55, pages 1-27. ACM New York, NY, USA.
Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007. In google we trust: Users' decisions on rank, position, and relevance. Journal of computer-mediated communication, 12(3):801-823.
Jeff Pasternack and Dan Roth. 2013a. Latent credibility analysis. In Proceedings of the 22nd international conference on World Wide Web, pages 1009-1020.
Jeff Pasternack and Dan Roth. 2013b. Latent Credibility Analysis. In Proc. of the International World Wide Web Conference (WWW).
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Oya Y Rieger. 2009. Search engine use behavior of students and faculty: User perceptions and implications for future research. First Monday.
Daniel E Rose and Danny Levinson. 2004. Understanding user goals in web search. In Proceedings of the 13th international conference on World Wide Web, pages 13-19.