SlowGuess/NAACL_2022_OCR huggingface dataset:
Sihao Chen* Siyi Liu* Xander Uytendaele Yi Zhang William Bruno Dan Roth University of Pennsylvania
{sihaoc, siyiliu, xanderu, yizhang5, wwbruno, danroth}@cis.upenn.edu
Many users turn to document retrieval systems (e.g. search engines) to seek answers to controversial or open-ended questions. However, classical document retrieval systems fall short at delivering users a set of direct and diverse responses in such cases, which requires identifying responses within web documents in the context of the query, and aggregating the responses based on their different perspectives.
The goal of this work is to survey and study the user information needs for building a multi-perspective search engine of such. We examine the challenges of synthesizing such language understanding objectives with document retrieval, and study a new multiperspective document retrieval paradigm. We discuss and assess the inherent natural language understanding challenges one needs to address in order to achieve the goal. Following the design challenges and principles, we propose and evaluate a practical prototype pipeline system. We use the prototype system to conduct a user survey in order to assess the utility of our paradigm, as well as understanding the user information needs when issuing controversial and open-ended queries to a search engine.
In the past two decades, web search has developed as a ubiquitous way for users to retrieve web information. Typically, a web search query is driven by a specific intent, or information need, i.e. the users' desire to seek answers to an open-ended question (Shneiderman et al., 1997; Rose and Levinson, 2004). Among the various types of questions, one of the most challenging types, from the perspective of designing an information retrieval (IR) system, is debate-worthy, controversial questions, for which an objectively-true answer or consensus seems impractical to reach. Rather, explicit or implied re
User Query: Should We all Wear Masks?
Document #1 (Washington Post)
Opinion | You need to wear a mask. Here's how Apr 2, 2020 - And wearing masks is not just a good thing for those who are actively sick: Any one of us might be harboring this virus asymptotically and could transmit it ...
Document #2 (Cleveland Clinic)
Here's How (and Why) to Wear a Face Mask
Apr 7, 2020 - Wearing cloth face masks are not only an effort to protect yourself, but it will help to protect others from you if you happen to be infected...
Redundant Perspective
Document #N (ucsf.edu)
Still Confused About Masks? The Science...
May 4, 2020 - What evidence do we have that wearing a mask is effective in preventing COVID-19? There are several strands of evidence supporting the efficacy of masks, ...
Response does not answer the user query
Figure 1: Example retrieved articles with snippets from a search engine with respect to the query "Should we all wear masks" dated from April to May 2020, during which mask wearing was a more controversial topic. The retrieved articles contain redundant information, or often the extracted summary response does not answer the user query directly.
sponses with different perspectives, can be found within topically-related web documents.
Classical IR systems fall short at delivering responses with a diverse set of perspectives to the user query. Instead, they typically provide a ranked list of references to relevant but not necessarily trustworthy web documents, with limited justifications on what part(s) of the document could answer the user query (Metzler et al., 2021). We argue that a document retrieval system should ideally be equipped with the ability to extract or generate such responses from documents conditioned on the user query. Furthermore, a retrieval system should recognize the semantic difference of responses in cross-document settings, and in turn organize and deliver a set of documents from diverse perspectives. These objectives can inherently be formulated into a suite of natural language understanding tasks (Chen et al., 2019b; Lamm et al., 2020).
The goal of this paper is to discuss and validate
* Equal Contributions
Findings of the Association for Computational Linguistics: NAACL 2022, pages 294 - 310
the users' information need of direct and diverse responses when issuing controversial, debate-worthy queries to a search engine. We examine the ideas of formulating such objectives as natural language understanding tasks, and incorporate them in a novel document retrieval paradigm. Through the lens of key design principles of a task-specific search engine (Rieger, 2009), we first discuss the challenges in designing such a multi-perspective search engine. We introduce a paradigm that revolves around the abstraction of responses, i.e. a text segment that serves as direct answer to users, and perspectives, the argument(s) expressed in a response in the context of a given user query (Chen et al., 2019b).
To assess the utility of the proposed paradigm, we develop and demonstrate a prototype open-domain news search engine. Given a user query of controversial topics, the search engine retrieves news articles, identifies/generates responses within the articles along with evidence paragraphs, and categorizes the responses according to their perspectives differences. With the prototype, we conduct a user study in comparison to a general-purpose search engine. With the goal of understanding the user needs related to controversial queries, we discuss and analyze the advantages/disadvantages of our prototype through the user study responses. We then summarize and offer insights on the technical challenges and trade-off to cater to the specific user information needs with respect to controversial queries. We hope our findings will facilitate future research on this topic.
1. We study a document retrieval paradigm with the objective of delivering direct and diverse summary responses to user query. We outline a list of natural language understanding challenges to achieve the goal.
2. We demonstrate and evaluate a prototype multi-perspective search engine  $^{1,2}$  that retrieve, organize and summarize the perspective behind news articles.
3. We conduct a user study with our prototype against a popular search engine to assess the utility of our paradigm, as well as to understand the information needs by users with respect to controversial, debate-worthy queries.
The importance of retrieval result diversity has long been recognized by the information retrieval community (Goffman, 1964; Clarke et al., 2008). The problem has mostly been studied in the context of topic or entity ambiguity in the user query (Agrawal et al., 2009), and most solutions focus on resolving the ambiguity and uncovering the user intent, or hierarchy of the target topics (Sakai and Song, 2011; Dang and Croft, 2013; Hu et al., 2015; Wang et al., 2017). The most notable hypothesis for this line of work, which our study also investigates, is the importance of modeling the inter-dependencies among documents during retrieval. The TREC 2009 Web track's diversity task (Clarke et al., 2009) introduces an evaluation protocol where the relevance of next retrieval item is dependent on the previous result.
Our study takes a different angle from query ambiguity, and hypothesize that in the case of debate-worthy query, users wish to see direct responses from diversified perspectives in the retrieved documents (Metzler et al., 2021). Such objectives require recognizing the argumentation structure of a document (Stab and Gurevych, 2014; Ein-Dor et al., 2020), and more importantly comparing the semantics and implications of the arguments made cross documents (Chen et al., 2019b; Bar-Haim et al., 2020). Chen et al. (2019a) formulate a argument retrieval problem that conceptually resembles our formulation. In their task, a system is expected to return single-sentence arguments from different perspectives given a user query. Our study instead focuses on the more practical and challenging setting of document-level retrieval.
The task of question answering (QA) aims to identify direct answers to users questions expressed in natural languages from either close- or open-domain information resources (Kolomiyets and Moens, 2011). Naturally, document retrieval has been an essential step during answer candidate generation (Chen et al., 2017). Among various types of QA tasks, Machine reading comprehension (Rajpurkar et al., 2016; Khashabi et al., 2018) and open-domain QA (Yang et al., 2015; Joshi et al., 2017; Dunn et al., 2017) resemble our problem closely, as both involves identifying a concise response from large- to web-scale corpora. Under such settings, researchers have found benefit in jointly learning QA and retrieval tasks (Das et al., 2018). The key difference between most QA tasks
$^{1}$ Demo: wwwultipiperspectivesearch.com  $^{2}$ Code is released at www.github.com/CogComp/multi-persp-search-engine
and our problem is that objective correctness does not exist for most controversial queries, and so they cannot be responded with a single correct answer. Instead, there exist dimensions such as stance (BarHaim et al., 2017) and persuasiveness (Carlile et al., 2018) to measure the semantic difference between relevant and arguably equally valid arguments. To this end, recent researchers have attempted to extract or generate explanations alongside search results (Lamm et al., 2020).
3 Design Challenges
We start by discussing the needs of users when issuing controversial and debate-worthy queries to a search engine. Rieger (2009) introduces three key factors in building a task-specific search engine that caters to user's information needs. We introduce each factor below, and discuss the connections and implications to the case of controversial, debatingworthy user queries.
Satisfaction refers to overall success of the retrieved results in providing help to the user's specific search need. Here there are two levels of user satisfaction to consider. The first dimension is whether the retrieved results contain the answer that user needs. The second more challenging yet crucial dimension is whether search engine is able to highlight and display the answer (Metzler et al., 2021). In the context of controversial queries, identifying direct answers from retrieved documents seems essential, as it will greatly reduce the amount of readings user need to do before deriving the answer. However, automating the process comes with technical challenges, and such automation often ties closely with concerns of model biases, as we will further discuss in §6.
Search Intent refers to the category of webpages or answers that a user is interested to see when issuing a query. In the case of controversial queries, as users typically do not know a priori the potential aspects or perspectives that could lead to the answer in their context, organizing and highlighting a comprehensive set of results with diverse perspectives in a parallel fashion becomes important in this case (Chen et al., 2019a).
Trust is originally defined by Pan et al. (2007) as users' trust in a search engine's capability of retrieving relevant results to the query. However, as concerns over the veracity of web information grow (Thorne et al., 2018), user trust revolves more closely around the trustworthiness of the sources
and content (Pasternack and Roth, 2013a). For controversial queries, as multiple perspectives exist for every issue, whether each perspective is corroborated by evidence from trustworthy sources becomes the key to ensure user trust (Chen et al., 2019b).
In the following sections, we take a closer look into the specific problems we discussed for the three factors, and the challenges in formulating NLP tasks to address the problems.
3.1 Identifying Direct Responses to Query
From the classical view of information retrieval, identifying responses, or explanations (Lamm et al., 2020) is inherently a challenging problem for term-based retrieval models.
Our problem formulation for this part first takes inspiration from neural re-ranking (Xiong et al., 2017), where we rely on a classical yet more scalable retrieval function to identify document candidates that potentially contain the responses. The next part of the problem, response identification, closely resembles the task of question answering. Given a document  $d$  and a user query  $q$ , we want to identify one of more text segments  $\{s_i\} \in d$  such that each  $s \in \{s_i\}$  can be used as a direct response to  $q$ . The key difference in our formulation, as we discussed in the previous section, lies within the difficulty to define a single, valid response. Previous research in argumentation community has developed a few dimensions, such as argument stance, relevance, and persuasiveness to measure the validity of a text segment being a valid argument towards a user query. Using the corpus resources developed for such tasks (Bar-Haim et al., 2017; Ein-Dor et al., 2020), we can build models to score the text segments within a document along dimensions; aggregate the dimensions and retrieve a ranked list of the valid responses.
One particular challenge for some genres of web documents, such as opinion pieces (Liu et al., 2021), is that the responses are usually implied instead of explicitly written. To address this, an alternative strategy is to use a text generation model to generate the responses, though it introduces additional challenge, such as evaluating the integrity of the text to the original article (Maynez et al., 2020).
Figure 2: An conceptual illustration of our multi-perspective document retrieval paradigm, as discussed in § 3.4. Instead of document, our retrieval pipeline revolves around direct responses within articles. The responses are extracted or generated from individual candidate documents, and then they are re-ranked, de-duplicated and categorized based on the implied perspectives (e.g. different colors in the figure) across documents. Finally, the responses from diverse perspectives, supporting evidence in context and relation between results are visualized in a parallel fashion.
3.2 Recognizing Different Perspectives in Responses
One key prerequisite for recognizing the perspectives, or semantic implications of response is to break the independence assumption of retrieved documents. When learning about controversial topic, a critical reader would naturally compare responses from various perspectives. We follow this intuition and argue for the need of a module in the document retrieval pipeline to recognize whether two responses share similar or different perspectives in the context of the query.
Given a user query  $q$  and a retrieved list of document candidates  $\{d_i\}$ , each with zero or more response text segments  $\{s_{d_i}\}$ . We want to build a similarity function  $Sim(s_{d_i}, s_{d_j} | q, d_i, d_j)$ . Compared to traditional sentence-level semantic similarity tasks (Ganitkevitch et al., 2013; Cer et al., 2017), the key difference is the conditional nature of similarity here. For example, in the context of "Should I ride bike or take taxi to work", "Riding bike costs less" and "Taxis cost more" are semantically equivalent, despite the fact that two statements have different predicate arguments.
In practice, apart from the difficulty in developing annotations and resources for such tasks, one major bottleneck in the development of such a context-aware semantic similarity function is that text segments are typically not standalone without document as context. For example, text segments may use pronoun references instead of named or nominal mentions exclusively. To address such problems, Choi et al. (2021) proposes a text generation task to de-contextualize an input segment using its document context.
3.3 Trustworthiness
User trust in the results from a search engine is largely governed by the empirical authoritativeness of their information sources and content (Pasternack and Roth, 2013b; Zhang et al., 2019). From the perspective of information sources, a source's trustworthiness is typically determined by the process it generates and publishes the content. For example, a news source is generally considered credible if all the published content go through fact-checking, and statement biases within the content are carefully removed (Entman, 2007).
However, such information is typically neither transparent nor available from the users perspective. To increase access to information about trustworthiness of information sources, we argue for the need of two alternative strategies. First, meta information about the sources helps user recognize the type of content they are seeing, and how to assess the credibility of the content accordingly. For example, it takes drastically different approach assessing the trust value of a official guidance on COVID-19 published by a government entity, versus a opinion piece from a major news outlet criticizing the guidance. Second, for each identified or generated responses in the context of the user query, we corroborate it with supporting evidence extracted from the document (Aharoni et al., 2014). These two strategies work together to provide user a potential way to visually assess the credibility of the content they see.
3.4 Putting it All Together: A Pipeline Approach
Figure 2 illustrates the resulting conceptual paradigm for a multi-perspective search engine. The paradigm centers around the abstraction of a
Figure 3: An example screenshot of results returned by our multi-perspective search prototype. Clicking on the Perspective 1 on the left page will direct to the page on the right. Please refer to our video (https://www.youtube.com/watch?v=VD1QubNiRR4) for a recorded live demo with the prototype search engine. See Appendix D for a more complete set of examples.
response to a query, represented by a text segment from a retrieved document. First, a set of document candidates can be retrieved using a general document retrieval system. A response identification module (§ 3.1) extracts and validates responses from individual documents. A cross-document module (§ 3.2) works to recognize the perspective differences between responses across documents. After corroborating evidences for each response (§ 3.3), the responses categorized by their perspectives, evidence from document context, along with meta information about the sources are displayed in parallel to the user.
4 A Prototype of Multi-Perspective Search Engine
Following the design principles outlined in the previous section, we implement a prototype search engine. Our motivation of the prototype is twofold. First, we want to understand the utility of our conceptual paradigm from the user perspective. Second, this prototype serves as a platform to study the user information needs with respect to controversial and debate-worthy queries. We defer the discussion of the study to  $\S 5$ .
Our prototype consists of four components. First, given a user query, we use Google Search API to retrieve the list of top- $k$  relevant articles from the web. For each article, we use a query-focused summarization model (Liu et al., 2021) to generate the key response to the query implied by the document. We employ a stance classification model (Chen et al.,
2019b) to categorize the responses based on their stance (e.g. pro/con) to the query. Next, we use an evidence extraction model (Ein-Dor et al., 2020) to extract sentences from each article that support the generated perspective of this article. Last, we cluster the articles according to their conditional semantic similarity on the query, and present them to the users with their lists of evidence. Details of each of the components are described in  $\S 4.2$
An example search result is shown in Figure 3. The user searches a query of Should wearing masks be mandatory?, and our system presents the searched articles to the users in two clusters of perspectives, one supporting the query, and one opposing the query. In each cluster, our system displays articles that share this perspective, along with their evidence retrieved from the articles.
4.2 Components
We now describe the four core components in our pipeline construction. As user interface design is not the focus of the study, we defer the discussion and more illustrative examples of our prototype website to the Appendix.
4.2.1 Candidate Document Retrieval
We use Google Programmable Search Engine and Custom Search JSON API<sup>3</sup> to facilitate our retrieval process. We configure our search engine to include a manually curated list of trustworthy websites to search from (See Table 3 in Appendix B for the complete list of sources), and use the API
<sup>3</sup>https://developers.google.com/custom-search/v1/overview
Task + Dataset	System	Metric	Score
Perspective Summarization (Liu et al., 2021)	BART	ROUGE2	11.34
+REL	11.51
+STANCE	11.53
+REL & STANCE	11.92
Evidence Mining (Ein-Dor et al., 2020)	Linear Regression SVM	MSE	6.39
GRADIENTBOOST	11.71
BERTbase	7.74
3.97
Stance Classification (Chen et al., 2019b)	BERTbase	F1	70.8
BERTlarge	79.6
ROBERTAbase	85.3
ROBERTALarge	91.6
Table 1: Evaluation results of the baselines and models on the three components. All the numbers are in percentage. Top performing systems are in bold. The setup of different baselines in Perspective Summarization is described more in detailed in (Liu et al., 2021).
to retrieve the webpages returned by our search engine. We then use Newspaper  $3\mathrm{k}^4$  to clean and extract the article body.
4.2.2 Query-Focused Summarization of A Document's Perspective
Given a user query and an article, we use a query-focused summarization system to extract or generate a single sentence thesis statement for the article (Liu et al., 2021). The generated summary is intended to serve as a direct response to the user query. Conceptually, the generated summary should reflect the same argument that expresses the consistent stance as the article itself.
We formulate the above objectives in a multi-task learning framework, as proposed by Liu et al. (2021). We start from a pretrained BART model (Lewis et al., 2020), and finetune on the MultiOpEd dataset, an English news editorials corpus for query-focused summarization (Liu et al., 2021). Following Chen et al. (2019b), we use two ancillary tasks to regularize the query-focused summarization model. Specifically, we pose the relevance and stance of the user query in the context of the article as two binary classification ancillary tasks. During the BART summarization model training, we use a relevance and a stance classification model as teacher models to score the decoded output, and train two linear classification heads on top of the BART model to mimic the predictions from the teacher models. For more details on the model
training, please refer to Liu et al. (2021).
We conduct our evaluation in-domain on the test split of MultiOpEd dataset. The best performing perspective summarization model achieves 11.92 Rouge-2 score on the test split of the dataset. We report the selection and performance of the baseline models in Table 1.
4.2.3 Query-Conditioned Similarity Between Perspectives
Next, our system is expected to group articles with similar perspectives within the context of the query. Following Chen et al. (2019b), we conceptually treat the problem as conditional similarity between two perspectives. Given a query, and two single-sentence perspective summaries, we learn a conditional similarity function for whether the two perspectives present similar arguments in the context of the query.