SlowGuess/NAACL_2021_OCR huggingface dataset:
MULTIOPED:
A Corpus of Multi-Perspective News Editorials
Siyi Liu Sihao Chen Xander Uysettendaele Dan Roth
{siyiliu, sihaoc, xanderu, danroth}@seas.upenn.edu
We propose MULTIOPED<sup>1</sup>, an open-domain news editorial corpus that supports various tasks pertaining to the argumentation structure in news editorials, focusing on automatic perspective discovery. News editorial is a genre of persuasive text, where the argumentation structure is usually implicit. However, the arguments presented in an editorial typically center around a concise, focused thesis, which we refer to as their perspective. MULTIOPED aims at supporting the study of multiple tasks relevant to automatic perspective discovery, where a system is expected to produce a single-sentence thesis statement summarizing the arguments presented. We argue that identifying and abstracting such natural language perspectives from editorials is a crucial step toward studying the implicit argumentation structure in news editorials.
We first discuss the challenges and define a few conceptual tasks towards our goal. To demonstrate the utility of MULTIOPED and the induced tasks, we study the problem of perspective summarization in a multi-task learning setting, as a case study. We show that, with the induced tasks as auxiliary tasks, we can improve the quality of the perspective summary generated. We hope that MULTIOPED will be a useful resource for future studies on argumentation in the news editorial domain.
News editorial is a form of persuasive text that conveys consensus opinion on a controversial topic from the editors of a newspaper. Much like an argumentative essay, a news editorial centers around a thesis, which represents the authors' perspective on the topic. Usually, a news editorial argues in favor of the authors' stance on the topic, and is substantiated by extensive factual evidence.
Figure 1: Structure of MULTIOPED. For each query on a controversial topic, two (rather long) news editorials respond to the query from different point-of-views. Each editorial comes with a single paragraph abstract plus a one-sentence perspective, that abstractively summarizes the editorial's key argument in the context of the query. The two resulting perspectives serve as responses with opposite stance to the query.
As news editorials function as professionally produced written discourse for conveying media attitude and guidance, they have traditionally been studied by the community as a rich resource for many argumentation-related tasks. (Wilson and Wiebe, 2003; Yu and Hatzivassiloglou, 2003; Bal and Saint-Dizier, 2009).
This work targets the problem of developing computational methods to identify and comparatively analyze the authors' perspectives and supporting arguments behind news editorials. One challenge to studying the argumentation structure in news editorials is that its elements are rarely expressed explicitly (El Baff et al., 2018). For example, Figure 1 shows two news editorials holding opposite views on whether a lockdown should continue. However, neither of them present their key perspectives explicitly. Instead, the perspective is conveyed through subtle rhetoric strategies to either affirm or challenge the readers' stance from prior belief on the topic, as a study by El Baff
<sup>1</sup>The authors would like to thank Daniel Ravner, the CEO of www.theperspective.com, for granting access to data from the site for academic research.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4345-4361 June 6-11, 2021. ©2021 Association for Computational Linguistics
et al. (2018) discovers. As Figure 1 shows, the statement "The lock down should stop" concisely summarizes the perspective expressed in the article on the left. We refer to such statements as "perspectives" throughout the paper. The ability to abstractively summarize the perspectives from the editorial would allow us to understand multiple topic-aligned editorials in context and reason about their inter-editorial argumentation structure.
To facilitate research along the line, we collect data from THEPERSPECTIVE website, and construct MULTIOPED, an open-domain English news editorial corpus that supports various tasks pertaining to the argumentation structure in news editorials, focusing on automatic perspective discovery (Chen et al., 2019). The structure of the data is shown in Figure 1. For each of the 1,397 natural language query on a different topic in our dataset, it features two (rather long) news editorials. Each editorial features a single-sentence perspective, which is abstractively summarized from the editorial by human experts. A short abstract then highlights the details in the editorial that support the perspective. The perspectives of the two editorials represent responses of opposite stances towards the query.
Naturally, the structure of the dataset induces a range of important argumentation-related natural language understanding tasks. For instance, the presence of the summary perspective allows for stance classification (Hasan and Ng, 2013) with respect to the query, which arguably is more tangible than inferring the stance from the entire editorial. Another example task is the conditional generation of the perspective from the abstract/editorial, which relates to the widely studied task of argument generation (Hua and Wang, 2018; Alshomary et al., 2020). We defer the more detailed description of the induced tasks to Section 3.
One key advantage of MULTIOPED that is absent from earlier datasets is that a large number of argumentation-related tasks can be studied jointly using a single high quality corpus. To demonstrate this benefit and the utility of the MULTIOPED dataset  $^{3}$  along with its induced tasks, we study the problem of perspective summarization in a multitask learning setting. We employ perspective relevance and stance classifications as two auxiliary tasks to the summarization objective. Our emperi
cal and human analysis on the generated summaries show that the multi-task learning setting improves the generated perspectives in terms of the argument quality and stance consistency.
In summary, our contributions in this work are three-fold. First, we propose a conceptual framework for identifying and abstracting the perspectives and the corresponding argumentation structure in news editorials, and define a set of tasks necessary for achieving this goal. Second, we propose the MULTIOPED dataset, a news editorial dataset that induces multiple argumentation-related tasks. Third, we demonstrate the utility of our multi-purpose dataset and induced tasks, by using the perspective summarization task as a case study. We include the induced tasks as auxiliary objectives in multi-task learning setting, and demonstrate their effectiveness to perspective summarization.
2 Design Principles
Our goal of perspective discovery follows similar definition proposed by Chen et al. (2019), and is closely related to a widely studied area of argumentation mining, i.e. identifying the argumentation structure within persuasive text (Stab and Gurevych, 2014b; Kiesel et al., 2015). However, most studies in this domain focus on extractive methods, which becomes less applicable to our study. As the arguments are usually presented in an subtle and implicit way in news editorials, we instead focus on the generation methods for the perspectives. This closely resembles the argument conclusion generation task (Alshomary et al., 2020). One key distinction here is the presence of query to provide topic guidance during the perspective generation.
Compared to other conditional text generation tasks, perspective generation subjects to a few more constraints with respect to the argumentation structure. For example, the perspective must constitute the same stance (Hasan and Ng, 2013) as the editorial towards the query. On the other hand, while the editorial may cover content not directly related to the query, the generated perspective must present a relevant argument in the query's context. Such structural constraints can be studied in the format of classification problems. And being able to study such problems along side the perspective summarization task on one high-quality corpus is important in our case, as it opens up the probability of modeling the tasks jointly. We show the benefit of
$^{2}$ https://www.theperspective.com/perspectives/ $^{3}$ Our code and data is available at http://cogcomp.org/page/publication_view/935
Dataset	Source	Open Domain	Cross Article	Abstractive
ARAU CARIADB (Reed et al., 2008)	News Ed.	✓	×	×
(Stab and Gurevych, 2014a)	Essay	✓	×	×
(Eckle-Kohler et al., 2015)	News	✓	×	×
(Hua and Wang, 2018)	Reddit/Wiki.	Politics only	×	✓
PERSPECTRUM (Chen et al., 2019)	Debate	✓	×	✓
MULTIOPED	News Ed.	✓	✓	✓
doing so by presenting a case study in section 5.
As the query provides topic guidance, it allows for the study of the topic-aligned pairs of editorials which presents counter-arguments to each other. Such property is absent from notable datasets of similar purposes to ours, as shown in Table 1. ARAUCARIADB (Reed et al., 2008) is the first effort to provide large-scale annotations of dense argumentation structure within individual news editorials. Stab and Gurevych (2014a); Eckle-Kohler et al. (2015) provide resources for extractive argumentation structure in persuasive essays and news articles, respectively. Later works (Hua and Wang, 2018; Chen et al., 2019) focus on the abstractive generation or identification of arguments from web corpora. All of these datasets focus on studies of argumentation structure within individual document. Instead, our proposed dataset presents the opportunity to study the cross-document argumentation structure.
Table 1: A comparison across datasets with similar purpose to MULTIOPED. We compare the datasets along three dimensions. Open Domain: whether the dataset features a wide variety of topics. Cross Article: whether the argumentation structure between documents are annotated. Abstractive: whether the elements in argumentation structure is abstractive or extractive.
Instance	Size	Avg. Len.	Min	Max
Query	1,397	7.4	3	15
Perspective	2,794	6.1	2	10
Abstract	2,794	101.9	47	160
Article	2,584	918.6	74	7,608
Table 2: Statistics of the MULTIOPED dataset. Size represents the number of each valid instance, Avg. Len. indicates the average length of each instance in terms of the number of tokens split by space, and Min and Max represent the number of tokens of the shortest and longest texts of each instance.
3 MULTIOPED and Induced Tasks
Following the design principles outlined in the previous section, we propose a topic-aligned English news editorial corpus, MULTIOPED. The structure
of an example instance in MULTIOPED is shown in Figure 1. To clarify our description of the dataset, we use the following notations. Let  $q$  be a query about a controversial topic. Each  $q$  in the dataset is paired with two editorials  $e_{pro}$  and  $e_{con}$ , that constitute supporting and opposing stances to the query  $q$  respectively. Each editorial is abstracted into and a single-sentence perspective  $p$ , which provides a high-level summarization of the key argument presented in the editorial. The premises, or relevant details to support the perspective, forms the abstract  $a$ .
Naturally, the relation between these elements induces several tasks, most of which encompasses similar definitions to existing argumentation-related tasks. We define and describe the tasks and their connection to our end goal of perspective discovery below.
1. Generating an Abstract: Given an editorial  $e$ , a system is expected to identify and summarize the relevant arguments into an abstract paragraph  $a$  to the context provided by the query  $q$ . This is closely related to the task of argument synthesis (El Baff et al., 2019; Hua et al., 2019). We set aside this problem in our case study in section 5, and use the abstract provided by the dataset.
2. Perspective Summarization: Given the generated abstract  $a$  and the query  $q$ , a system is expected to generate the perspective  $p$ , a concise summary of the arguments presented in  $a$ . Conceptually, this problem resembles the task of argument conclusion generation (Alshomary et al., 2020). We adopt a slightly different setting where the target topic is expressed in the form of a natural language query.
3. Stance Classification: Our goal is to infer the
Figure 2: Perspective Summarization: Generate an single sentence argument that represents the key perspective expressed in the news editorial.
editorials  $e$  's stance towards a query  $q$  . The generated perspective  $p$  from editorial  $e$  allows us to focus on a simpler task definition of classifying the stance of the perspective to the query  $q$  (Hasan and Ng, 2013; Bar-Haim et al., 2017).
Figure 3: Stance Classification: Decide if the perspective supports or opposes the query.
4. Assessing the Relevance of Perspective: We want to measure the validity of the perspective by assessing whether the perspective presents a relevant argument towards the query (Chen et al., 2019; Ein-Dor et al., 2020). This can be formulated as a classification problem with the query  $q$  and a perspective  $p$  as inputs, as we show in section 5.
Figure 4: Relevance classification: Decide if the perspective is relevant to the query.
4.1 Data Collection
We extract the query, editorial article pairs, abstract paragraph pairs, along with their perspective summaries from THEPERSPECTIVE<sup>4</sup> website. The website presents controversial topics in the form of queries. For each query, two related editorial articles with opposing views from different sources
are selected by the writers from the website. The writers create a concise one-sentence summary of each article as the response to the query, and an abstract paragraph to summarize the relevant arguments from the article. An example structure of the data is shown in Figure.1.
We use BEAUTIFULSOUP  $^{5}$  and NEWSPAPER3K  $^{6}$  to extract and clean the perspective and news data.
4.2 Crowdsource Verification & Annotation
To verify the structure from the website, and collect additional annotations, such as stance of the perspectives, we conduct a few annotation experiments with Amazon Mechanical Turk<sup>7</sup>. For all of our annotation experiments, we require the workers to be located in the United States, as the controversial topics covered by the website are most applicable in the U.S. context. We also require the workers to have masters qualifications (i.e. Top performers recognized by MTurk among all workers). We compensate the workers  $0.75,$ 1.00 and $1.25 per 10 queries for the implicit reference resolution, topic annotation, and stance annotation tasks respectively. The compensation rates are determined by estimating the average completion time for each annotation experiments. Example screenshots of our annotation interface and more detailed annotation guidelines can be found in Appendix B.
4.2.1 Stance Annotation
In our dataset, each query is presented with two perspectives with opposite stance to the query. However, the raw data that we collected does not specify the stance of each perspective individually.
We ask two expert annotators label whether each perspective is offering a supporting or opposing view with respect to its query. The two experts discuss and adjudicate their decisions. We then ask on average three crowdsourced workers per instance to verify the annotations.
From the annotations collected by experts, we find that 30 out of 1,397 queries do not constitute a clear stance. Such queries are typically "open-ended" questions which cannot be responded with a yes or no answer, i.e. why or what questions. We leave these instances unlabeled and exclude them from the next verification step.
4https://www.theperspective.com/ perspectives/
$^{5}$ https://www.cummy.com/software/BeautifulSoup/bs4/doc/  
 $^{6}$ https://newspaper.readthedocs.io/  
 $^{7}$ https://www.mturk.com/
To assess the quality of stance labels created, we randomly sample 500 perspectives, and ask three MTurk workers per instance to verify stance labels. We computed the inter-rater agreement fleiss'  $\kappa = 0.81$  among workers, and the agreement between majority decision from works and the expert's adjudicated annotations is cohen's  $\kappa = 0.92$ . We describe how we measure the two types of agreements respectively in Appendix A.3.
4.2.2 Implicit Reference Resolution in Perspectives
Some of the perspectives in our dataset have implicit references to certain subjects in the query. For instance, for a query "Is Trump Right To Criticize Mail-In Voting?", and a perspective "It's far too risky for an election", the word "It" in the perspective refers to "Mail-in Voting" in the query. As we assume that a perspective should present a complete, valid argument on itself, we decide to replace such implicit reference in a perspective with the correct referent in the query. For example, the corrected perspective in the previous example would become "Mail-in voting is far too risky for an election".
We ask one expert annotator to identify implicit references and make modifications for every perspective in the dataset. In total, 1,301 out of 2,794 perspectives are identified and corrected by the expert annotator. We ask three Turkers to verify that the modifications do not introduce any grammatical error or change the original meaning. We randomly sample 500 modified perspectives and present Turkers with the question of "Will this modification change the original meaning or introduce grammar error?" The percentage of majority answers being "No" is  $84\%$ . We include both changed and original versions of the perspectives in our datasets.
4.2.3 Topic Annotations
We create 9 topic labels according to the categorization from THEPERSPECTIVE website and major news outlets. We then as ask three MTurk workers to assign one of the 9 topic labels to each query. We regard the majority answer by the Turkers as the annotation for its topic category. In cases where all three annotators choose different categories (43 cases out of all 1397 queries), we label it as other topics. We show the distribution of topic categories in Figure 5. The inter-agreement among three annotators for this 9-class classification task is  $\kappa = 0.65$
Figure 5: Topic distribution of the 1397 queries in MULTIOPED. Note that the two editorials for each query fall in the same topic category as the query.
4.3 Dataset Statistics
MULTIOPED consists of 1,397 queries about different news topics. Each query is presented with two perspectives, two abstracts and two linked news editorials. Despite a few stale urls and invalid redirections, we manage to extract the text for 2,584 news editorials. More detailed statistics are reported in Table 2.
5 Case Study: Multi-task Learning for Perspective Summarization
5.1 Multi-Task Framework
To demonstrate the benefits of modeling the induced tasks on the argumentation structure, we present a case study on the task of perspective summarization. Given a query and an abstract from the related editorial, a system is expected to produce a concise and fluent summary perspective for the editorial. In addition, the generated perspective ideally should satisfy a few structural constraints with respect to the query. For instance, the generated perspective must constitute the same stance as the editorial towards the query. Also the perspective should be relevant in the context of the query. The two requirements resemble the perspective stance and relevance classification" tasks defined in section 3 respectively.
Motivated by this, we study the two tasks together with perspective summarization in a multitask learning framework. We choose BART (Lewis et al., 2020) as our base summarization model. BART is a pretrained auto-regressive transformer (Vaswani et al., 2017) encoder-decoder model, that have been proven effective in conditional text generation and other NLP tasks.
Model	ROUGE1	ROUGE2	ROUGE L	BERTSCORE	REL. %	STANCE %
BART	28.24	11.34	26.96	88.67	91.91	72.32
+ Rel	28.35	11.51	27.12	88.69	92.98	72.68
+ Stance	28.19	11.53	26.93	88.75	91.25	73.39
+ Rel & Stance	29.18	11.92	27.94	88.74	94.64	74.29
Table 3: Results of our multitask perspective summarization models. We compare to BART as a baseline, and experiment with different combinations of the auxiliary tasks. We report the  $F_{1}$  score under ROUGE{1,2,L} and BERTSCORE metrics, as well as the percentage of summaries with the correct relevance and stance label, as predicted by our pretrained classification models respectively. See Appendix A for training details and hyperparameters settings.
Model	RANK	%1ST	REL.	STANCE
BART	2.09	49.50	77.00	70.50
+ Rel	1.74	60.50	87.00	70.50
+ Stance	1.78	58.50	83.00	79.50
+ R & S	1.76	59.00	82.00	69.00
Table 4: Human Evaluations results. "RANK" shows a model's averaged rank judged by the raters (1 = best, 4 = worst) "1ST" represents the percentage of generated summaries from one model that are ranked the best. We allow ties in the ranking. REL. and STANCE are the percentages of generated summaries that are relevant to and have the correct stance with respect to the query.
We start with a pretrained BART base model with  $139M$  parameters, and finetune the model to output the target perspective given the query and abstract concatenated as input. In addition, we put two separate linear layers over the pooled embeddings of the last decoder layer, and predict the relevance and stance labels of the generated summary respectively. The two tasks and the perspective summarization are learned jointly, and share the underlying model parameters from BART.
One obvious challenge in the setup is that we do not have access to the ground truth stance and relevance labels for the generated summaries during training. To address this, we adopt similar strategies as in knowledge distillation (Hinton et al., 2015). We first train two separate BERT (Devlin et al., 2019) classifiers as the teacher models for stance and relevance classificaion respectively. Due to the size limit of our dataset, we pretrain both models on the PERSPECTRUM dataset (Chen et al., 2019), which contain over 7,000 instances of training data, with similar formats and definition to our (query, perspective) pairs. We further fine-tune the models on our training set. When measured against our test set, the relevance and
stance models achieve binary accuracy of  $92\%$  and  $75\%$  respectively.
During the perspective summarization model training, we use the pre-trained BERT models for relevance and stance classification to predict labels for each generated summary. We expect the BART plus linear layers to "mimic" the predictions made by the two pretrained BERT models respectively. Specifically:
$$
\mathcal {H} _ {\mathrm {Q}} = \operatorname {E O S} (\mathcal {D} _ {\mathrm {B A R T}} (\mathcal {E} _ {\mathrm {B A R T}} (\mathcal {Q})))
$$
$$
\mathcal {H} _ {\mathrm {A}} = \operatorname {E O S} \left(\mathcal {D} _ {\mathrm {B A R T}} \left(\mathcal {E} _ {\mathrm {B A R T}} (\mathcal {A})\right)\right)
$$
We feed the query and the abstract separately through the BART encoder  $(\mathcal{E}_{\mathrm{BART}})$  and decoder  $(\mathcal{D}_{\mathrm{BART}})$ . We get their hidden representations  $\mathcal{H}_{\mathrm{Q}}$  and  $\mathcal{H}_{\mathrm{A}}$  as the embedding of the end-of-sentence  $(< / \mathrm{s} > )$  token from the decoder. We then concatenate  $\mathcal{H}_{\mathrm{Q}}$  and  $\mathcal{H}_{\mathrm{A}}$ , and feed the concatenation to the two linear layers. Finally, a softmax layer is applied to get stance/relevance predictions  $\tilde{y}_{rel}$  and  $\tilde{y}_{stance}$ .
$$
\tilde {y} _ {\text {s t a n c e}} = \operatorname {S O F T M A X} \left(W _ {s} ^ {T} \left[ \mathcal {H} _ {\mathrm {Q}}, \mathcal {H} _ {\mathrm {A}} \right]\right)
$$
$$
\tilde {y} _ {r e l} = \operatorname {S O F T M A X} \left(W _ {r} ^ {T} \left[ \mathcal {H} _ {\mathrm {Q}}, \mathcal {H} _ {\mathrm {A}} \right]\right)
$$