SlowGuess/NAACL_2021_OCR huggingface dataset:
Next, We feed the query and the generated summary to the two pretrained BERT classification models to get the soft stance and relevance labels  $y_{rel}$  and  $y_{stance}$ . We use two mean square error (MSE) loss terms to measure the discrepancy between the BART predictions and the soft labels.
$$
\mathcal {L} _ {\text {R E L}} = \text {M S E L O S S} \left(y _ {r e l}, \tilde {y} _ {r e l}\right)
$$
$$
\mathcal {L} _ {\text {S T A N C E}} = \text {M S E L O S S} \left(y _ {\text {s t a n c e}}, \tilde {y} _ {\text {s t a n c e}}\right)
$$
We combine  $\mathcal{L}_{\mathrm{REL}}$  and  $\mathcal{L}_{\mathrm{STANCE}}$  with the summarization objective,  $\mathcal{L}_{\mathrm{SUM}}$ , which is the negative log-likelihood loss between generated and target perspective. The auxiliary losses  $\mathcal{L}_{\mathrm{REL}}$  and  $\mathcal{L}_{\mathrm{STANCE}}$
are weighted by tunable hyperparameters  $\alpha_{1}$  and  $\alpha_{2}$  respectively.
$$
\mathcal {L} = \mathcal {L} _ {\text {S U M}} + \alpha_ {1} \cdot \mathcal {L} _ {\text {R E L}} + \alpha_ {2} \cdot \mathcal {L} _ {\text {S T A N C E}}
$$
5.2.1 Automatic Evaluations
Table 3 shows our evaluation results of our multi-task model with different combinations of auxiliary tasks. The reported results are averaged over three trained models with different random initialization. We first evaluate the generated perspective summaries against the target perspective with ROUGE (Lin, 2004) and BERTSCORE (Zhang et al., 2020) metrics. We observe that relevance and stance auxiliary tasks both increase the ROUGE and BERTSCORE, and combining the two objectives yields the best performance under the ROUGE metrics.
To empirically verify whether the perspectives generated by our multi-task model are improved in terms of the relevance and stance correctness, we again use the two pretrained BERT classifiers to measure the percentage of generated summary with correct relevance and stance label. The results potentially suggest that by "mimicking" the predictions made by the two pretrained classifiers, our multi-task framework is able to generate summaries with higher quality along the two dimensions.
5.3 Human Evaluations
We randomly sampled 100 instances of abstracts with query from the test set, and ask two human raters to judge the quality of perspectives generated by the four systems. For the four summaries generated from an abstract by the different systems, we shuffle their order and ask the raters to rank each summary by the overall quality, with four criteria considered (1) Fluency (2) Grammatical Correctness (3) Faithfulness to the arguments offered in the original abstract (4) Salience. We allow ties among different summaries. We report their averaged ranks and the number of times a system is ranked first place in Table 4. The results are the averaged scores between the two annotators, and the level of agreement between them for this 4-class ranking task is  $\kappa = 0.35$ .
For each summary, we ask the raters to annotate whether it (1) represents a relevant argument to the query (2) constitutes the correct stance as the target stance label. The kappa agreement between the two
Query	Should trump accept democrats’
gov’t spending bill?
BART	A shutdown is the best
deal he can get.
+ REL	Trump should accept
the GOP budget.
Gold	This deal is the most
achievable compromise.
Table 5: An example where relevance auxiliary task helps the perspective summarization process
Query	Is apple's iphone x technology any good?
BART	Apple's new iPhone X offers many great opportunities
+ STANCE	Apple's new face-recognition technology raises many ethical issues
Gold	Apple's new Iphone X raises many security concerns
Table 6: An example where stance auxiliary task helps the perspective summarization process
raters for these two tasks are 0.54 and 0.70, respectively. We show the human evaluation results in Table 4. We observe that while both the relevance and stance auxiliary tasks improve the quality of the generated perspective, combining the two auxiliary tasks does not guarantee a better summary quality.
5.4 Analysis and Discussion
The results on ROUGE, BERTSCORE and human evaluation suggest that the perspective summarization model learning benefits from both the relevance and stance tasks. However, we also observe that the vanilla BART present a strong baseline in both automatic and human evaluations.
We list two typical cases where we observe the relevance and stance objectives improve the quality of the generated summary. For the query shown in Table 5, the BART model generates an out-of-context word "shutdown", which exists in the abstract, but is not applicable in the context provided by the query. The model with relevance objective, on the other hand, generates a perspective that is coherent to the context provided. For the query shown in Table 6, the baseline BART model incorrectly produces a supporting perspective to the
query, while the editorial or abstract presents the opposite stance. The model with the stance objective generates a perspective with a matching stance.
While we choose relevance and stance classification as the two auxiliary tasks in this case study, there exist many other candidate tasks that might be helpful in the setting. For instance, measuring the quality (Toledo et al., 2019), or more specifically persuasiveness (Carlile et al., 2018) of the perspective might be two, amongst other, viable options. As our study assumes that the abstract is provided for each editorial, the overall performance of perspective summarization will likely drop, if we use model-generated abstract instead of ground truth as input.
6.1 Argumentation in News Editorials
News editorials have been studied as a resource for studying many argumentation-related tasks. Wilson and Wiebe (2003); Yu and Hatzivassiloglou (2003) use editorials for the study on sentiments and opinions. Later works (Reed et al., 2008; Bal and Saint-Dizier, 2009; Chow, 2016) shift focus on the argumentation structure within editorials, and their persuasiveness effect (Al Khatib et al., 2016; El Baff et al., 2020). A few other recent studies have explored argument quality (El Baff et al., 2018) and generation (El Baff et al., 2019) when using editorials as a resource.
Our proposed dataset and study focus on the interplay between elements of the argumentation structure presented in editorial articles. Unlike previous work, we study these elements as the abstractive instead of extractive summary from the news editorials.
6.2 Argument Generation
Most early efforts in argument generation, i.e. generating components in an argumentation structure, study rule-based synthesis methods based on argumentation theories (Reed et al., 1996; Zukerman et al., 2000). With the recent progress in neural, sequence to sequence text generation methods (Sutskever et al., 2014), a few studies have adapted such techniques for end-to-end argument generation. (Wang and Ling, 2016; Hua and Wang, 2018; Hua et al., 2019).
The task of perspective generation in this work closely relates to argument conclusion generation (Alshomary et al., 2020). Our study focuses on
the setting where the target topic, or the query, is given as input to the generation model. Due to the implicit nature of the perspectives (Habernal et al., 2018), one key challenge to the task is keep the semantics of the perspective generated truthful to the abstract and editorial article. We approach this by measuring the compatibility of the perspective to the context along the dimensions of content salience (Bar-Haim et al., 2020) and stance correctness (Bar-Haim et al., 2017). Our multi-task generation approach conceptually resembles the work by Guo et al. (2018), where multiple auxiliary tasks is employed to improve the quality of the generated summary.
We present MULTIOPED an open-domain news editorial corpus that induces a number of argumentation-related tasks. The proposed dataset presents a few properties that are absent from existing datasets. First, the elements in the annotation structure are presented as abstraction over the text in editorial, as such elements usually exist implicitly in editorials. Second, as the pairs of editorials are aligned by topic, and exhibit opposing stance to each other, such structure allows for studies on cross-document argumentation structure. Third, the dataset allows for the study of multiple argumentation-related tasks together.
To demonstrate the power of having multiple related tasks in a single high-quality dataset, we study the problem of perspective summarization in a multi-task learning setting. Our analysis shows that modeling stance and relevance classification jointly with the summarization task improves the overall quality of the perspective generated.
In future work, we hope to utilize the corpus to improve the multi-task framework for perspective summarization. As we set aside the problem of abstract generation in our case study, we would also like to identify the challenges and potential solution to the problem. We hope that MULTIOPED presents opportunities and challenges to future research in argumentation.
We collected data for MULTIOPED by automatically extracting data from www.theperspective.com/perspectives. The CEO of the website, Daniel Ravner, granted us permission to extract and use their data for academic research. We fur
ther annotated the data using crowd-workers. All crowd-workers were compensated by a fair wage determined by estimating the average completing time of each annotation task. Please refer to section 4.2 for more details.
The queries, abstracts, and perspectives in MULTIOPED are written by the professional writers of the website. The website aims at presenting the perspectives in each article without unnecessary subjective interpretation, but there is no guarantee that no subjectivity is involved in their content creation process.
The authors would like to thank Daniel Ravner, the CEO of www.theperspective.com, for kindly granting access to data from the site for academic research. This work was supported in part by a Focused Award from Google, and a gift from Tencent.
Khalid Al Khatib, Henning Wachsmuth, Johannes Kiesel, Matthias Hagen, and Benno Stein. 2016. A News Editorial Corpus for Mining Argumentation Strategies. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3433-3443.
Milad Alshomary, Shahbaz Syed, Martin Potthast, and Henning Wachsmuth. 2020. Target inference in argument conclusion generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4334-4345, Online. Association for Computational Linguistics.
Bal Krishna Bal and Patrick Saint-Dizier. 2009. Towards an analysis of argumentation structure and the strength of arguments in news editorials. The Society for the Study of Artificial Intelligence and the Simulation of Behaviour.
Roy Bar-Haim, Indrajit Bhattacharya, Francesco Dinuzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251-261, Valencia, Spain. Association for Computational Linguistics.
Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav, and Noam Slonim. 2020. From arguments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4029-4039, Online. Association for Computational Linguistics.
Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. 2018. Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621-631, Melbourne, Australia. Association for Computational Linguistics.
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims. In Proc. of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Marisa Chow. 2016. Argument identification in Chinese editorials. In Proceedings of the NAACL Student Research Workshop, pages 16-21, San Diego, California. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Judith Eckle-Kohler, Roland Kluge, and Iryna Gurevych. 2015. On the role of discourse markers for discriminating claims and premises in argumentative discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2236-2242, Lisbon, Portugal. Association for Computational Linguistics.
Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, et al. 2020. Corpus wide argument mining-a working solution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7683-7691.
Roxanne El Baff, Henning Wachsmuth, Khalid Al Khatib, Manfred Stede, and Benno Stein. 2019. Computational argumentation synthesis as a language modeling task. In Proceedings of the 12th International Conference on Natural Language Generation, pages 54-64, Tokyo, Japan. Association for Computational Linguistics.
Roxanne El Baff, Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2018. Challenge or empowerment: Revisiting argumentation quality in a news editorial corpus. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 454-464, Brussels, Belgium. Association for Computational Linguistics.
Roxanne El Baff, Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. 2020. Analyzing the
Persuasive Effect of Style in News Editorial Argumentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3154-3160, Online. Association for Computational Linguistics.
Joseph L. Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3):613-619.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687-697, Melbourne, Australia. Association for Computational Linguistics.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1930-1940.
Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348-1356, Nagoya, Japan. Asian Federation of Natural Language Processing.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Xinyu Hua, Zhe Hu, and Lu Wang. 2019. Argument generation with retrieval, planning, and realization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661-2672, Florence, Italy. Association for Computational Linguistics.
Xinyu Hua and Lu Wang. 2018. Neural argument generation augmented with externally retrieved evidence. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 219-230, Melbourne, Australia. Association for Computational Linguistics.
Johannes Kiesel, Khalid Al-Khatib, Matthias Hagen, and Benno Stein. 2015. A shared task on argumentation mining in newspaper editorials. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 35-38, Denver, CO. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chris Reed, Derek Long, and Maria Fox. 1996. An architecture for argumentative dialogue planning. In International Conference on Formal and Applied Practical Reasoning, pages 555-566. Springer.
Chris Reed, Raquel Mochales Palau, Glenn Rowe, and Marie-Francine Moens. 2008. Language resources for studying argument. In LREC 2008.
Christian Stab and Iryna Gurevych. 2014a. Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1501-1510, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Christian Stab and Iryna Gurevych. 2014b. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46-56, Doha, Qatar. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112.
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. Automatic argument quality assessment - new datasets and methods. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5625-5635, Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 47-57, San Diego, California. Association for Computational Linguistics.
Theresa Wilson and Janyce Wiebe. 2003. Annotating opinions in the world press. In Proceedings of the Fourth SIGdial Workshop of Discourse and Dialogue, pages 13-22.
Hong Yu and Vasileios Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 129-136.
A.1 Experiment Settings
In this section, we describe our experiment settings in more details for reproducibility. We randomly split the MULTIOPED dataset into  $70\% / 10\% / 20\%$  splits for training, validation, and testing, respectively. We train each system for 6 epochs using the same training set, and use the validation set to find the best  $\alpha_{1}$  and  $\alpha_{2}$ . We report the test set results in Table 3. All test set results are averaged results using three different random initializations. The approximate training time for a system trained with 6 epochs on a 12GB GPU is less than an hour.
For results shown in Table 3, we use  $\alpha_{1} = 30$  in  $+\mathrm{REL}$ ,  $\alpha_{2} = 1$  in  $+\mathrm{Stance}$ , and  $\alpha_{1} = 1$ ,  $\alpha_{2} = 1$  in  $+\mathrm{REL}$  & STANCE, as they achieved the best results in the dev set during hyperperamater tuning A.2.
A.1.1 BART
BART pre-trained model has been proved effective on text generation, question answering, and summarization tasks (Lewis et al., 2020). Given the limited size of our dataset, we finetune on BART to transfer learn from the large amount of data it was pre-trained on. We use BART base with 6 encoder and decoder layers with hidden size of 768. We use AdamW with learning rate 3e-5 as our optimizer.
A.1.2 BERT Relevance Classifier
BERT pre-trained model has demonstrated its power in question answering, language inference, and text classification tasks (Devlin et al., 2019). We finetune BERT-mini on PERSPEC-TRUM dataset first on relevance classification task (Chen et al., 2019), and then finetune it on our dataset, yielding an accuracy of  $92\%$  on evaluation set ( $20\%$  of the data). The finetuned BERT model has 4 layers and hidden size of 256.
A.1.3 BERT Stance Classifier
As the Relevance Classifier, we also finetune it on PERSPECTRUM dataset before training on our dataset. However, we use a larger BERT model with 8 layers and hidden size of 768 since it is a slightly more difficult task than the relevance task. For the 30 queries (2% of the whole corpus) that are labeled as open-ended questions and do not constitute a clear stance, we exclude them from the training of BERT stance classifier. Our classifier eventually achieves 75% accuracy in the 20% evaluation set.
To control the degree we penalize our model using auxiliary loss, we introduce hyperparameters  $\alpha$ . We use the  $10\%$  validation set to choose our best parameters. We tune  $\alpha$  for different values from 0.1 to 50 to examine its affect on our model. We choose the best  $\alpha$  according to their relevance and stance scores, and if there is a tie, we select the one with the higher ROUGE2 score. Table 7, Table 8, and Table 9 show the validation set results for tuning the BART+Rel, BART+Stance, and BART+Rel & Stance systems, respectively.
α1	Relevance Score
0.1	90.0
1	91.43
5	88.21
15	92.5
30	92.5
50	92.5
Table 7: Tuning  $\alpha_{1}$  for BART + Rel. Here we choose  $\alpha_{1} = 30$  since it has the highest ROUGE2 score.
α2	Stance Score
0.1	72.14
1	72.50
5	64.64
15	64.29
30	60.36
50	62.50
Table 8: Tuning  ${\alpha }_{2}$  for BART + Stance
α2	α1	Relevance	Stance
0.1	1	92.86	71.07
0.1	5	88.21	63.93
0.1	50	92.5	64.64
1	1	93.93	68.93
1	5	90.71	71.79
1	50	93.57	70.36
Table 9: Tuning  $\alpha_{1}$  and  $\alpha_{2}$  for BART + Rel & Stance. Here we choose  $(\alpha_{1} = 1, \alpha_{2} = 1)$  over  $(\alpha_{1} = 5, \alpha_{2} = 1)$  since it has a higher ROUGE2 score.
A.3 Measure of Agreement
We use Cohen's and Fleiss kappa to measure the inter-rater agreement among annotators (Fleiss and Cohen, 1973). We calculate Cohen's kappa agree
ment when there are only two raters, and Fleiss's kappa when there are more than two.
Cohen's Kappa: Let  $n$  be the number of instances to be labeled by A and B two raters.  $g$  is the number of distinct categories, and  $f_{ij}$  denotes the frequency of the number of subjects with the ith categorical response for rater A and the jth categorical response for rater Y. The kappa agreement is then calculated as
$$
p _ {0} = \frac {1}{n} \sum_ {i = 1} ^ {g} f _ {i i}
$$
$$
p _ {e} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {g} f _ {i +} f _ {+ i}
$$
$$
\kappa = \frac {p _ {0} - p _ {e}}{1 - p _ {e}}
$$
where  $f_{i+}$  is the total for the  $i$ th row  $f_{+i}$  and is the total for the  $i$ th column in the frequency table.