SlowGuess/NAACL_2021_OCR huggingface dataset:
Benjamin Schiller and Johannes Daxenberger and Iryna Gurevych
Ubiquitous Knowledge Processing Lab
http://www.ukp.tu-darmstadt.de/
We rely on arguments in our daily lives to deliver our opinions and base them on evidence, making them more convincing in turn. However, finding and formulating arguments can be challenging. In this work, we present the Arg-CTRL—a language model for argument generation that can be controlled to generate sentence-level arguments for a given topic, stance, and aspect. We define argument aspect detection as a necessary method to allow this fine-granular control and crowdsourced a dataset with 5,032 arguments annotated with aspects. Our evaluation shows that the Arg-CTRL is able to generate high-quality, aspect-specific arguments, applicable to automatic counter-argument generation. We publish the model weights and all datasets and code to train the Arg-CTRL. $^{1}$
Language models (Bengio et al., 2003) allow to generate text through learned distributions of a language and have been applied to a variety of areas like machine translation (Bahdanau et al., 2015), summarization (Paulus et al., 2018), or dialogue systems (Wen et al., 2017). A rather new field for these models is the task of producing text with argumentative content (Wang and Ling, 2016). We believe this technology can support humans in the challenging task of finding and formulating arguments. A politician might use this to prepare for a debate with a political opponent or for a press conference. It may be used to support students in writing argumentative essays or to enrich one-sided discussions with counter-arguments. In contrast to retrieval methods, generation allows to combine and stylistically adapt text (e.g. arguments) based on a given input (usually the beginning of a sentence). Current argument generation models, however, produce lengthy texts and allow the user little
control over the aspect the argument should address (Hua et al., 2019; Hua and Wang, 2018). We show that argument generation can be enhanced by allowing for a fine-grained control and limiting the argument to a single but concise sentence.
Controllable language models like the CTRL (Keskar et al., 2019) allow to condition the model at training time to certain control codes. At inference, these can be used to direct the model's output with regard to content or style. We build upon this architecture to control argument generation based solely on a given topic, stance, and argument aspect. For instance, to enforce focus on the aspect of cancer for the topic of nuclear energy, we input a control code "Nuclear Energy CON cancer" that creates a contra argument discussing this aspect, for instance: "Studies show that people living next to nuclear power plants have a higher risk of developing cancer".
To obtain control codes from training data, we pre-define a set of topics to retrieve documents for and rely on an existing stance detection model to classify whether a sentence argues in favor (pro) or against (con) the given topic (Stab et al., 2018a). Regarding argument aspect detection, however, past work has two drawbacks: it either uses simple rule-based extraction of verb- and noun-phrases (Fujii and Ishikawa, 2006) or the definition of aspects is based on target-concepts located within the same sentence (Gemechu and Reed, 2019). Aspects as we require and define them are not bound to any part-of-speech tag and (1) hold the core reason upon which the conclusion/evidence is built and (2) encode the stance towards a general but not necessarily explicitly mentioned topic the argument discusses. For instance:
Topic: Nuclear Energy
Argument: Running nuclear reactors is costly as it
involves long-time disposal of radioactive waste.
The evidence of this argument is based upon the two underlined aspects. While these aspects encode
'https://github.com/UKPLab/controlled-argument-generation
Association for Computational Linguistics: Human Language Technologies, pages 380-396
Figure 1: (1) Gather data from large data sources. Extract sentences, classify arguments, and detect aspects. Arguments sharing a topic & stance & aspect ( $\cong$  control code) are concatenated into training documents. (2) The model is fine-tuned on each document with the control code prepended to each input sequence. (3) At inference, the model only needs a control code to generate an argument that follows the given topic & stance & aspect.
a negative stance towards the topic of "Nuclear Energy", the topic itself is not mentioned explicitly in the argument.
Our final controlled argument generation pipeline (see Figure 1) works as follows: (1) We gather several million documents for eight different topics from two large data sources. All sentences are classified into pro-, con-, and non-arguments. We detect aspects of all arguments with a model trained on a novel dataset and concatenate arguments with the same topic, stance, and aspect into training documents. (2) We use the collected classified data to condition the Arg-CTRL on the topics, stances, and aspects of all gathered arguments. (3) At inference, passing the control code [Topic] [Stance] [Aspect] to the model will generate an argument that follows these commands.
Our evaluation shows that the Arg-CTRL is able to produce aspect-specific, high-quality arguments, applicable to automatic counter-argument generation. The contributions are as follows: (i) We adapt and fine-tune the CTRL for aspect-controlled neural argument generation. (ii) We show that detecting argument aspects and conditioning the generation model on them are necessary steps to control the model's training process and its perspective while generating. (iii) We propose several methods to analyze and evaluate the quality of (controllable) argument generation models. (iv) We develop a new scheme to annotate argument aspects and release a dataset with 5,032 samples.
Argument Aspect Detection Early work by Fujii and Ishikawa (2006) focuses mainly on Japanese
and restricts aspects to noun- and verb-phrases, extracted via hand-crafted rules. Boltužić and Šnajder (2017) extract noun-phrases and aggregate them into concepts to analyze the microstructure of claims. Misra et al. (2015) introduce facets as low level issues, used to support or attack an argumentation. In that, facets are conceptually similar to aspects, but not explicitly phrased and instead seen as abstract concepts that define clusters of semantically similar text-spans of summaries. Bilu et al. (2019) define commonplace arguments that are valid in several situations for specified actions (e.g. “ban”) and topics (e.g. “smoking”). These actions are similar to aspects, but limited in number and manually defined. Gemechu and Reed (2019) detect, amongst others, concepts and aspects in arguments with models trained on expert annotations. However, in their definition, aspects have to point to a target concept mentioned in the argument. In our definition, aspects refer to a general topic which is not necessarily part of the sentence and our annotation scheme is applicable by non-experts.
The concept of framing dimensions (Boydstun et al., 2014) is close to argument aspects. In the field of argument mining, Ajjour et al. (2019) recently applied frames to label argument clusters. Yet, their method does not allow to detect frames. Other works present methods to automatically label sentences of news articles and online discussions with frames (Hartmann et al., 2019; Naderi and Hirst, 2017). These methods are, however, limited to a small set of predefined frames that represent high-level concepts. Contrarily, we operate on a fine-grained span-level to detect aspects that are explicitly mentioned in arguments.
Argument Generation Early approaches rely on rules from argumentation theory and user preference models (Carenini and Moore, 2006; Zukerman et al., 1998). In a more recent work, Sato et al. (2015) construct rules to find arguments in a large data source, which are then filtered and ordered with a neural network based ranker. Baff et al. (2019) use a clustering and regression approach to assemble discourse units (major claims, pro and con statements) to argumentative texts. However, most of these approaches rely on hand-crafted features and do not generalize well. Moreover, they all require permanent access to large data sources and are not able to generate new arguments.
Recently, research on generating arguments with language models gained more attention. Hua and Wang (2019) use a sequence to sequence model (Sutskever et al., 2014) to generate argumentative text by attending to the input and keyphrases automatically extracted for the input from, for example, Wikipedia. Other work focuses on generating argumentative dialogue (Le et al., 2018) and counter-arguments (Hidey and McKeown, 2019; Hua et al., 2019) based on a given input sentence, or on generating summaries from a set of arguments (Wang and Ling, 2016). Contrarily, we train a language model that does not require a sentence-level input for generation and allows for direct control over the topic, stance, and aspect of the produced argument.
Xing et al. (2017) design a language model that attends to topic information to generate responses for chatbots. Dathathri et al. (2019) train two models that control the sentiment and topic of the output of pre-trained language models at inference. Gretz et al. (2020a) fine-tune GPT-2 on existing, labeled datasets to generate claims for given topics. However, the latter works do not explore generation for such a fine-grained and explicit control as proposed in this work. We show that argument generation requires the concept of argument aspects to shape the produced argument's perspective and to allow for diverse arguments for a topic of interest.
3 Argument Aspect Detection
Argument aspect detection is necessary for our argument generation pipeline, as it allows for a fine-grained control over the generation process. We create a new dataset, as existing approaches either rely on coarse-grained frames or cannot be applied by non-expert annotators in a scalable manner.
We base our new aspect detection dataset on the UKP Sentential Argument Mining Corpus (UKPCorpus) by Stab et al. (2018b), as it already contains sentence-level arguments and two of the control codes we aim to use: topics and stance labels. More precisely, it contains 25,474 manually labelled sentences for eight controversial topics in English. Each sample consists of a topic and a sentence, labelled as either being supporting, attacking, or no argument towards the given topic. As we are only interested in arguments, we do not consider the non-argumentative sentences.
Step 1: Preliminary annotations To ensure the feasibility of creating a dataset for this task, two experts (a post-doctoral researcher and an undergraduate student with NLP background) independently annotate 800 random samples (from four topics, 200 per topic) taken from the UKP-Corpus. The annotations are binary and on token-level, where multiple spans of tokens could be selected as aspects. The resulting inter-annotator agreement of this study is Krippendorff's  $\alpha_{u} = .38$ . While this shows that the task is generally feasible, the agreement on exact token spans is rather low. Hence, in the following steps, we reduce the complexity of the annotation task.
Step 2: Annotation scheme Instead of free span-level annotations, we present annotators with a ranked list of aspect recommendations. To generate meaningful recommendations, we train a ranking model using the preliminary annotations (Step 1).
Step 2a: Data preparation for ranking To create training data for the ranker, we use a simple heuristic to calculate scores between 0 and 1 for all N-grams of a sentence by dividing the number of aspect tokens within an N-gram by its length  $N$ :  $\frac{\# \text{ aspect tokens}}{N} \in [0,1]$ . Our analysis reveals that  $96\%$  (783 of 814) of all aspects in the preliminary annotation dataset only contain one to four tokens. We thus decide to ignore all candidates with more than four tokens. No other limitations or filtering mechanisms are applied.
Step 2b: Training the ranker We use BERT (Devlin et al., 2019) and MT-DNN $^{2}$  (Liu et al., 2019) (base and large) to train a ranker. For training, we create five splits: (1) one in-topic split using a random subset from all four topics and (2) four
$^{2}$ BERT, fine-tuned on several NLP tasks via multi-task learning.
Topic	Five most frequent aspects (frequency)
Gun control	right (30), protect (18), background checks (17), gun violence (14), criminal (13)
Death penalty	cost (16), innocent (12), retribution (10), murder rate (9), deterrent (8)
Abortion	right (21), pain (10), choice (10), right to life (9), risk (9)
Marijuana legalization	dangerous (16), cost (13), risk (12), harm (10), black market (9)
General aspects	dangerous (in 8 of 8 topics), cost / life / risk / safety (in 7 of 8 topics)
Table 1: The five most frequent aspects for four exemplary topics and overall.
Setting	Rec@5	Rec@10	Rec@15	Rec@20
In-topic	0.7701	0.8468	0.8661	0.8925
Cross-topic	0.5951	0.7415	0.8164	0.8630
Table 2: In- and cross-topic Recall@k of the ranker used for aspect recommendations.
cross-topic splits using a leave-one-topic-out strategy. The cross-topic setup allows us to estimate the ranker's performance on unseen topics of the UKP-Corpus.
A single data sample is represented by an argument and an 1- to 4-gram of this argument, separated by the BERT architecture's [SEP] token. This technique expands the 800 original samples of the dataset to around 80,336. The model is trained for 5 epochs, with a learning rate of  $5 \times 10^{-5}$ , and a batch size of 8. We use the mean squared error as loss and take the recall@k to compare the models. The in- and cross-topic results of the best-performing model (MT-DNN<sub>BASE</sub>) are reported in Table 2. All results are the average over runs with five different seeds (and over all four splits for the cross-topic experiments).
Step 2c: Creating the annotation data For each of the four topics that are part of the preliminary annotation dataset, we use the in-topic model to predict aspects of 629 randomly chosen, unseen arguments from the UKP-Corpus. For the other four topics of the UKP-Corpus, we choose the best cross-topic model to predict aspects for the same amount of samples. To keep a recall of at least  $80\%$ , we choose the ten and fifteen highest-ranked aspect candidates for samples as predicted by the in-topic and cross-topic model, respectively. We remove aspect candidates that include punctuation, begin or end with stopwords, or contain digits.
Step 3: Annotation study We use Amazon Mechanical Turk to annotate each sample by eight different workers located in the US, paying  $7.6 per hour (minimum wage is$ 7.25 per hour). Based on a subset of 232 samples, we compute an  $\alpha_{u}$  of .67 between crowdworkers and experts (three doctoral researchers). Compared to the initial study, the
new approach increases the inter-annotator agreement between experts by approx. 11 points (see App. A for further details on the annotation study). Based on this promising result, we create a dataset of 5,032 high-quality samples that are labelled with aspects, as well as with their original stance labels from the UKP-Corpus. We show the most frequent (lemmatized) aspects that appear in some topics in Table 1.
We create a cross-topic split with the data of two topics as test set (gun control, school uniforms), one topic as dev set (death penalty), and the remaining topics as train set and evaluate two models with it. First, we use the ranking approach described in Step 2a-2b to fine-tune MT-DNN<sub>BASE</sub> on the newly generated data ("Ranker"). At inference, we choose the top  $T$  aspects for each argument as candidates. We tune  $T$  on the dev set and find  $T = 2$  to be the best choice. Second, we use BERT for sequence tagging (Wolf et al., 2020) and label all tokens of the samples with BIO tags. As previously done with the ranker, we experiment with BERT and MT-DNN weights and find BERT<sub>LARGE</sub> to be the best choice (trained for 5 epochs, with a learning rate of  $1 \times 10^{-5}$  and a batch size of 32). We flatten the predictions for all test samples and calculate the  $F_1$ , Precision, and Recall macro scores. All models are trained over five seeds and the averaged results are reported in Table 3.
BERT $_{LARGE}$  predicts classes B and I with an  $F_1$  of .65 and .53, hence aspects with more than one token are less well identified. A difference is to be expected, as the class balance of B's to I's is 2,768 to 2,103. While the ranker performs worse based on the shown metrics, it has a slightly higher recall for class I. We assume this is due to the fact that it generally ranks aspects with more than one token on top, i.e. there will often be at least one or more I's in the prediction. In contrast to that, BERT $_{LARGE}$  focuses more on shorter aspects, which is also in accordance with the average aspect length of 1.8 tokens per aspect in the dataset.
Model	F1 macro	Precision	Recall
Majority (baseline)	.3085	.2871	.3333
Ranker	.6522	.6685	.6474
BERTBASE	.6980	.6927	.7040
BERTLARGE	.7100	.7240	.6993
Table 3: Test set results of the models for aspect detection. Majority only predicts class O.
In total,  $\mathrm{BERT}_{LARGE}$  outperforms the ranker by almost 6 percentage points in  $\mathrm{F}_1$  macro.
4 Data Collection Pipeline
This section describes the data collection and preprocessing for the argument generation pipeline. We aim to train a model that is able to transfer argumentative information concisely within a single sentence. We define such an argument as the combination of a topic and a sentence holding evidence with a specific stance towards this topic (Stab et al., 2018b). Consequently, the following preprocessing steps ultimately target retrieval and classification of sentences. To evaluate different data sources, we use a dump from Common-Crawl $^3$  (CC) and Reddit comments $^4$  (REDDIT) to fine-tune two separate generation models. The CC dump is from July 2016 and contains 331M documents (3.6TB) after dedduplication. The REDDIT dump contains 2.5B documents (1.6TB) from December 2012 to May 2019. We choose to compare these two sources, as REDDIT is focused around user discussions and CC contains mixed sources with potentially higher quality.
Document Retrieval We index REDDIT and CC with ElasticSearch $^5$  and, for both, gather up to 1.5M documents for each of the eight topics of the UKP-Corpus. To increase the search results, we add synonyms (see App. B) for most topics.
Argument and Stance Classification We split the sentences of all documents and remove duplicates. We notice that many sentences are not relevant with regard to the document's topic. To enforce topic-relevance, we decide to filter out all sentences that do not contain at least one token of the respective topic or its defined synonyms (see App. B). We use the ArgumentText API's<sup>6</sup> argument and stance classification models (Stab et al., 2018a) to classify
all sentences into argument or non-argument  $(\mathrm{F}_1$  macro  $= .7384)$  , and remaining arguments into pro or con with regard to the topic  $(\mathrm{F_1}$  macro  $= .7661)$
Aspect Detection We detect aspects on all remaining arguments. To speed up the detection on millions of sentences, we use  $\mathrm{BERT}_{BASE}$  instead of  $\mathrm{BERT}_{LARGE}$  (see Table 3).
Training Document Generation We create the final training documents for the argument generation model by concatenating all arguments that have the same topic, stance, and aspect (i.e. the same control code). Further, we aggregate all arguments that include an aspect with the same stem into the same document (e.g. arguments with cost and costs as aspect). To cope with limited hardware resources, we restrict the total number of arguments for each topic and stance to 100,000 (i.e. 1.6M over all eight topics). Also, as some aspects dominate by means of quantity of related arguments and others appear only rarely, we empirically determine an upper and lower bound of 1,500 and 15 arguments for each document, which still allows us to retrieve the above defined amount of training arguments.
5 Model Training and Analysis
In the following, we describe the architecture and the training process of the Arg-CTRL and analyze its performance.
5.1 Model and Training
Model The goal of a statistical language model is to learn the conditional probability of the next word given all (or a subset of) the previous ones (Bengio et al., 2003). That is, for a sequence of tokens  $x = (x_{1},\dots,x_{n})$ , the model learns  $p(x_{i}|x_{<i})$  where  $x_{i}$  is the  $i$ -th word of sequence  $x$ . For this work, we use the 1.63 billion-parameter Conditional Transformer Language Model (CTRL) by Keskar et al. (2019), which is built on a transformer-based sequence to sequence architecture (Vaswani et al., 2017). The CTRL has shown to produce high quality text, is general enough to be adapted for conditioning on the control codes we aim to use, and we do not need to pre-train the weights from scratch. Formally, the CTRL adds an extra condition to each sequence by preponding a control code  $c$ , hence learning  $p(x_{i}|x_{<i},c)$ . The control code is represented by a single token and can then be used to direct the model output at inference. We extend the model from its previous limit of a single-token control code to accept multiple tokens. For
3https://commoncrawl.org 4https://files.pushshift.io/reddit/ comments/ 5https://www.elastic.co 6https://apiargumentsearch.com
cloning CON unrespectable . Cloning humans for reproductive purposes is unethical and unacceptable , but creating cloned embryos solely for research - which involves destroying them anyway - is downright criminal . (0.97)
cloning CON disfavored . , cliques ) to them . (0.36)
nuclear energy PRO safe . In addition , we must continue developing safer technologies like small modular reactors which will help us meet our nation 's need for reliable , emission-free sources of low-emission energy [...] . (0.96)
nuclear energy CON leak . “ We are concerned about the possibility of further releases of radioactivity due to possible melting or cracking of fuel rods at the No . (0.47)
marijuana legalization PRO safer : Legalizing cannabis will help reduce crime rates ( especially violent crimes ) and make society safer overall . (0.96)
marijuana legalization PRO benefits . Decrease amount of police officers needed 6 . (0.37)
Table 4: Generated arguments of highest/lowest quality with  $\mathrm{Arg - CTRL_{CC}}$ . Bold text shows the used control code. Quality score in brackets as predicted by the argument quality model. "[...] " signals shortened text.
decoding at inference, we use penalized sampling as proposed by Keskar et al. (2019). It defines a near-greedy sampling strategy that uses a penalty constant, effectively lowering the probability of previously generated tokens to prevent repetitions. Training The CTRL was trained on 140GB of data from several large resources like Wikipedia, subreddits, and news data. We base our experiments on the pre-trained weights for a sequence length of 256 and fine-tune (see App. C for technical details) two models:  $\mathrm{Arg - CTRL_{CC}}$  (on the CC data) and  $\mathrm{Arg - CTRL_{REDDIT}}$  (on the REDDIT data). All training documents are sampled randomly for training. The respective control code is prepended to each sequence of 256 subwords of a document.