Venue: ACL 2020

# Analyzing the Persuasive Effect of Style in News Editorial Argumentation

Roxanne El Baff \({}^{1,2}\)   Henning Wachsmuth \({}^{3}\)    Khalid Al-Khatib \({}^{2}\)   Benno Stein \({}^{2}\)

\({}^{1}\) German Aerospace Center (DLR), Germany, roxanne.elbaff@dlr.de

\({}^{2}\) Bauhaus-Universitat Weimar, Weimar, Germany, <first>.<last>@uni-weimar.de

\({}^{3}\) Paderborn University, Paderborn, Germany, henningw@upb.de

###### Abstract

News editorials argue about political issues in order to challenge or reinforce the stance of readers with different ideologies. Previous research has investigated such persuasive effects for argumentative _content_. In contrast, this paper studies how important the _style_ of news editorals is to achieve persuasion. To this end, we first compare content- and style-oriented classifiers on editorials from the liberal NYTimes with ideology-specific effect annotations. We find that conservative readers are resistant to NYTimes style, but on liberals, style even has more impact than content. Focusing on liberals, we then cluster the leads, bodies, and endings of editorials, in order to learn about writing style patterns of effective argumentation.

+
Footnote †: July 5 - 10, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

The interaction between the author and the intended reader of an argumentative text is encoded in the linguistic choices of the author and their persuasive effect on the reader Halmari and Virtanen (2005). News editorials, in particular, aim to challenge or to reinforce the stance of readers towards controversial political issues, depending on the readers' ideology El Baff et al. (2018). To affect readers, they often start with an enticing lead paragraph and end their argument with a "punch" Rich (2015).

Existing research has studied the persuasive effect of argumentative content and structure Zhang et al. (2016); Wachsmuth et al. (2016) or combinations of content and style Wang et al. (2017); Persing and Ng (2017). In addition, some works indicate that different types of content affect readers with different personalities Lukin et al. (2017) and beliefs Durmus and Cardie (2018). However, it remains unexplored so far what stylistic choices in argumentation actually affect which readers. We expect such choices to be key to generating effective argumentation Wachsmuth et al. (2018).

This paper analyzes the persuasive effect of style in news editorial argumentation on readers with different political ideologies (conservative vs. liberal). We model style with widely-used features capturing argumentativeness Somasundaran et al. (2007), psychological meaning Tausczik and Pennebaker (2010), and similar (Section 3). Based on the NYTimes editorial corpus of El Baff et al. (2018) with ideology-specific effect annotations (Section 4), we compare style-oriented with content-oriented clasifiers for persuasive effect (Section 5).1

Footnote 1: For reproducibility, the code of our experiments can be found here: [https://github.com/webis-de/acl20-editorials-style-persuasive-effect](https://github.com/webis-de/acl20-editorials-style-persuasive-effect)

While the general performance of effect prediction seems somewhat limited on the corpus, our experiments yield important results: Conservative readers seem largely unaffected by the style of the (liberal) NYTimes, matching the intuition that content is what dominates opposing ideologies. On the other hand, the style features predict the persuasive effect on liberal readers even better than the content features -- while being complementary. That is, style matters as soon as ideology matches.

Knowing about the specific structure of news editorials, we finally obtain common stylistic choices in their leads, bodies, and endings through clustering. From these, we derive writing style patterns that challenge or reinforce the stance of (liberal) readers of (liberal) news editorials, giving insights into what makes argumentation effective.

## 2 Related Work

Compared to other argumentative genres Stede and Schneider (2018), news editorials use many rhetorical means to achieve a persuasive effect on readers van Dijk (1995). Computational research has dealt with news editorials for retrieving opinions Yu and Hatzivassiloglou (2003); Bal (2009), mining arguments Al-Khatib et al. (2017), and

analyzing their properties (Bal and Dizier, 2010; Scheffler and Stede, 2016). While Al-Khatib et al. (2016) modeled the structure underlying editorial argumentation, we use the corpus of El Baff et al. (2018) meant to study the persuasive effects of editorials depending on the readers' political ideology. Halmari and Virtanen (2005) state that four aspects affect persuasion in editorials: linguistic choices, prior beliefs of readers, prior beliefs and behaviors of authors, and the effect of the text.

Persuasive effectiveness reflects the rhetorical quality of argumentation (Wachsmuth et al., 2017). To assess effectiveness, Zhang et al. (2016) modeled the flow of content in debates, and Wachsmuth et al. (2016) the argumentative structure of student essays. Others combined different features for these genres (Persing and Ng, 2015). The impact of content selection relates to the notion of _framing_(Ajjour et al., 2019) and is well-studied in theory (van Eemeren, 2015). As Wang et al. (2017), however, we hypothesize that content and style achieve persuasion jointly. We target argumentative style here primarily, and we analyze its impact on liberal and conservative readers.

In related work, Lukin et al. (2017) found that emotional and rational arguments affect people with different personalities, and Durmus and Cardiie (2018) take into account the religious and political ideology of debate portal participants. In follow-up work, Longpre et al. (2019) observed that style is more important for decided listeners. Unlike them, we focus on the stylistic choices made in well-planned argumentative texts.

The lead paragraphs and the ending of an editorial have special importance (Rich, 2015). Hynds (1990) analyzes how leads and endings changed over time, whereas Moznette and Rarick (1968) examined the readability of an editorial based on them. To our knowledge, however, no one investigated their importance computationally so far. In this paper, we close this gap by analyzing what style of leads and endings is particularly effective compared to the editorial's body.

## 3 Style Features

To model style, we need to abstract from the content of a news editorial. This section outlines the feature types that we employ for this purpose. Most of them have been widely used in the literature. Table 1 summarizes all features.

LiwcPsychological word usage is reflected in the Linguistic Inquiry and Word Count (Tausczik and Pennebaker, 2010). LIWC is a lexicon-based text analysis that assigns words to psychologically meaningful categories (Tausczik and Pennebaker, 2010). We use the LIWC version of Pennebaker et al. (2015), which contains 15 dimensions listed in the following with examples.

(1) _Language metrics_: words per sentence, long words. (2) _Function words_: pronouns, auxiliaries. (3) _Other grammar_: common verbs, comparisons. (4) _Affect words_: positive and negative emotion. (5) _Social word_: family, friends. (6) _Cognitive processes_: discrepancies, certainty. (7) _Perceptual processes_: feeling, seeing. (8) _Biological processes_: body, health. (9) _Core drives and needs_: power, reward focus. (10) _Time orientation_. (11) _Relativity_. (12) _Personal concerns_. (13) _Informal speech_. (14) _Punctuation_. (15) _Summary variables_.

The last dimension (15) contains four variables, each of which is derived from various LIWC dimensions: (a) _Analytical thinking_(Pennebaker et al., 2014): The degree to which people use narrative language (low score), or more logical and formal language (high score). (b) _Clout_(Kacewicz et al., 2014): The relative social status, confidence, and leadership displaced in a text. (c) _Authenticity_(Newman et al., 2003): The degree to which people reveal themselves authentically. (d) _Emotional tone_(Ohn et al., 2004): Negative emotions, for scores lower than 50, and positive emotions otherwise.

NRC Emotion&SentimentTo represent the mood of editorials, we use the NRC lexicon of Mohammad and Turney (2013). NRC contains a set of English words and their associations with (1) emotions such as _anger_, _disgust_, and _fear_ as

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Feature Base** & **Overview** & **Reference** \\ \hline Linguistic inquiry and word count & Psychological meaningfulness in percentile & Pennebaker et al. (2015) \\ NRC emotional and sentiment lexicon & Count of emotions (e.g. _sad_, etc.) and polarity words & Mohammad and Turney (2013) \\ Webis Argumentative Discourse Units & Count of each evidence type (e.g., _statistics_) & Al-Khatib et al. (2017) \\ MPQA Arguing Lexicon & Count of 17 types of arguing (e.g., _assessments_) & Somasundaran et al. (2007) \\ MPQA Subjectivity Classifier & Count of subjective and objective sentences & Riloff and Wiebe (2003) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of the style feature types in our dataset. Each feature is quantified at the level of the editorial.

 

well as (2) _negative_ and _positive_ sentiment polarities. These features are represented as the count of words associated with each category.

Webis ADUsTo identify argumentative units in editorials that present evidence, we use the pre-trained evidence classifier of Al-Khatib et al. (2017). For each editorial, we identify the number of sentences that manifest _anecdotal_, _statistical_, and _testimonial_ evidence respectively.

MPQA ArguingSomasundaran et al. (2007) constructed a lexicon that includes various patterns of arguing such as _assessments_, _doubt_, _authority_, _emphasis_. For each lexicon, we have one feature that represents the count of the respective pattern in an editorial.

MPQA SubjectivityWe apply the subjectivity classifier provided in OpinionFinder 2.0Riloff and Wiebe (2003); Wiebe and Riloff (2005) on the editorials, in order to count the number of _subjective_ and _objective_ sentences there.

## 4 Data

As the basis of our analysis, we use the Websiditorial-Quality-18 corpus El Baff et al. (2018). The corpus includes persuasive effect annotations of 1000 English news editorials from the liberal New York Times (NYTimes).2 The annotations capture whether a given editorial _challenges_ the prior stance of readers (i.e., making them rethink it, but not necessarily change it), _reinforces_ their stance (i.e., helping them argue better about the discussed topic), or is _ineffective_ for them. Each editorial has been annotated by six annotators: three with liberal and three with conservative ideology.

Footnote 2: For copyright reasons, the corpus provides only annotations for IDs of editorials. The actual texts of these editorials come from the NYTimes Annotated Corpus Sandhaus (2008).

To evaluate an editorial's persuasive effect on liberals, we computed the majority vote of their annotations for the editorial (and, similarly, for conservatives). We ended up with 979 editorials with effect labels for liberals and conservatives, because we found 21 duplicate editorials with the same content but different IDs (for these, we use the majority vote across all duplicates).

The corpus does not have predefined evaluation datasets. To mimic real-life scenarios, we chronologically split it into a training set (oldest 80%) and a test set (newest 20%). Table 2 shows the distribution of ideology-specific effects in the datasets.

## 5 Prediction of Persuasive Effects

To assess the impact of news editorial style on readers, we employ our style-based features on the task of predicting an editorial's persuasive effect: Given either of the two ideologies (liberal or conservative), predict for each editorial whether it is _challenging_, _reinforcing_, or _ineffective_.

We developed separate prediction models for the effect on liberals and conservatives, respectively. For each style feature type and for their combinations, we trained one SVM model with a linear kernel on the training set using _scikit-learn_Pedregosa et al. (2011).

Given the dataset split mentioned above (training set 80%, test set 20%), we tuned the SVM's cost hyperparameter using grid search with 5-fold cross-validation on the training set. Since the distribution of effect labels is highly skewed, we set the hyperparameter _class_weight_ to "balanced". We then trained the best model on the whole training set and evaluated it on the test set. For comparison, we also built models for standard _content_ features (lemma 1- to 3-grams), and we consider the _random baseline_ that picks an effect class by chance.

For both ideologies, Table 3 reports the macro- and micro F\({}_{1}\)-scores for the style features, their best-performing combination,3 the content features, and the best combination of content and style.4

Footnote 3: Best style literals: _LIWC, MPQA Subjectivity_. Best style conservatives: _NRC Emotion& Sentiment, Webis ADUs_

Footnote 4: Content+style liberals: _LIWC, MPQA Arguing, MPQA Subjectivity, Content_. Conservatives: _MPQA Arguing, Content_

We computed significance using Wilcoxon's test to reveal differences between each two approaches among _best style_, _content_, _best content_+_style_, and _baseline_.5 We obtained the means of F\({}_{1}\)-scores used in the significance tests by conducting five-fold cross-validation on the test set, using the same SVM hyperparameters as above.

[TABLE:A1.T2][ENDTABL 

In general, the results indicate that the persuasive effect seems hard to predict on the given corpus. Still, we observe that the style features play a notable role in predicting the effect of editorials on _liberals_. They achieve a significantly better macro F\({}_{1}\)-score of 0.43 when combined with content compared to 0.36 when using content alone, at \(p<\) 0.05. On the other hand, the F\({}_{1}\)-scores of content (macro 0.37, micro 0.38) and style (both 0.36) in predicting the effect on _conservatives_, are insignificantly different even from the baseline (0.33, 0.34).

These results suggest that style is important as soon as the ideology of a reader matches the one of the news portal (at least, this holds for liberal ideology), but not if it mismatches (here, conservative).

## 6 Identification of Style Patterns

Observing that the style of NYTimes editorials affects liberal readers, we seek to learn what patterns of writing style makes their argumentation effective. To this end, we (1) abstract each discourse part of an editorial (lead, body, ending) into a style label using cluster analysis and (2) identify sequential patterns of style labels that are specific to challenging, ineffective, and reinforcing editorials.

Clustering Styles of Discourse PartsGiven the importance of specific discourse parts of editorials Rich (2015), we split each editorial into lead, body, and ending. For each part, we separately perform three steps on the training set of the given corpus:6

Footnote 6: The corpus of Sandhaus (2008) contains lead and paragraph annotations. The lead spans either the first two paragraphs (994 editorials), the first three (5), or the first only (1). We consider the last paragraph as the ending in all cases.

1. Extract the style features from Section 3.
2. Perform a cluster analysis on the style features using cosine \(k\)-means. \(k\) is determined with the elbow method on the inertia of the clusters.
3. Derive cluster labels from the most discriminating features across clusters: For each cluster, we determine those 2-3 values (e.g., "high tone, low authenticity") whose combination suffices to significantly distinguish a cluster from others. With _high_ to _very low_, we mean here a feature has significantly higher or lower scores compared to other clusters.7 Footnote 7: For each feature (e.g., _tone_), we measured significance using Anova (in case of homogeneity and normality) or Kruskal (otherwise). In the case of \(p<\) 0.05, we conducted post-hoc analysis (independent \(t\)-test in case of normality, Mann-Whitney otherwise) with Bonferroni correction for each cluster pair, and we calculated the effect size \(r\). Based on the effect size values, we deduced the labels of each cluster and the relative differences between them (_high_ to _very low_).

Table 4 shows the distribution of lead, body, and ending clusters over challenging, ineffective, and reinforcing editorials.

For each discourse part, the most discriminating feature is _tone_, followed by _authenticity_. The former combines positive (higher scores) and neg

\begin{table}
\begin{tabular}{l l r r r} \hline \hline  & \multicolumn{3}{c}{**Liberals**} & \multicolumn{3}{c}{**Conservatives**} \\ \cline{2-5} Features & Macro & Micro & Macro & Micro \\ \hline LIWC & 0.31 & 0.40 & 0.25 & 0.26 \\ NRC Emotion\&Sentiment & 0.33 & 0.39 & 0.28 & 0.29 \\ Webis ADUs & 0.28 & 0.36 & 0.31 & 0.31 \\ MPQA Arguing & 0.33 & 0.41 & 0.29 & 0.29 \\ MPQA Subjectivity & 0.33 & 0.38 & 0.26 & 0.28 \\
**Best Style** & *0.38 & *0.49 & 0.36 & 0.37 \\ Content & 0.36 & *0.49 & **0.37** & **0.38** \\
* & ***0.43*
* & ***0.54*
* & 0.36 & 0.36 \\ \hline Random baseline & 0.23 & 0.26 & 0.33 & 0.34 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Test set micro and macro F\({}_{1}\)-scores of each feature type and their best combinations in classifying the persuasive effect on liberals and conservatives.
* and \({}^{\dagger}\) indicate significant differences at \(p<\) 0.05 against the _Random baseline_ and _Content_ respectively.
 

ative (lower scores) emotional tones Cohn et al. (2004). The latter indicates the degree to which people authentically reveal themselves; the higher the score, the more personal, humble, or vulnerable the writer is Newman et al. (2003). In Table 4, we observe, for example, that the lead of challenging editorials over-proportionally often shows low authenticity, or that bodies with positive tone but low authenticity tend to be ineffective.

Identification of Style PatternsFrom Table 4, we determine the (maximum) two labels for each discourse part that are most specific to each of the three persuasive effect classes. From these, we build all possible lead-body-ending sequences, as visualized in Figure 1. According to a \(\chi\)-square test, the distributions of these sequences differ significantly at _p_< 0.05. They reveal the following patterns of NYTimes editorials for liberal readers:

* _Challenging editorials_ often begin with a polar emotional tone, followed by a negative tone. They tend to have low authenticity (i.e., not humble/personal) in the whole discourse (see Figure 2 for an example).
* _Ineffective editorials_ over-proportionally often start with authenticity and dull tone. They then tend to diffuse in different directions and to have a short ending paragraph.
* _Reinforcing editorials_ tend to start and end with a negative tone. They often avoid relativity in the actual arguments (i.e., in the body). While these insights are naturally still vague to some extent and require more analysis in follow-up research, they show a first way of capturing the style of editorial argumentation.

## 7 Conclusion

This paper analyzes the importance of news editorials _style_ in achieving persuasive effects on readers with different political ideologies. We find evidence that style has a significant influence on how a (liberal) editorial affects a (liberal) reader. Inspired by the theory of the high importance of the lead and ending in writing editorials Rich (2015), we also reveal common effective and ineffective style sequences (lead-body-ending) statistically.

Our findings help to understand how effective argumentation works in the political sphere of editorial argumentation -- and how to _generate_ such argumentation. In related work, El Baff et al. (2019) revealed the impact of style features on generating pathos- and logos-oriented short argumentative texts based on the rhetorical strategies discussed by Wachsmuth et al. (2018). With the findings of this paper, we go beyond, defining the basis of a style-dependent generation model for more sophisticated argumentation, as found in news editorials.

Figure 1: Sequences of lead, body, and ending styles most specific to challenging, ineffective, and reinforcing news editorials. The triangles denote whether the given style attribute is high, medium, or (very) low. The ordering of attributes reflects their importance.

Figure 2: Example of a challenging editorial, along with the styles observed for its lead, body, and ending.

 



## References

* Ajjour et al. (2019) Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019. Modeling frames in argumentation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2922-2932, Hong Kong, China. Association for Computational Linguistics.
* Al-Khatib et al. (2017) Khalid Al-Khatib, Henning Wachsmuth, Matthias Hagen, and Benno Stein. 2017. Patterns of argumentation strategies across topics. In _2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017_, pages 1362-1368. Association for Computational Linguistics.
* Al-Khatib et al. (2016) Khalid Al-Khatib, Henning Wachsmuth, Johannes Kiesel, Matthias Hagen, and Benno Stein. 2016. A news editorial corpus for mining argumentation strategies. In _26th International Conference on Computational Linguistics (COLING 2016)_, pages 3433-3443. Association for Computational Linguistics.
* Bal (2009) Bal Krishna Bal. 2009. Towards an analysis of opinions in news editorials: How positive was the year? (project abstract). In _Proceedings of the Eight International Conference on Computational Semantics_, pages 260-263. Association for Computational Linguistics.
* Bal and Dizier (2010) Bal Krishna Bal and Patrick Saint Dizier. 2010. Towards building annotated resources for analyzing opinions and argumentation in news editorials. In _Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)_. European Languages Resources Association (ELRA).
* Cohn et al. (2004) Michael A Cohn, Matthias R Mehl, and James W Pennebaker. 2004. Linguistic markers of psychological change surrounding september 11, 2001. _Psychological science_, 15(10):687-693.
* van Dijk (1995) Teun A. van Dijk. 1995. Opinions and ideologies in editorials. In _Proceedings of the 4th International Symposium of Critical Discourse Analysis, Language, Social Life and Critical Thought_, Athens.
* Durmus and Cardie (2018) Esin Durmus and Claire Cardie. 2018. Exploring the role of prior beliefs for argument persuasion. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 1035-1045.
* van Emeren (2015) Frans H. van Emeren. 2015. _Strategic Maneuvering_, pages 1-9. American Cancer Society.
* Baff et al. (2019) Roxanne El Baff, Henning Wachsmuth, Khalid Al-Khatib, Manfred Stede, and Benno Stein. 2019. Computational argumentation synthesis as a language modeling task. In _12th International Natural Language Generation Conference_. ACL.
* Baff et al. (2018) Roxanne El Baff, Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2018. Challenge or empower: Revisiting argumentation quality in a news editorial corpus. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_, pages 454-464. Association for Computational Linguistics.
* Halmari and Virtanen (2005) Helena Halmari and Tuija Virtanen. 2005. _Persuasion across Genres: a Linguistic Approach_, volume 130. John Berjamins Publishing.
* Hynds (1990) Ernest C Hynds. 1990. Changes in editorials: A study of three newspapers, 1955-1985. _Journalism Quarterly_, 67(2):302-312.
* Kacewicz et al. (2014) Ewa Kacewicz, James W Pennebaker, Matthew Davis, Moongee Jeon, and Arthur C Graesser. 2014. Pronoun use reflects standings in social hierarchies. _Journal of Language and Social Psychology_, 33(2):125-143.
* Longpre et al. (2019) Liane Longpre, Esin Durmus, and Claire Cardie. 2019. Persuasion of the undecided: Language vs. the listener. In _Proceedings of the 6th Workshop on Argument Mining_, pages 167-176, Florence, Italy. Association for Computational Linguistics.
* Lukin et al. (2017) Stephanie Lukin, Pranav Anand, Marilyn Walker, and Steve Whittaker. 2017. Argument strength is in the eye of the beholder: Audience effects in persuasion. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 742-753. Association for Computational Linguistics.
* Mohammad and Turney (2013) Saif M Mohammad and Peter D Turney. 2013. Crowd-sourcing a word-emotion association lexicon. _Computational Intelligence_, 29(3):436-465.
* Moznette and Farick (1968) James Moznette and Galen Farick. 1968. Which are more readable: Editorials or news stories? _Journalsim Quarterly_, 45(2):319-321.
* Newman et al. (2003) Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003. Lying words: Predicting deception from linguistic styles. _Personality and social psychology bulletin_, 29(5):665-675.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. _Journal of Machine Learning Research_, 12:2825-2830.
* Pennebaker et al. (2015) James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015. Technical report, University of Texas at Austin.
* Pennebaker et al. (2014) James W Pennebaker, Candy K Chung, Joey Frazee, Gary M Lavergne, and David I Beaver. 2014. When small words forettell academic success: The case of college admissions essays. _PloS one_, 9(12):e115844

Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 543-552. Association for Computational Linguistics.
* Persing and Ng (2017) Isaac Persing and Vincent Ng. 2017. Lightly-supervised modeling of argument persuasiveness. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 594-604, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Rich (2015) Carole Rich. 2015. _Writing and reporting news: A coaching method_. Cengage Learning.
* Riloff and Wiebe (2003) Ellen Riloff and Janyce Wiebe. 2003. Learning extraction patterns for subjective expressions. In _Proceedings of the 2003 conference on Empirical methods in natural language processing_.
* Sandhaus (2008) Evan Sandhaus. 2008. The New York Times Annotated Corpus. _Linguistic Data Consortium, Philadelphia_, 6(12):e26752.
* Scheffler and Stede (2016) Tatjana Scheffler and Manfred Stede. 2016. Realizing argumentative coherence relations in German: A contrastive study of newspaper editorials and Twitter posts. In _Proceedings of the COMMA Workshop: Foundations of the Language of Argumentation_, pages 73-80.
* Somasundaran et al. (2007) Swapna Somasundaran, Joser Ruppenhofer, and Janyce Wiebe. 2007. Detecting arguing and sentiment in meetings. In _Proceedings of the SIGdial Workshop on Discourse and Dialogue_, volume 6.
* Stede and Schneider (2018) Manfred Stede and Jodi Schneider. 2018. _Argumentation Mining_. Number 40 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool.
* Tausczik and Pennebaker (2010) Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: Liwc and computerized text analysis methods. _Journal of language and social psychology_, 29(1):24-54.
* Wachsmuth et al. (2016) Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. 2016. Using argument mining to assess the argumentation quality of essays. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 1680-1691. The COILNG 2016 Organizing Committee.
* Wachsmuth et al. (2017) Henning Wachsmuth, Nona Naderi, Yufeng Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 176-187. Association for Computational Linguistics.
* Wachsmuth et al. (2018) Henning Wachsmuth, Manfred Stede, Rocanne El Baff, Khalid Al Khatib, Maria Skeppstedt, and Benno Stein. 2018. Argumentation synthesis following rhetorical strategies. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3753-3765. Association for Computational Linguistics.
* Wang et al. (2017) Lu Wang, Nick Beauchamp, Sarah Shugars, and Kechen Qin. 2017. Winning on the merits: The joint effects of content and style on debate outcomes. _Transactions of the Association for Computational Linguistics_, 5:219-232.
* Wiebe and Ridoff (2005) Jansce Wiebe and Ellen Riloff. 2005. Creating subjective and objective sentence classifiers from unannotated texts. In _International conference on intelligent text processing and computational linguistics_, pages 486-497. Springer.
* Yu and Hatzivassiloglou (2003) Hong Yu and Vasileios Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In _Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing_, pages 129-136. Association for Computational Linguistics.
* Zhang et al. (2016) Justine Zhang, Ravi Kumar, Sujith Ravi, and Cristian Danescu-Niculescu-Mizil. 2016. Conversational flow in Oxford-style debates. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 136-141, San Diego, California. Association for Computational Linguistics.