SlowGuess/NAACL_2022_OCR huggingface dataset:
The embeddings and weights generated by the model can be analyzed separately for further insights. First, in Figure 1 we compare the distribution of audience embeddings  $(\alpha)$  for the liberal and conservative reviewers. According to our theory, these can be interpreted as individuals reticence to being persuaded. While a majority of reviewers are close to 0, we see two liberals with larger negative values (meaning they are particularly open to the messages) and several conservatives on the right (suggesting they are more closed off to these messages). This supports El Baff et al. (2020) observation that conservatives are generally resistant
Figure 1: Reviewer Embeddings for the Editorial Rasch Model on the x-axis. Blue represents liberal reviewers, red represents conservative reviewers.
to the New York Times style; however, the fact that the majority of reviewers from both sides have similar embeddings, suggests that the pattern is not very strong.
This data also contained information from each reviewers Big 5 Personality test. We measured the Pearson correlation between the reviewers embeddings and found a strong correlation with extroversion  $(r = -0.568, p < 0.05)$  and openness  $(r = -0.344, p < 0.1)$ . These findings closely match El Baff et al. (2018)'s analysis between Big 5 Personality Ratings and the affectedness labels. The audience embedding is a latent parameter, thus, it does not explicitly represent personality or political preferences. This analysis has two implications: first, the IRT framework is successful in situations where additional data about the audience is not available; second, analyzing the embeddings lets us learn qualities of the audience post-hoc.
For style, the highly weighted features included negative sentiment markers (nrc_negative, liwc_negative emotions); this aligns with El Baff et al. (2020)'s observation that ineffective editorials tend to have a neutral tone (although their study only focuses on liberal reviewers). The quality features do not show consistent behavior: the quality_mean feature has a large negative weight (e.g. sign of a bad editorial), but the 75th and 25th percentile features have positive weights; suggesting that the quality measure does not transfer well to editorials.
6.2 Debate Results
The Debates data is approximately 10 times larger than Editorials and contains a more diverse audience. The results are shown in Table 2. Without the
15 In addition, there is natural polarity in the Debate task that lends itself to the 2PL model, as  $\phi$  in equation 2 is designed to model such an effect.
Model	Accuracy
Random	0.500
Style	0.561
Text	0.581
Speaker	0.611
Speaker + Style	0.626
-β (popularity) layer	0.604
Table 2: Results for Debates Task (2PL Model).
Figure 2: Distribution of one-dimensional audience embeddings on the y-axis.
popularity parameter,  $\beta$  the performance decreases, which confirms the theory that both polarity and popularity are necessary to adequately represent the argument and the speaker. The Speaker stance model outperforms just Text; a probable explanation is that the stances are a proxy for the actual opinions expressed in the text that a simple unigram representation can not capture.
To understand the latent audience embeddings we compare them to the self-reported political affiliations from their profiles. Figure 2 shows a clear separation between liberals and conservatives (the two largest groups). This finding supports the work of Durmus and Cardie (2019) which showed that similarity on 'Big Issue Stance' between the speaker and the audience member is a good indicator for predicting outcome. As with Editorialis, the advantage of our approach is that we were able to infer audience member preferences without using their profiles.
To understand what  $\phi$  and  $\beta$  tells us about persuasive theory, we will focus on the Speaker+Style model:
High Polarity: Abortion, Gay Marriage, Progressive Tax;
Figure 3: Contrast of weights from popularity vs polarity features.
Low Polarity: Border Fence, Gun Rights, Homeschooling;
High Popularity: quality_max, quality_range, liwc_differ;
Low Popularity: liwc_Exclam, liwc Authentic, liwc_drives
For popularity the significant factors are related to style and quality. The high 'quality_max' feature suggests that the quality model transfers better to this context than Editorials. The low popularity value for 'liwc Authentic' is interesting, as El Baff et al. (2020) also found that authenticity generally led to No Effect editorials.
For polarity, the highest weighted are the stances. 'Polarity High' corresponds to having a Pro stance on those issues, which in this case represent a Liberal view point. This corresponds with the Liberal recipient embeddings in Figure 2 having generally positive embeddings (alignment in weights results in positive final weight). The opposite is true for the Conservative issues and embeddings. This alignment reinforces the finding that prior beliefs play a strong role in outcomes (Durmus and Cardie, 2018).
Figure 3 plots the weights learned for each feature for the polarity and popularity parameters. $^{16}$
Notably, the orthogonal pattern extends beyond the top features, features that strongly predict whether the audience responds to an argument do not usually strongly predict whether the argument is popular overall.
This figure excludes features that had very small weights along both dimensions.
Overall	Audience Average	Org Average
Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
Org Prior	0.608	0.514	0.606	0.263	0.630	0.513
Audience Prior	0.710	0.415	0.716	0.318	0.714	0.472
Org Only	0.757	0.667	0.759	0.589	0.728	0.573
Org + Style	0.781	0.708	0.761	0.662	0.771	0.678
-β (popularity)	0.750	0.653	0.749	0.643	0.756	0.654
Sep Feat V1	0.725	0.619	0.726	0.571	0.700	0.520
Sep Feat V2	0.748	0.678	0.750	0.604	0.698	0.654
Table 3: Results For Advocacy Task (2PL Model).
6.3 Advocacy Results
Table 3 shows the results for the Advocacy task. $^{17}$  The overall accuracy and macro-F1 scores represent results across all data, while the Org and Audience average accuracy represent data for individual organizations and respondents. Due to the variation in action rate and sample size, the macro-F1 results are particularly important.
While the Org Only model performs well,[18] the improved performance with the additional of Style suggests that the style of an email still affects the user. The style features may have an advantage for recipients associated with a diverse set of organizations. Without  $\beta$ , the performance is significantly worse, again confirming the need for both parameters.
To better understand the effect of style and org features, two additional models are trained that separate between polarity and popularity. In Sep Feat V1,  $\phi$  receives style features,  $\beta$  receives org indicators. In this setting,  $(\alpha \cdot \phi)$  represents how individuals are affected by style, while  $\beta$  models the organizations base rate. In Sep Feat V2 the features are reversed. V1 has the worst performance of all five 2PL models, suggesting that modeling the interaction between the recipient and organization  $(\alpha \cdot \phi)$  is important. Org-Only and V2 have mixed performance on accuracy, but V2 performs better on macro-F1, suggesting that style influences the recipients' decisions to act.
Finally, we analyze the features with lowest and highest magnitudes from  $\beta$  in the Org+Style model.
The highest weighted features include concreteness, average-logical appeal, word count and quality 75th percentile. The lowest weighted features (unlikely to produce action) include valence, quality mean, arousal and liwc-we. Similar to the Editorials, the quality features are contradictory, suggesting the connection between sentence level and document level quality needs to be investigated further. The logical appeal feature shows they are particularly effective (the corresponding scores for emotional and credibility appeals had smaller, negative weights).
In this paper, we validate the social psychology frameworks for persuasion using the IRT framework to explicitly model the audience and the speaker. Our approach lets us analyze how different audience members respond to the same argument, and we show that our representation implicitly learns latent audience features modeled explicitly by other models.
We empirically showed several additional insights about persuasion. In the Debates and Advocacy tasks, the Popularity parameter improved performance showing that certain stylistic elements are universally appealing. In the Debates task, the audiences' embeddings aligned with their political affiliation, showing that prior beliefs play a strong role in their argument perception. While the background information about the audiences was available for these tasks, we did not need to model it explicitly; as a result this setup allows us to make predictions for audiences who do not report their affiliation.
A potential negative side of the models is they may learn latent characteristics of the speaker or
<sup>17</sup>Due to computational constraints, we omitted the raw text model from this task. <sup>18</sup>One likely explanation for this performance is that audience is not independent of the speaker - by virtue of receiving emails from this organization, recipients may also have similar preferences.
audience they may not be aware of or consider private. However, all datasets studied in this paper were either public and anonymous or private with audiences who consented to analysis.
This study focused on simple representations to show the viability of our method and provide for explainability. To build on this foundation in future work, we will: expand argument text representations with contextual word embeddings and stance detection models; include higher dimensional embedding for audience and item parameters (the IRT models easily generalize to this set-up). These improvements will allow us to better capture the elements of persuasion, especially in a complex case like Advocacy.
AERA, APA, and NCME. 2014. Standards for Educational and Psychological Testing.
A. L. Birnbaum. 1968. Some latent trait models and their use in inferring an examinee's ability. Statistical theories of mental test scores.
R Darrell Bock and Murray Aitkin. 1981. Marginal maximum likelihood estimation of item parameters: Application of an em algorithm. Psychometrika, 46(4):443-459.
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911.
Amparo Elizabeth Cano-Basave and Yulan He. 2016. A study of the impact of persuasive argumentation in political debates. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1405-1413, San Diego, California. Association for Computational Linguistics.
Dennis Chong and James N. Druckman. 2007. Framing theory. Annual Review of Political Science, 10(1):103-126.
R.B. Cialdini. 2009. Influence: The Psychology of Persuasion. Collins Business Essentials. HarperCollins e-books.
Joshua Clinton, Simon Jackman, and Douglas Rivers. 2004. The statistical analysis of roll call data. American Political Science Review, 98(2):355-370.
Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg. 2012. Echoes of power: Language effects and power differences in social interaction. In Proceedings of WWW, pages 699-708.
Esin Durmus and Claire Cardie. 2018. Exploring the role of prior beliefs for argument persuasion. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Esin Durmus and Claire Cardie. 2019. A corpus for modeling user and language effects in argumentation on online debating. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 602-607, Florence, Italy. Association for Computational Linguistics.
Roxanne El Baff, Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2018. Challenge or empowerment: Revisiting argumentation quality in a news editorial corpus. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 454-464, Brussels, Belgium. Association for Computational Linguistics.
Roxanne El Baff, Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. 2020. Analyzing the persuasive effect of style in news editorial argumentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3154-3160, Online. Association for Computational Linguistics.
Gerhard H. Fischer. 1973. The linear logistic test model as an instrument in educational research. Acta psychologica, 37(6):359-374.
Sean M. Gerrish and David M. Blei. 2012. The issue-adjusted ideal point model.
Martin Gleize, Eyal Shnarch, Leshem Choshen, Lena Dankin, Guy Moshkowich, Ranit Aharonov, and Noam Slonim. 2019. Are you convinced? choosing the more convincing evidence with a siamese network. CoRR, abs/1907.08971.
Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2019. A large-scale dataset for argument quality ranking: Construction and analysis. CoRR, abs/1911.11408.
Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? analyzing and predicting convincingness of web arguments using bidirectional LSTM. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589-1599, Berlin, Germany. Association for Computational Linguistics.
Benjamin E. Lauderdale and Tom S. Clark. 2014. Scaling politically meaningful dimensions using texts and votes. American Journal of Political Science, 58(3):754-771.
Liane Longpre, Esin Durmus, and Claire Cardie. 2019. Persuasion of the undecided: Language vs. the listener. In Proceedings of the 6th Workshop on Argument Mining, pages 167-176, Florence, Italy. Association for Computational Linguistics.
Frederic M Lord. 1980. Applications of item response theory to practical testing problems. Routledge.
Stephanie M. Lukin, Pranav Anand, Marilyn Walker, and Steve Whittaker. 2017. Argument strength is in the eye of the beholder: Audience effects in persuasion.
Arya D. McCarthy, Kevin P. Yancey, Geoff T. LaFlair, Jesse Egbert, Manqian Liao, and Burr Settles. 2021. Jump-starting item parameters for adaptive language tests. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 883-899, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Saif Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. CoRR, abs/1308.6297.
Prathiba Natesan, Ratna Nandakumar, Tom Minka, and Jonathan D Rubright. 2016. Bayesian prior choice in irt estimation using mcmc and variational bayes. Frontiers in psychology, 7:1422.
Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. In Proceedings of the 12th international conference on artificial intelligence and law, pages 98-107.
James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of liwc2015. Technical report.
Richard E Petty and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. In Communication and persuasion, pages 1-24. Springer.
Keith T Poole and Howard Rosenthal. 1985. A spatial model for legislative roll call analysis. American Journal of Political Science, pages 357-384.
G. Rasch. 1960. Probabilistic Models for Some Intelligence and Attainment Tests. Studies in mathematical psychology. Danmarks Paedagogiske Institut.
Leslie Rutkowski, Matthias Von Davier, and David Rutkowski. 2014. Handbook of international large-scale assessment. Background, technical issues, and methods of data analysis.
Burr Settles, Geoffrey T. LaFlair, and Masato Hagiwara. 2020. Machine learning-driven language assessment. Transactions of the Association for Computational Linguistics, 8:247-263.
Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2007. Detecting arguing and sentiment in meetings. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, pages 26-34.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of WWW.
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezuelan, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. Automatic argument quality assessment - new datasets and methods. CoRR, abs/1909.01007.
Keyon Vafa, Suresh Naidu, and David M Blei. 2020. Text-based ideal points. arXiv preprint arXiv:2005.04232.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5635-5649, Florence, Italy. Association for Computational Linguistics.
Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior research methods, 45(4):1191-1207.
Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, and Eduard Hovy. 2019. Let's make your request more persuasive: Modeling persuasive strategies via semi-supervised neural nets on crowdfunding platforms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3620-3630, Minneapolis, Minnesota. Association for Computational Linguistics.
A Model Training Details
The models described in section 6 were trained as follows. In equation (6),  $c$  is set to  $1e^{-4}$  for all experiments.  $L2$  loss is used for the Editorials and Advocacy corpus and text model for Debates,  $L1$  is used for the remaining models in the Debates corpus. Editorial models are trained for 200 epochs; Debates for 25; Advocacy for 5. A learning rate of 0.01 is used for Editorials and Debates; 0.005 is used for Advocacy.
All results are reported over 5-fold cross-validation, with the splits performed at an argument level. All models are fit using the AdamW optimizer. The  $\alpha$  embedding initializations are drawn from a uniform distribution of  $-0.5$  to  $0.5$ .