SlowGuess/NAACL_2022_OCR huggingface dataset:
Anastassia Kornilova
Daniel Argyle
Vlad Eidelman
FiscalNote Research
anastassia, daniel, vlad@fiscalnote.com
In this paper, we apply Item Response Theory, popular in education and political science research, to the analysis of argument persuasiveness in language. We empirically evaluate the model's performance on three datasets, including a novel dataset in the area of political advocacy. We show the advantages of separating these components under several style and content representations, including evaluating the ability of the speaker embeddings generated by the model to parallel real-world observations about persuadability.
Persuasion is the art of instilling in someone a given belief or desire to take a given action. The action can be expressing agreement with the speaker in a debate (Durmus and Cardie, 2019), making a donation to a crowdfunding campaign (Yang et al., 2019) or non-profit (Wang et al., 2019), or a Supreme Court ruling (Danescu-Niculescu-Mizil et al., 2012). Social psychology frameworks for understanding persuasion, such as the Elaboration Likelihood Model (ELM), argue that attributes of successful persuasion fall into three groups: (1) message, the text of the argument; (2) audience; and (3) speaker, the source of the argument. (Petty and Cacioppo, 1986; Lukin et al., 2017; Cialdini, 2009).
Although much attention has been given to studying the text, text in isolation fails to capture how the audiences' prior beliefs and predispositions can affect their response to the same argument. Several recent studies have considered all three factors within the context of specific datasets by creating features to represent the audience as a whole or by building separate models for different types of audiences (Lukin et al., 2017; Tan et al., 2016; Durmus and Cardie, 2019; El Baff et al., 2020). In this paper, we present a broad framework that can represent individual audience members in one model
across a diverse set of persuasion tasks.
Since implementing the ELM framework requires separate data about the speaker, audience, and argument, it is difficult to validate empirically. Often, we only have access to the observed outcome (e.g. did the person donate money). Both the persuadability of the audience and the persuasiveness of the argument are unobserved. Motivated by this, we explicitly model a persuasive scenario as a function of latent variables describing the persuasiveness of the audience and the persuasiveness of the text.
Our approach is based on Item Response Theory (IRT), a framework for modeling the interaction between latent traits and observable outcomes. While these types of models are well known in the context of education (Fischer, 1973; Lord, 1980; McCarthy et al., 2021) and politics (Clinton et al., 2004), to our knowledge this is the first application of an IRT model to study persuasion. Using this framework, we model the interaction between the grouped argument and speaker, and the audience, explicitly. The argument and speaker are grouped together because in practice it is hard to separate their effects, especially in the written tasks covered in this study.
We explore two variations on the IRT framework and apply it to three different persuasion tasks. In addition to two previously studied tasks, we introduce a novel setting related to political advocacy group campaigns, where a recipient is asked by an organization to take a specific action.
We evaluate these models with different parameterizations, including style and content features, showing that they are both effective for predicting persuasion, and have the ability to uncover latent characteristics of the audience that were modeled explicitly in previous works.
Our contributions are as follows: 1) we formalize the use of IRT model formulations for persuasion and show the advantages of them over existing approaches, 2) we introduce a new dataset of
Findings of the Association for Computational Linguistics: NAACL 2022, pages 77 - 86 July 10-15, 2022 ©2022 Association for Computational Linguistics
political advocacy emails, 3) we apply the formulations with style and content features on three persuasion tasks, and 4) we show that the separate latent audience component is interpretable and consistent with external information. All code associated with the paper is available at https://github.com/akornilo/IRT_Persuasion.
2 Item Response Theory
Item Response Theory (IRT) represents a set of models that explain an observed outcome based on latent traits. These models are frequently used when an outcome is easily observed, but the factors predicting that model are unobservable. For example, in education an outcome could be a student's answer to an exam question, and the latent predictive traits are students knowledge and the difficulty of the question; in politics an outcome could be a vote on a bill and the unobservable traits are the legislator's and bill's ideology. Crucially, an IRT model provides both a prediction of the outcome, and an interpretable measurement of the latent variables.
In applying IRT to persuasiveness, we can view the audience as having a response to the item, where the item is an argument composed of the speaker and message pair.
2.1 Rasch Testing Model
We build on two specific IRT parameterizations. The first, the Rasch model (Rasch, 1960) is commonly used in education research to model the difficulty of standardized test questions (Fischer, 1973; Lord, 1980). In it the probability that an individual  $i$  answers test question  $j$  is given by:
$$
p \left(y _ {i j} = 1 \mid \alpha , \beta\right) = \sigma \left(\alpha_ {i} - \beta_ {j}\right) \tag {1}
$$
where  $\alpha_{i}$  represents a respondent (e.g. a student's ability) and  $\beta_{j}$  represents the item (e.g. the difficulty of a test question). Intuitively, if the ability is greater than the question difficulty, then the student will answer the question correctly. Given a series of exam sessions one can estimate values of  $\alpha$  and  $\beta$  for all of the students and questions in the dataset. This can be done using a variety of optimization strategies, such as Expectation Maximization or Bayesian techniques (Bock and Aitkin, 1981; Natesan et al., 2016).
However, one limitation of this approach is that it cannot be used to perform inference on new test questions because all parameters are estimated
simultaneously. To solve this problem, Fischer (1973) proposed the linear logistic test model that parameterizes the difficulty,  $\beta$ , as a weighted linear combination of test features. In this formulation, the student  $(\alpha)$  remains a latent variable, but the  $\beta$  of an unseen question can be predicted using attributes of the question itself.
Following Fischer (1973), the parameterization used to predict the item parameters is a weighted linear sum of features:
$$
\beta_ {j} = \sum_ {k = 1} ^ {K} w _ {k} \times \psi_ {j k} \tag {2}
$$
where  $\psi_{k}$  is an input feature representing the item, and  $w_{k}$  is the associated weight.
In order to apply this model to persuasion, we propose considering argumentation as follows: First, arguments can vary in quality, similar to test questions having different difficulty levels. Second, we can only measure the quality of an argument based on how the audience reacted; similar to how a students ability is measured via their performance. Third, it is possible that a good argument is matched with an audience reticent to persuasion, similar to a good student receiving a particularly hard question. Note that this requires an audience member observe multiple arguments, and that each argument be heard by multiple audience members. Inspired by the linear logistic model, we model the latent argument parameter as a function of attributes of the argument itself, thus allowing us to include attributes of the speaker and text in the model directly.
2.2 Two Parameter IRT
While the simplicity of the Rasch model is powerful, a two parameter generalization of an IRT model (a two parameter logistic - 2PL) provides additional benefits for our application (Birnbaum, 1968). In the simplest version, a two parameter model (so called because the item is modeled with two parameters) is as follows:
$$
p \left(y _ {i j} = 1 \mid \boldsymbol {\alpha}, \boldsymbol {\phi}, \boldsymbol {\beta}\right) = \sigma \left(\alpha_ {i} \cdot \phi_ {j} - \beta_ {j}\right) \tag {3}
$$
where as before,  $\alpha_{i}$  represents the respondent (students ability), and  $\beta_{j}$  is the item's difficulty, but
<sup>1</sup>Analogous to the Rasch model, this tells us the overall difficulty level of the question
now  $\phi_j$  represents the item's discrimination. We similarly generalize this model by estimating the two item parameters,  $\beta_j$  and  $\phi_j$ , as linear functions of features as in Equation 2.
This framework has commonly been used to explain legislator voting behavior (Clinton et al., 2004), a useful analogy as many of the persuasion contexts we consider have political undertones. In this case, the response  $y_{ij}$  is a vote by respondent  $i$  (a legislator) on item  $j$  (a bill). Clinton et al. (2004) show that the parameter  $\alpha_{i}$  can then be interpreted as the respondent's ideology (e.g. negative values are more liberal, positive values are more conservative);  $\phi_{j}$  is referred to the bills polarity (i.e. discrimination);  $\beta_{j}$  represents the bill's popularity (i.e. difficulty). Persuasion is a generalization of this framework because popularity can correspond to properties of arguments that are appealing overall, while polarity represents techniques or topics that appeal only to a subset of the audience.
2.3 Audience Analysis
Once a Rasch or a 2PL model is fit, the learned  $\alpha$  can be interpreted as a one-dimensional respondent embedding. In the legislator voting context these values can be interpreted as ideologies: legislators with very negative or very positive embeddings reflect very liberal and conservative stances, respectively, while those with small-value embeddings map to moderate legislators. While interpretation of these values will depend on the task, in general, similar embeddings will map to similar audience members and can be grouped together for further analysis.
Audience Effects The properties of the audience in relation to argument persuasiveness have previously been examined in several predictive studies. Lukin et al. (2017) show that audiences with a more "open" personality respond better to emotional arguments, while El Baff et al. (2020) show that liberals are more affected by the style of a new editorial
than conservatives. Wang et al. (2019) also find that people with different personality types respond differently to emotional vs. logical appeals. Tan et al. (2016) show how "malleable" different Reddit users are to new perspectives. Durmus and Cardie (2018, 2019) show that prior beliefs play a strong role in how persuasive someone is. Cano-Basave and He (2016) study persuasiveness of style in political speeches. In contrast to these studies, our method is designed to work when we have limited or no information about the audience of an argument.
Item Response Theory As described in the previous section, IRT models have primarily been applied in politics to measure the ideology of politicians (Clinton et al., 2004; Poole and Rosenthal, 1985). While most IRT implementations here rely only on the responses as data, more recent work augment the models to take advantage of the text through a simultaneously estimated topic model (Gerrish and Blei, 2012; Vafa et al., 2020; Lauderdale and Clark, 2014).
The efficacy of IRT has been applied on large-scale datasets to verify the validity of standardized tests both in the U.S. and internationally (AERA et al., 2014; Rutkowski et al., 2014). Recent advances have focused on polytomous test questions and creating new questions (the 'cold-start' problem: Settles et al., 2020; McCarthy et al., 2021). In this paper, we focus on the simplest form, but this area of research points to many possible extensions.
Argument Quality Argument mining has been studied in various domains (Palau and Moens, 2009). Most relevant here, several studies have attempted to study argument quality through pairwise ranking as the outcome (Habernal and Gurevych, 2016; Gleize et al., 2019; Toledo et al., 2019).
Framing Theory In the study of framing effects, the expectancy value model (Chong and Druckman, 2007) represents an attitude as  $\sum_{i} v_{i} \times w_{i}$ , where  $v_{i}$  is the favorability of the object of evaluation (e.g. a candidate), on dimension  $i$  (e.g. foreign affairs or personality), and  $w_{i}$  is the salience weight  $(\sum_{i} w_{i} = 1)$ . Our parameterization of  $\beta_{j}$  and  $\phi_{j}$  can be seen in this paradigm as identifying frames in communication, with each feature of the style and content as a dimension, and learning the framing effect of each.
$^{2}$ Discrimination is how well the question is able to tell which students perform better, a high value indicates clearly separates high scoring students from low scoring, a negative value would indicate that low performing students are more likely to get the question right than high performing.
<sup>3</sup>Large negative or positive values indicate that a bill is strongly ideological, a value close to zero means the vote isn't strongly driven by ideology.
Large values indicate a bill that is "difficult" to vote for and is less likely regardless of ideology.
In order to apply the IRT framework, an audience member must respond to multiple arguments (and arguments must be observed by multiple audience members). Too few responses implies that an audience member's latent value will be driven entirely by the one or two arguments. While not many existing argument mining datasets meet this criteria, we are able to study three diverse settings. Additionally, our advocacy task is akin to many real-world settings where users on one-platform are asked to complete an arbitrary task (e.g. a retail mailing list getting users to click on a promotion).
4.1 NYTimes Editorial
The NYTimes Editorial corpus consists of 975 editorials from the New York Times news portal (El Baff et al., 2018). Each publication was reviewed by 3 conservatives and 3 liberals from a pool of 12 conservative and 12 liberal reviewers.
Each reviewer rated the editorials as either 'challenging', 'reinforcing' or 'no effect'. These labels must be approached with care as reinforcing could imply 'reinforced view against the article's stance'. El Baff et al. (2020) study this corpus in a ternary setting by aggregating the liberal and conservative votes and building separate models for each side. For our study, we construct a binary task for predicting 'whether this article had an effect'. While this framing elides whether the speaker succeeded according to her intent, it does relay whether the argument was persuasive.
4.2 Debates (DDO) Corpus
DDO is a corpus of 78k debates scraped from debate.org. Each debate has two speakers and an audience votes on a winner. In addition, each audience member can fill out their profile with their political and religious ideology, and stance on various political issues (e.g. Abortion or the Border Wall). Originally, it was used to study how prior beliefs and similarities between the audience and the speaker affected debate outcomes (Durmus and Cardie, 2018, 2019).
To preprocess the data, we removed all debates that have fewer than three rounds, end in a forfeit or a tie, have fewer than 100 words per side, or have fewer than 5 points awarded total. In addition, we excluded debates not on the following issues: Politics, Religion, Society, Philosophy, Education and Economics. Since we are interested in modeling individual audience members, we identify audience members who have responded on at least 10 debates, then remove debates where none of those members responded. The final dataset contains approximately 60k datapoints; 6320 debates and 1131 responders.
Each debate has one side with a pro argument and one side with a con argument, resulting in the wining side being "assigned more points". The prediction task consists of whether a responder gave more points to a given debate side. Since our models only consider one argument at a time, we treat each side of the debate as a separate item, concatenating texts from all rounds from that speaker.[8]
4.3 Advocacy Campaign Corpus
Grassroots advocacy is the process wherein organizations (e.g. companies, non-profits, coalitions) encourage individual citizens to influence their government. In the United States, such lobbying often takes the form of advocacy email campaigns, sent by an organization to specific audiences, asking them to take an action, such as contacting their legislators to vote yes or no on a particular bill.
We construct a dataset containing the text and metadata of these emails, from a popular advocacy software platform, paired with whether recipients took the requested action. Organizations will send different messages to the same audience over time, allowing us to identify which emails (items) elicited a response from specific recipients. Thus, it is possible to distinguish messages that did not generate interest overall (popularity) from messages that did not resonate with specific groups of recipients (polarity).
The dataset contains 63,795 individual recipients of 7,067 email campaigns from 328 different organization, resulting in approximately 2 million individual data points. Each recipient has data for
$^{5}$ https://webis.de/data/webis-editorial-quality-18.html $^{6}$ https://www.cs.cornell.edu/\~esindurmus/ddo.html
7While the audience can assign points to various aspects of the debate, this study will only consider the cumulative sum of the points.
<sup>8</sup>We are interested in how a single unit of argument affects the audience, and leave extension of this to account for both simultaneously to future work.
Due to privacy concerns, this dataset will not be released, but platform users agreed to terms of services providing for internal analysis.
15 to 100 emails and had an action rate between of  $5\% -95\%$ . Each email included in the dataset had at least 6 responses.
The data is not balanced with respect to organizations; while the largest organizations sent over 200 emails, the median is 6. One possibility of this imbalance is overfitting a feature that is only pertinent to one, particularly prevalent organization. To mitigate such effects, we include an indicator variable to specify the organization.[11]
5 Model Features
Argument analysis is often separated into style and content features (Cano-Basave and He, 2016; Longpre et al., 2019; El Baff et al., 2020), with additional categories included for argument quality and task specific properties. Since we group the speaker and the argument text together, we combine features representing both as inputs to  $\phi$  and  $\beta$ .
Lexicon Style Features Style features represent higher-level properties of words and rhetorical structures. We chose the following sets of such features from lexicons that were commonly used in previous argumentation literature:
LIWC lexicon of 93 metrics ranging from parts-of-speech to thinking styles to emotions (Pennebaker et al., 2015);[12] Valence, Arousal, Dominance (Warriner et al., 2013); Concreteness (Brysbaert et al., 2014). (These features were shown to be useful for argument quality analysis by Tan et al. (2016).) Argument features developed by Somasundaran et al. (2007), including necessity, emphasizing, desire, contrasting and rhetorical question; NRC Lexicon: Word-level level associations for emotions like anger, disgust and fear (Mohammad and Turney, 2013); Sentiment and Subjectivity: as implemented in the TextBlob Python Library.[13]
Argument Text We use TF-IDF unigrams to represent the text directly (tuned with respect each
task). While we initially explored using deep, contextual text representations, they did not show benefit, and the motivation for this paper is to understand the benefits of the IRT framework, rather than optimize performance based on the argument alone.
Debate-Only Speaker Features In the debate platform, users can optionally specify a stance - for, against, undecided or no stance - on 48 issues such as Abortion, Death Penalty or Gay Marriage. These can be viewed as a proxy for the content as users often present arguments that align with their views.
Advocacy-Only Org Indicator An indicator to account for the large variation in action rate between organizations. Additional indicators are used to represent the industry and organization size.
Advocacy-Only Appeals Using data from Wang et al. (2019), we built a multi-class classifier to recognize 'emotional', 'logical' and 'credibility' appeals. The classifier was applied at a sentence level to the emails, and features were created for the average and the sum of the scores across the sentences.
Advocacy-Only Misc Features : The day of the week and time of day have a strong effect on email click rate. $^{14}$  We include indicator features for the day of the week and the hour of day. We include an urgency indicator feature, based on a custom list of words indicative of high urgency and timeliness (e.g. “soon”, “now”, “hurry”).
IBM Quality Gretz et al. (2019) released a dataset of 30k sentence-level arguments with 0-1 quality ratings. Unlike our tasks where quality is a latent property, these sentences were assessed for quality directly. We re-implemented the BERT-FT model from this paper, using the MACE-P score. Since these scores were trained on short texts, we apply them to individual sentences in the input text, then use the min, max, average, range, 25th, 50th, and 75th percentiles of these scores. As far as we know, this is the first study to transfer the quality model to longer texts. These features will be grouped with Style for the analysis.
14https://sleeknote.com/blog/best-time-to-send-email
10 Those with a lower or higher action rate are unlikely to be illustrative of persuasion characteristics.
Alternatively, we could construct separate models for each organization, but refrain from doing so for three reasons. First, about a quarter of recipients are 'multi-org' - they receive emails from multiple sources, thus, we would like to model their behavior across all of them. Second, as many of the organizations are not well represented, they benefit from patterns that appear across different organizations. Finally, maintaining a separate model for every recipient and recipient is not as efficient or scalable.
12We purchased a copy of the software from liwc. wpgine.com to obtain these labels.
<sup>13</sup>https://textblob.readthedocs.io/
Model	Accuracy
Audience Prior	0.662
Style	0.741
Text	0.754
Style + Text	0.750
Table 1: Results for the Editorials Task (Rasch Model).
6 Models and Results
Since the Editorials corpus is the smallest, we use the simpler Rasch parameterization, while the 2PL model is used for the Debates and Advocacy tasks. $^{15}$  Each of the models is trained using a regularized binary cross-entropy loss:
$$
\begin{array}{l} L \left(\hat {y} _ {i}, y _ {i}\right) = - y _ {i} \log \hat {y} _ {i} - \left(1 - y _ {i}\right) \log \left(1 - \hat {y} _ {i}\right) \\ + c \cdot \| \alpha , \beta , \phi \| \\ \end{array}
$$
where  $\hat{y}_i$  is the output from equation 1 or 3, and  $y_i$  is the binary label, representing if the persuasion was successful. The second part of the equation represents a regularization parameter. Details on the experimental parameters can be found in Appendix A. For each task, an audience prior baseline is used. It is generated by calculating the rate at which the audience member was persuaded in the training data (e.g. did the article have an effect, how many recipients took the requested action), then drawing labels on the test data accordingly.
6.1 Editorial Results
The results on the Editorial task are shown in Table 1. The performance for all three feature sets is relatively similar, with all outperforming the audience prior.