“I understand your perspective”: LLM Persuasion through the Lens of Communicative Action Theory
Authors: Esra Dönmez, Agnieszka Falenska
Accepted at: acl25

“I understand your perspective”: LLM Persuasion through the Lens of Communicative Action Theory

Esra Dönmez and Agnieszka Falenska
Institute for Natural Language Processing, University of Stuttgart
Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart
esra.doenmez@ims.uni-stuttgart.de

Abstract

Large Language Models (LLMs) can generate high-quality arguments, yet their ability to engage in nuanced and persuasive communicative actions remains largely unexplored. This work explores the persuasive potential of LLMs through the framework of Jürgen Habermas’ Theory of Communicative Action. It examines whether LLMs express illocutionary intent (i.e., pragmatic functions of language such as conveying knowledge, building trust, or signaling similarity) in ways that are comparable to human communication.

We simulate online discussions between opinion holders and LLMs using conversations from the persuasive subreddit ChangeMyView. We then compare the likelihood of illocutionary intents in human-written and LLM-generated counter-arguments, specifically those that successfully changed the original poster’s view. We find that all three LLMs effectively convey illocutionary intent – often more so than humans – potentially increasing their anthropomorphism. Further, LLMs craft responses that closely align with the opinion holder’s intent, a strategy strongly associated with opinion change. Finally, crowd-sourced workers find LLM-generated counter-arguments more agreeable and consistently prefer them over human-written ones. These findings suggest that LLMs’ persuasive power extends beyond merely generating high-quality arguments. On the contrary, training LLMs with human preferences effectively tunes them to mirror human communication patterns, particularly nuanced communicative actions, potentially increasing individuals’ susceptibility to their influence.

1 Introduction

Public discourse is essential for shaping opinions, exposing individuals to diverse perspectives, and challenging existing beliefs. Today, much of this discourse takes place online, especially on social media platforms (Monti et al., 2022), where AI-generated and edited content is becoming increasingly prevalent (Hanley and Durumeric, 2024). What role does AI play in public discourse, particularly in shaping and changing opinions?

Recent research highlights concerning trends in this regard, suggesting that humans may be susceptible to undue influence from LLMs on topics ranging from politics to social and environmental issues. A key challenge is that humans often struggle to recognize AI-generated text due to cognitive shortcuts, such as associating first-person pronouns or family-related topics with human authorship (Jakesch et al., 2023). Additionally, LLMs can infer the psychological dispositions of social media users (Peters and Matz, 2024) and provide sycophantic responses (i.e., convincing, well-written responses that confirm a user’s mistaken beliefs), reinforcing confirmation bias (Sharma et al., 2024).

Consider the conversation illustrated in Figure 1 (white boxes). A user expresses their opinion on

![Discussion from /r/ChangeMyView subreddit annotated with pragmatic social dimensions (black boxes). Top: a post titled “CMV: Atheists in Western nations aren’t currently being persecuted or oppressed in any meaningful way”. Below: two human-written comments (one opinion-changing, marked with Δ). Bottom: a comment generated by Llama-2-7B.](page_1012_186_393_357.png)

Figure 1: Discussion from /r/ChangeMyView subreddit annotated with pragmatic social dimensions (black boxes). Top: a post titled “CMV: Atheists in Western nations aren’t currently being persecuted or oppressed in any meaningful way”. Below: two human-written comments (one opinion-changing, marked with Δ). Bottom: a comment generated by Llama-2-7B.
atheists in Western nations using a conflicting statement “atheists are the worst”. Two users reply: the first shares knowledge on the topic by listing facts and anecdotes, while the second simply states their opinion. Framing an argument within such illocutionary intents (social dimensions; see Deri et al. (2018) and further examples in Table 1) can influence an interlocutor’s opinion beyond the argument’s quality alone (Monti et al., 2022). Effectively, language is not just a medium for conveying information but also a tool for exercising illocutionary force (Austin, 1975) – the capacity to influence a listener’s perspective and foster cooperation based on a shared understanding of reality. This is particularly effective when building “shared knowledge, trust, and mutual understanding” (Habermas, 1984), which can drive opinion change and cooperation (Krauss and Morsella, 2000; Fronzetti Colladon et al., 2023), a process Habermas (1984) terms Communicative Action.

Monti et al. (2022) examined the role of social dimensions in online persuasive discourse and found that counter-arguments containing at least one of the dimensions are more likely to change an individual’s opinion. Additionally, matching the social dimension found in the original post – such as responding to conflict with conflict – increases the likelihood of persuasion. Clearly, social dimensions are strong indicators of persuasiveness yet remain underexplored in the context of LLMs. As exemplified in Figure 1 (bottom, blue box), models can also employ these strategies, potentially influencing readers’ opinions in ways that remain unknown. Therefore, in this work1, we examine the extent to which LLM-generated arguments contain social dimensions and how their expressed intent dynamics compare to human persuasive discourse on social media. Specifically, we aim to address the following research questions:

RQ1 Which social dimensions are present in LLM-generated arguments, and how do they compare to human-written ones?

RQ2 What are the (dis-)similarities between the social dimensions expressed in successful human-written arguments and those generated by LLMs?

RQ3 To what extent can LLMs capture interactional dynamics of social dimensions between opinion holders and successful commenters?

RQ4 Do people prefer human-written opinion-changing arguments over LLM-generated ones? And do their preferences align more closely with the strategies used by humans or by LLMs?

To address these research questions, we simulate conversations between human opinion holders and LLMs using discussions from the persuasive online forum /r/ChangeMyView (CMV). Specifically, we generate counter-arguments for CMV posts (see blue box in Figure 1) using three LLMs and compare the likelihood of these texts expressing nine social dimensions to that of human-written opinion-changing comments. We find that LLM-generated texts contain more social dimensions than human counter-arguments and consistently express trust in the opinion holder (§5.1). Examining the interaction dynamics, we observe that LLMs exhibit a capacity to model the communicative intent of the opinion holder and craft replies that mirror persuasive human strategies – in the case of GPT-3.5-turbo, even more strongly than the opinion-changing human responses (§5.3). Finally, in a direct comparison between human-written opinion-changing comments and LLM-generated ones, crowdworkers found the AI-generated responses more agreeable and overwhelmingly preferable. In fact, in 83% of cases, they judged LLM-generated arguments as more likely to change the opinion holder’s view (§5.4).

Our work makes two key contributions. First, in the field of persuasive AI, we show that LLMs’ persuasive capabilities extend beyond generating high-quality arguments. This underscores the need to study not just what AI “says” but how it engages with and adapts to human discourse, shaping opinion formation in new and unexpected ways. Second, regarding communication dynamics, we build on findings of Monti et al. (2022), extending them to AI-driven interactions. Our results suggest that human preferences are influenced by more than just the quality of arguments, highlighting the complex interplay between social intent and persuasion.

2 Related Work

Much of the literature on LLM persuasiveness is focused on humans’ perceptions of it, often comparing LLMs to humans. Karinshak et al. (2023) examined pro-vaccination messages created by both AI models and human authors, discovering that AI-generated messages were often perceived as more persuasive, except when they were explicitly la-
<table>
  <tr>
    <th>Dimension</th>
    <th>Description</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>Knowledge</td>
    <td>Exchange of ideas or information; learning, teaching</td>
    <td><i>History shows the benefits of vaccinations outweigh the risks.</i></td>
  </tr>
  <tr>
    <td>Power</td>
    <td>Having power over behavior and outcomes of another</td>
    <td><i>The only rights which exist in reality are legal rights.</i></td>
  </tr>
  <tr>
    <td>Status</td>
    <td>Conferring status, appreciation, gratitude, or admiration</td>
    <td><i>I have nothing but the utmost respect for service men.</i></td>
  </tr>
  <tr>
    <td>Trust</td>
    <td>Will of relying on the actions or judgments of another</td>
    <td><i>I understand your perspective.</i></td>
  </tr>
  <tr>
    <td>Support</td>
    <td>Giving emotional or practical aid and companionship</td>
    <td><i>I’d feel sympathy for their situation.</i></td>
  </tr>
  <tr>
    <td>Similarity</td>
    <td>Shared interests, motivations or outlooks</td>
    <td><i>I’m glad to know we agree on this.</i></td>
  </tr>
  <tr>
    <td>Identity</td>
    <td>Shared sense of belonging to the same group</td>
    <td><i>They are members of their tribe (…)</i></td>
  </tr>
  <tr>
    <td>Fun</td>
    <td>Experiencing leisure, laughter, and joy</td>
    <td><i>Realize how much you sound like Chamberlin when Hitler (…)</i></td>
  </tr>
  <tr>
    <td>Conflict</td>
    <td>Contrast or diverging views</td>
    <td><i>Atheists are the worst for this (…)</i></td>
  </tr>
</table>

Table 1: Social dimensions of language, historically recognized in the social sciences and analyzed in a survey by Deri et al. (2018). Examples are taken from the data and Monti et al. (2022).

beled as AI-generated. Similarly, Goldstein et al. (2024) found that GPT-3 could produce highly persuasive text, as measured by participants’ agreement with propaganda theses in a survey of U.S. respondents. Bai et al. (2023) conducted a randomized controlled trial exposing a diverse group of individuals to policy commentaries written by either humans or LLMs and found that both methods were equally effective in influencing participants’ policy support. Potter et al. (2024) showed that likely voters changed opinions and expressed a desire for more interaction after engaging with an LLM, even without being prompted to persuade them.

Personalization plays a key role in the persuasiveness of models, as tailoring content to align with individuals’ psychological traits can significantly influence their online behavior (Matz et al., 2017; Teeny et al., 2020). Matz et al. (2024) demonstrated that personalized messages crafted by ChatGPT were significantly more persuasive than non-personalized ones. Pauli et al. (2024) found that assigning different personas to LLMs could significantly alter the text’s persuasiveness. Similarly, Costello et al. (2024) engaged conspiracy believers in personalized, evidence-based dialogues with GPT-4-turbo and reported a reduction in conspiracy beliefs by 20%. In a controlled study by Salvi et al. (2024), concerns were raised about the implications of personalized persuasion for the governance of social media and online environments. Lastly, Sharma et al. (2024) analyzed human preference data showing that when AI responses matched users’ views, they were more likely to be preferred. Their research also suggested that both humans and preference models sometimes favored sycophantic responses over correct ones, highlighting a potential issue in AI behavior.

While a small body of research has explored whether LLMs are persuasive to humans, there is a lack of literature explaining <b>why they are persuasive, particularly in relation to established theoretical frameworks</b>. Closely related to our work, Breum et al. (2024) examined the effectiveness of LLM-generated arguments when designed to convey specific social dimensions. Their findings showed that arguments combining factual knowledge, trust markers, expressions of support, and status cues were rated as most persuasive by both humans and AI agents. Notably, humans found knowledge-based arguments particularly compelling, highlighting the importance of factual support in persuasive messaging. However, their study was conducted in a synthetic persuasion dialogue setting, leaving open the question of whether LLMs naturally incorporate social dimensions when countering real-world social media posts without being explicitly prompted. Furthermore, no direct comparison has been made between LLM-generated and human-written messages regarding the presence of social dimensions or their effects on human perceptions of persuasiveness.

3 Data

This study draws on discussions from the subreddit /r/ChangeMyView, a forum where users share their opinions and invite others to challenge them. When a commenter successfully persuades the original poster (OP) to reconsider their stance, they receive a “delta” (\( \Delta \)) as a reward, signaling a successful change of opinion (see \( \Delta \) example in Figure 1).

We use the publicly available ChangeMyView (CMV) corpus collected by Tan et al. (2016). It is a well-established dataset that underwent multiple further analyses (Hidey et al., 2017; Falenska et al., 2024, inter alia). Crucially, since the dataset consists of posts and comments written before 2016, we can be certain that none were gen-
erated by LLMs. Since we do not train models on this data, we merge the training and held-out portions for analysis, which consists of 20,626 posts and 1,260,266 comments.

3.1 Filtering Posts and Comments
The raw CMV includes deleted posts and Reddit-specific meta-level comments. To refine the data, we removed entries with missing or deleted text, missing corresponding posts or comments, or were too short to present a meaningful argument (\( \leq 2 \) words, whitespace-tokenized). After this cleaning process, we were left with 20,151 posts and 1,193,483 comments.

3.2 Extracting Sociopolitical Posts
For compatibility with Monti et al. (2022), we use only sociopolitical posts in our analysis. For filtering, we use their classifier trained on data from sociopolitical subreddits (detailed in Appendix A.1.2), obtaining 13,504 posts and 864,890 comments. Henceforth, this is the data used in the experiments.

3.3 Finding Delta Comments
/r/ChangeMyView forum employs a DeltaBot to allow OPs to award \( \Delta s \) to opinion-changing comments. Whenever an OP awards a \( \Delta \), the bot generates a comment within the same thread, marking the user who received it. Since these meta-level comments can appear at any point in the thread – when the OP decides that the author has changed their view, which may have occurred several messages earlier – we treat all parent comments by the author within the thread as the opinion-changing ones. We refer to these as \( \Delta \) **comments**, totaling 7,254 in our dataset.

4 Methods

4.1 LLMs and Counter-Argument Generation
Our study investigates whether state-of-the-art LLMs convey illocutionary intent in argumentative contexts. To enable a focused and in-depth analysis, we select three representative models – chosen for their widespread use and strong performance in generating high-quality arguments – since these are the most likely to express meaningful illocutionary intents and carry societal relevance.

LLMs We start from a selection of seven widely used LLMs, including five open-access models: **LLAMA2-chat (7B, 70B, Touvron et al. (2023))**, **LLAMA3-chat (7B, 70B, Touvron et al. (2023))**, and **MISTRAL-7B-instruct** (Jiang et al., 2023), and two API-access GPT models: **GPT-3.5-turbo** (Brown et al., 2020) and **GPT-4** (OpenAI, 2023).2

Inference and prompts We use the Hugging-Face text generation inference pipeline3 for open-access and the OpenAI text completion API4 for the API-access models. To simulate an interaction in which an LLM attempts to change an individual’s opinion, we take an original post and append “*You have one chance to change my view. Answer:*”. We then prompt the models with nucleus sample decoding (temp=.9, top_p=.6) and obtain a single counter-argument per post from each model. All generated outputs are capped at 600 tokens.

LLM selection To select the final three models for the analysis, we sub-sampled 50 sociopolitical posts from CMV and generated counter-arguments to them with all seven models. We then examined their outputs with the argument quality classifiers from Falk and Lapesa (2023). Based on the effectiveness and impact scores – two measures that quantify the persuasiveness and the degree of likability of the arguments – we selected three final models: **LLAMA2-7B**, **MISTRAL-7B-instruct**, and **GPT-3.5-turbo**. Since this analysis is not the focus of our work, we leave the details for the Appendix A.2.1. In general, the LLAMA3-chat family generated substantially lower quality arguments compared to the rest. Secondly, MISTRAL-7B-instruct overall scored the highest. Finally, LLAMA2-chat and GPT models generated arguments that are similar in quality within their families. The scores among LLAMA2-chat models were not significantly different (t-test with \( \alpha < 0.01 \)). Thus, we chose the smaller model in favor of reducing the environmental and computational costs. Similarly, GPT-4 did not generate significantly dissimilar arguments than GPT-3.5-turbo. Since it is approx. fifteen times more expensive and currently suffers from API-request timeouts due to high demand, we selected the smaller model.

4.2 Measuring Social Dimensions
To infer and compare the illocutionary intents conveyed in CMV posts, comments, and LLM-generated counter-arguments, we ground the anal-

2Accessed between Dec. 8 - 19. 2024.
3https://huggingface.co/docs/text-generation-inference
4https://platform.openai.com/docs/guides/gpt
ysis in the theoretical model of social dimensions (see examples in Table 1). Throughout this process, we directly apply the methodology from Monti et al. (2022) and below explain the required deviations.

Social dimension extraction   To extract social dimensions from texts, we use classifiers developed by Choi et al. (2020). The models estimate the likelihood that a text \( t \) conveys a social dimension \( d \) by predicting a score between 0-1 (i.e., least likely to most likely). For each text, we obtain scores for nine dimensions from their respective models and then binarize them with a 85\textsuperscript{th} percentile threshold.

Monti et al. (2022) reports that the probability of being labeled with \( d \) naturally increases with the length of the message. Thus, not to penalize the short arguments, we follow their approach and apply weight discounting to the binary dimension scores (for more details, see Appendix A.2.3).

Social intent and opinion change   To analyze the relationship between social dimensions and opinion change (indicated by the presence of \( \Delta \)) across various texts, we follow Monti et al. (2022) and use the odds ratios (OR) as the measure of the association of a dimension \( d \) (further details are available in Appendix A.2.4):

\[
\mathrm{OR}(p(d|\Delta), p(d|\overline{\Delta})) = \frac{\mathrm{odds}(p(d|\Delta))}{\mathrm{odds}(p(d|\overline{\Delta}))},
\]

where \( \mathrm{odds} = \frac{p}{1-p} \), \( \overline{\Delta} \) are comments that did not receive \( \Delta \), and \( p(d|\Delta) \) is the conditional probability that a comment contains \( d \), given that it received \( \Delta \) (similarly for \( \overline{\Delta} \)).

To compare human-written and LLM-generated comments, we use a comparable odds ratio expression: \( \mathrm{OR}(p(d|\mathrm{LLM}), p(d|\Delta)) \), where we contrast the likelihood of a dimension \( d \) appearing in LLM-generated comments versus in human-authored \( \Delta \) comments.

Finally, to assess the impact of illocutionary intent reciprocation on opinion change, and to determine whether LLMs exhibit patterns similar to those of successful human commenters (see RQ3), we measure the odds ratios for post-comment pairs (analogously for \( \overline{\Delta} \) and LLM generated texts):

\[
\mathrm{OR}(p_{\Delta}(d_i|d_j), p(d)) = \frac{\mathrm{odds}(p_{\Delta}(d_i|d_j))}{\mathrm{odds}(p(d))},
\]

where \( p_{\Delta}(d_i|d_j) \) is the conditional probability of a \( \Delta \) comment containing dimension \( d_i \), given that its corresponding post contains dimension \( d_j \), and \( p(d) \) is the prior probability of a message being labeled with dimensions \( d \). We use \( p(d) \) as an offset representing the baseline probability that a post with a dimension \( d_j \) receives a comment with dimension \( d_i \) at random – essentially estimating the chance of that dimension occurring in a message at random. We report the variation of the probability of achieving \( \Delta \) given a combination of dimensions (analogously to compare \( p(d_i|d_j) \) between the LLM-generated and \( \Delta \) comments) only for the statistically highly significant odds ratio variations (\( P < 0.01 \)) calculated as \( \mathrm{OR}(p_{\Delta}(d_i|d_j), p(d)) - \mathrm{OR}(p_{\overline{\Delta}}(d_i|d_j), p(d)) \).

![Bar charts showing % of comments containing a particular dimension and % of comments containing a certain number of dimensions for CMV, CMV^A, Llama-2-7B, Mistral-7B, and GPT-3.5-turbo](page_1092_120_367_627.png)

Figure 2: Statistics of comments written by CMV users and generated by LLMs.

In the following, we analyze social dimensions in counter-argument comments generated by three selected models: LLAMA2-7B, MISTRAL-7B-instruct, and GPT-3.5-turbo.
<table>
  <tr>
    <th></th>
    <th>Odds ratio</th>
    <th>Odds ratio</th>
    <th>Odds ratio</th>
    <th>Odds ratio</th>
  </tr>
  <tr>
    <td>knowledge</td>
    <td>1.84</td>
    <td>375.31</td>
    <td>434.93</td>
    <td>611.28</td>
  </tr>
  <tr>
    <td>similarity</td>
    <td>1.58</td>
    <td>6.32</td>
    <td>6.64</td>
    <td>6.67</td>
  </tr>
  <tr>
    <td>nodimensions</td>
    <td>0.60</td>
    <td>2.54</td>
    <td>2.65</td>
    <td>2.59</td>
  </tr>
  <tr>
    <td>identity</td>
    <td>0.57</td>
    <td>2.13</td>
    <td>2.21</td>
    <td>2.20</td>
  </tr>
  <tr>
    <td>fun</td>
    <td>0.55</td>
    <td>1.98</td>
    <td>2.00</td>
    <td>1.99</td>
  </tr>
  <tr>
    <td>conflict</td>
    <td>0.50</td>
    <td>1.73</td>
    <td>1.75</td>
    <td>1.72</td>
  </tr>
  <tr>
    <td>power</td>
    <td>0.42</td>
    <td>1.68</td>
    <td>1.71</td>
    <td>1.70</td>
  </tr>
  <tr>
    <td>status</td>
    <td>0.36</td>
    <td>0.61</td>
    <td>0.62</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>trust</td>
    <td>0.19</td>
    <td>0.53</td>
    <td>0.53</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>support</td>
    <td>0.15</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>nodimensions</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
</table>

(a) \( \Delta \) vs. \( \overline{\Delta} \)   (b) LLAMA2-7B vs. \( \Delta \)   (c) MISTRAL-7B vs. \( \Delta \)   (d) GPT-3.5-turbo vs. \( \Delta \)

Figure 3: Odds ratios between probabilities of social dimensions present in texts measured as in Equation (1). (a) comparison of human-written \( \Delta \) and \( \overline{\Delta} \) comments; (b), (c), (d) comparison of LLM-generated and \( \Delta \) comments. Error bars represent the 95% confidence intervals (see Equation (8) and details in Appendix A.2.4).

5.1 Social Dimensions in LLM Comments

We start by addressing RQ1: Which social dimensions are present in LLM-generated arguments? Figure 2a illustrates the percentage of messages containing a given dimension across different sets. First, human-written comments (yellow bars) show relatively balanced distributions across dimensions, indicating that CMV users do not favor any particular social dimension-based rhetorical strategy. However, the \( \Delta \) comments (green bars) more frequently contain knowledge and similarity – a result that corroborates Monti et al.’s (2022) findings.

In contrast, all three LLMs overwhelmingly express trust – often starting with phrases like I understand your perspective – with nearly all generated messages incorporating this dimension. This finding sharply differentiates LLM-generated texts from \( \Delta \) comments, where trust is the second least frequent dimension. Beyond overusing trust, LLMs also produce responses that incorporate more social dimensions within a single message. Figure 2b shows that most human comments (yellow and green bars) tend to rely on a single dimension. Moreover, around 23% contains no identifiable dimensions, while only about 2% includes four. In contrast, generated messages consistently include at least one dimension, with some containing as many as six.

In summary, models not only heavily rely on markers of trust but also convey consistently more social intent than humans – a pattern that remains remarkably consistent across all three LLMs.

5.2 LLMs vs. Opinion-changing Comments

Having confirmed the presence of social dimensions in LLM-generated arguments, we explore further differences between these texts and human-written opinion-changing comments (RQ2).

\( \Delta \) vs. \( \overline{\Delta} \) comments We start by identifying the social dimensions vital for opinion-changing human arguments. Figure 3a presents odds ratios (see Equation (1)) between the \( \Delta \) and \( \overline{\Delta} \) comments. We observe that \( \Delta \) comments are more likely to convey knowledge (84%)5 and similarity (58%) compared to \( \overline{\Delta} \). Conversely, comments that do not contain any dimension (nodimensions) are 40% less likely to change the opinion holder’s view.6

LLMs vs. \( \Delta \) comments We shift the focus back to LLMs and compare generated arguments with \( \Delta \) comments in Figures 3b to 3d. The odds of dimensions present in model-generated messages across all three models are highly similar. Consistent with the findings from the previous section (see trust in Figure 2), all three LLMs are far more likely to express trust than \( \Delta \) comments. Similarly, models are highly unlikely to generate arguments that do not convey any social dimension. Compared to the \( \Delta \) comments, models are more likely to convey support, status, power, conflict, identity, and fun by approx. 554%, 159%, 118%, 99%, 73%, 70% respectively. Conversely, they are less likely to convey similarity and knowledge by 39% and 47%.

5A ratio of 1 shows no difference, while a ratio of 1.84 reads as 84% more likely.
6The differences from Monti et al. (2022) likely arise from a different selection of CMV posts, yet the main findings remain consistent.
Figure 4: Variation of the probability of expressing a particular combination of dimensions in post-comment pairs measured by taking the difference of odds ratios calculated in Equation (2) between Δ and \( \overline{\Delta} \) comments.

In conclusion, LLM-generated comments convey more illocutionary intent within a single comment than human-written ones and exhibit similar distributions of social dimensions, with the exception of trust. However, they differ markedly from successful (\( \Delta \)) human-written comments. In particular, LLMs are less likely to express the two dimensions most strongly associated with opinion change: knowledge and similarity.

5.3 Social Intent Dynamics

To what extent can LLMs capture interactional dynamics of social dimensions between opinion holders and successful commenters? (RQ3)? To answer this question, we first establish the dynamics in human-human conversations. We then contrast characteristics of LLM-generated comments with human-written opinion-changing comments.

Human-human conversations Figure 4 shows the differences between the \( \Delta \) and \( \overline{\Delta} \) post-comment pairs measured by taking the difference of odds ratios calculated in Equation (2). We observe a similar phenomenon as reported by Monti et al. (2022), i.e., comments that make similar appeals to that of their corresponding posts are more likely to receive a \( \Delta \) (blue cells on the diagonal in the figure). We observe this pattern for three out of nine dimensions: if a comment conveys knowledge, appeals to power or similarity, it is more likely to receive a \( \Delta \) by 15%, 45%, and 13% resp. Moreover, if a comment expresses trust in response to a post appealing to identity or knowledge in response to no dimensions, it is more likely to be awarded a \( \Delta \) by 38% and 42%, resp. Conversely, if a post appeals to similarity, a comment containing no dimensions is 17% less effective.

LLM-generated arguments Figure 5 presents the variation of the probability of expressing a particular combination of dimensions between LLM-generated texts and \( \Delta \) comments (note that the trust dimension is removed due to its high occurrence and to prevent visual clutter). The highlighted areas show where model-generated comments significantly differ from the \( \Delta \) comments: in five dimension pairs for LLAMA2-7B, eight for MISTRAL-7B-instruct, and thirteen for GPT-3.5-turbo. For LLAMA2-7B, the odds of the model conveying similarity in its reply to a post with the same dimension is 22% more likely than the CMV \( \Delta \) comments. The opposite happens for power where the model is less likely to make a similar appeal by 47%. Interestingly, MISTRAL-7B-instruct, the only non-chat model in this study, is the most dissimilar to human social dimension dynamics. The model is overall less likely to reply with certain dimensions than the \( \Delta \) comments in 8 out of 81 pairs of dimensions.

Focusing on the diagonal – i.e., whether comments reciprocate the social dimensions expressed in the original posts – we find that model-generated comments exhibit patterns similar to those in successful (\( \Delta \)) human-written comments. For LLAMA2-7B and GPT-3.5-turbo, there is no statistically significant difference in six out of nine cases, and for MISTRAL-7B-instruct, in eight out of nine. On average, LLAMA2-7B and MISTRAL-7B-instruct tend to under-reciprocate compared to \( \Delta \) comments, whereas GPT-3.5-turbo shows a notable deviation: it is significantly more likely to reciprocate social dimensions – support (by 103%), knowledge (69%), and similarity (50%) – than human \( \Delta \) comments. Furthermore, GPT-3.5-turbo is also more likely to appeal to similarity in response to support (by 89%) and conflict (by 28%).

To sum up, models not only differ from \( \Delta \) comments but also from each other when it comes to the post-comment dynamics of social dimensions. Among them, GPT-3.5-turbo exhibits stronger reciprocity by aligning more closely with the post’s intent. This suggests that if individuals are, in fact, more likely to change their view when a challenger demonstrates similar illocutionary intent in their counter-argument, then they might be sim-
Figure 5: Variation of the probability of expressing a particular combination of dimensions in post-comment pairs measured by taking the difference of odds ratios calculated in Equation (2) between LLM-generated and human-written \( \Delta \) comments, displayed for statistically highly significant variations (P < 0.01).

ilarly (or even more) likely to be convinced by GPT-generated arguments.

5.4 Human Preferences

We observed significant differences in how CMV users and LLMs convey illocutionary intent. Do these differences correlate with human preferences (RQ4)? To answer this question, we focus on the two differences that significantly stood out throughout our analyses so far: the overwhelming presence of the trust dimension in the LLM-generated texts and the comparably lower use of knowledge – a dimension strongly associated with opinion change.

Method Since it is not possible to ask the original CMV posters if they would change their opinion at the time when they wrote their posts, we approximate this situation through crowdsourcing. In an exploratory study, we showed human annotators an original CMV post and first asked whether they agreed with its stance. Then, they were presented with two counter-arguments – one human-written \( \Delta \) comment and one LLM-generated – and asked whether they agreed with any of them. Annotators could indicate agreement or disagreement with a post or a comment independently of their agreement with the other. Finally, they were asked to select the message more likely to change the opinion holder’s view.

We annotated 100 triples (CMV post, \( \Delta \) comment, LLM comment), with each reviewed by three annotators. The triples were sampled to equally represent trust and knowledge dimensions (25 for each combination from Table 2). Finally, since GPT-3.5-turbo stands out most in its expression of social intent dynamics (see §5.3) and is among the most widely used language models, we selected arguments generated by this model for our study.

Annotators were given guidelines, including a tutorial on how to perform the task. The annotation was performed on prolific7, and the worker pool consisted of U.S. residents8. Each annotator annotated eight instances and got compensated £12/h (more details on crowdsourcing in Appendix B).

Results Overall, crowdworkers found GPT-3.5-turbo-generated messages more persuasive, selecting them as more likely to change the opinion holder’s view in 83% of cases according to the majority vote. The agreement among the annotators was measured at 0.79 using Krippendorff’s alpha.

<table>
  <tr>
    <th rowspan="2">Dimension</th>
    <th colspan="2">\( \Delta \) Comments</th>
  </tr>
  <tr>
    <th>Knowledge</th>
    <th>Trust</th>
  </tr>
  <tr>
    <td>GPT-Generated</td>
    <td>Trust</td>
    <td>0.62</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td></td>
    <td>Knowledge and Trust</td>
    <td>0.81</td>
    <td>1.00</td>
  </tr>
</table>

Table 2: Krippendorf’s alpha scores as inter-annotator agreement obtained from three binary preference ratings per dimension pairs (25 pairs of a \( \Delta \) from CMV and GPT-3.5-turbo-generated comment per category).

7https://www.prolific.com/
8Crowdworkers were fluent English speakers aged 18 or older. They represented diverse ethnicities and countries of birth, and the sample was gender-balanced between female and male – the only gender variables available on the platform.
We further analyzed these results from two angles. First, we found that the preference for opinion-changing arguments was not affected by the combination of knowledge and trust. That is, whether both the human and model responses relied solely on trust, or the model’s trust-based argument was compared to a human argument grounded in knowledge, the preference ratio remained largely consistent. Additionally, we observed a higher Krippendorff’s alpha for samples where model responses included both knowledge and trust, compared to those based on trust alone, which indicates stronger agreement among annotators (see Table 2).

Second, we examined the relationship between annotators’ opinions on the comment stances and their preferences. In 58 cases, the crowdworkers indicated a (dis-)agreement with both counter-arguments while still, on average, preferring generated ones. In the majority of the remaining cases (37 out of 42 posts), annotators selected the counter-argument they also agreed with, suggesting a potential preference bias. Interestingly, in 84% of these cases (31 out of 37), this selected and preferred comment was LLM-generated, implying that the crowdworkers found GPT-generated messages also more agreeable (majority agreement with 92 model comments and 66 human Δ comments, see Table 4).

6 Conclusions and Discussion

As LLM-generated content becomes increasingly prevalent online, understanding its influence on human opinions becomes essential. To move towards this understanding, we examined LLM persuasiveness through the lens of Communicative Action – a theoretical framework that views language as a means of expressing social intent through reasoned dialogue aimed at achieving mutual understanding, which is crucial for human opinion change.

Using real-world persuasive discourse as a reference point, we showed that models effectively convey illocutionary intent – often more frequently and densely than humans. Notably, all three models we analyzed consistently express trust in their conversation partners, potentially affirming their views and contributing to perceived biased likability (Sharma et al., 2024). While all three models show similar patterns of social intent reciprocity to that of the successful human comments, the post-comment interactions of GPT-3.5-turbo-generated arguments reveal even stronger patterns of reciprocity – a behavior closely linked to opinion change (Monti et al., 2022).

Finally, LLMs employ rhetorical strategies that differ from those found in human-written opinion-changing arguments – especially in conveying the two dimensions previously linked to successful persuasion: knowledge and similarity. Importantly, crowdworkers not only find these LLM-generated arguments more agreeable but also consistently prefer them over human-written opinion-changing ones. While further research is needed to identify which dimensions drive their decision the most, our findings show that LLMs are capable of engaging in nuanced communicative actions, potentially shaping human opinions in ways not yet fully understood. Moreover, the varying patterns observed between instruction and chat models suggests that alignment training for LLMs may not only reinforce existing human biases but also increase individuals’ susceptibility to AI-driven influence on critical issues like politics, social justice, and the environment. In light of these concerns, we urge further research into alignment training and its broader effects on public discourse and opinion formation.

7 Limitations

We simulate exchanges between opinion holders and LLMs using data from the /r/ChangeMyView subreddit. Discussions on other online platforms and real one-to-one conversations with LLMs might have different characteristics, highlighting an open area for further research.

Additionally, this study has some constraints related to the exploratory annotation experiment. First, humans write texts with various lengths, some substantially shorter or longer than the average. In comparison, the artificial setup in which the LLM-generated texts have a fixed maximum length might make them more or less expressive than human-written comments. Although this does not affect our findings in the main study, as we normalize probabilities by length, it might affect human reading comprehension and perception. In other words, arguments might have different writing styles depending on the length, potentially influencing human preferences. To account for these effects, we sampled both human-written and LLM-generated comments with varying lengths.

Second, we do not control for potential topic confounders in the annotation study. Discussion topics may influence both the model-generated texts and
the crowdworkers’ judgments, as annotators are biased in agreeing with messages that align with their prior beliefs. Our randomly sampled annotation instances contain various topics to account for this. To account for this, our annotation instances were randomly sampled from a variety of topics. However, the impact of topic confounders on opinion change remains an open area for future research.

Finally, in our annotation study, we ask crowdsourced workers whether they believe a given argument could change an individual’s opinion. Since these are not their own posts, they can only speculate on the argument’s persuasiveness. To mitigate this limitation, we also ask annotators to evaluate their own agreement with the post and comments and include this dimension in the analysis.

8 Ethical Considerations

Understanding why LLMs are persuasive to humans has the potential dual-use risk. The developers of such models can use our findings to train LLMs to be more persuasive to specific target groups for harmful purposes. We not only advise against such efforts but also advocate for detecting and preventing manipulative model deployment.

9 Acknowledgements

We acknowledge the support of the Ministerium für Wissenschaft, Forschung und Kunst BadenWürttemberg (MWK, Ministry of Science, Research and the Arts Baden-Württemberg under Az. 33-7533-9 19/54/5) in Künstliche Intelligenz & Gesellschaft: Reflecting Intelligent Systems for Diversity, Demography and Democracy (IRIS3D) and the support by the Interchange Forum for Reflecting on Intelligent Systems (IRIS) at the University of Stuttgart. We would like to thank Eva Maria Vecchi for her valuable feedback during the early stages of this work, and Neele Falk for her help in selecting appropriate metrics for evaluating argument quality.

References

John L. Austin. 1975. How To Do Things With Words: The William James Lectures delivered at Harvard University in 1955. Oxford University Press.

Hui Bai, Jan G Voelkel, Johannes C Eichstaedt, and Robb Willer. 2023. Artificial intelligence can persuade humans on political issues.

Simon Martin Breum, Daniel Vædele Egdal, Victor Gram Mortensen, Anders Giovanni Møller, and Luca Maria Aiello. 2024. The persuasive power of large language models. Proceedings of the ... International AAAI Conference on Weblogs and Social Media/Proceedings of the International AAAI Conference on Weblogs and Social Media.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models Are Few-Shot Learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc.

Minje Choi, Luca Maria Aiello, Krisztian Zsolt Varga, and Daniele Quercia. 2020. Ten social dimensions of conversations and relationships. In Proceedings of The Web Conference 2020, WWW ’20, page 1514–1525, New York, NY, USA. Association for Computing Machinery.

Thomas H. Costello, Gordon Pennycook, and David G. Rand. 2024. Durably reducing conspiracy beliefs through dialogues with ai. Science.

Sebastian Deri, Jeremie Rappaz, Luca Maria Aiello, and Daniele Quercia. 2018. Coloring in the links: Capturing social ties as they are perceived. Proc. ACM Hum.-Comput. Interact., 2(CSCW).

Agnieszka Falenska, Eva Maria Vecchi, and Gabriella Lapesa. 2024. Self-reported demographics and discourse dynamics in a persuasive online forum. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14606–14621, Torino, Italia. ELRA and ICCL.

Neele Falk and Gabriella Lapesa. 2023. Bridging argument quality and deliberative quality annotations with adapters. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2469–2488, Dubrovnik, Croatia. Association for Computational Linguistics.

Andrea Fronzetti Colladon, Laura Toschi, Elisa Ughetto, and Francesca Greco. 2023. The language and social behavior of innovators. Journal of Business Research, 154:113317.

Josh A Goldstein, Jason Chao, Shelby Grossman, Alex Stamos, and Michael Tomz. 2024. How persuasive is ai-generated propaganda? PNAS Nexus, 3(2):pgae034.

Juergen Habermas. 1984. The Theory of Communicative Action, Vol. 1, 'Reason and the Rationalization of Society'. Polity.
Hans W. A. Hanley and Zakir Durumeric. 2024. Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites. In Proceedings of the Eighteenth International AAAI Conference on Web and Social Media, ICWSM 2024, Buffalo, New York, USA, June 3-6, 2024, pages 542–556. AAAI Press.

Christopher Hidey, Elena Musi, Alyssa Hwang, Smaranda Muresan, and Kathy McKeown. 2017. Analyzing the semantic types of claims and premises in an online persuasive forum. In Proceedings of the 4th Workshop on Argument Mining, pages 11–21, Copenhagen, Denmark. Association for Computational Linguistics.

Maurice Jakesch, Jeffrey T. Hancock, and Mor Naaman. 2023. Human heuristics for ai-generated language are flawed. Proceedings of the National Academy of Sciences of the United States of America.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.

Elise Karinshak, Sunny Xun Liu, Joon Sung Park, and Jeffrey T. Hancock. 2023. Working ai to persuade: Examining a large language model’s ability to generate pro-vaccination messages. Proc. ACM Hum.-Comput. Interact., 7(CSCW1).

Robert M. Krauss and Ezequiel Morsella. 2000. Communication and conflict. In The handbook of constructive conflict resolution: Theory and practice, pages 131–143. San Francisco: Jossey-Bass.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.

Sandra Matz, Jake Teeny, Sumer S. Vaid, H. Peters, Gabriella M. Harari, and Moran Cerf. 2024. The potential of generative ai for personalized persuasion at scale. Scientific Reports.

Sandra C. Matz, Michal Kosinski, Gideon Nave, and David J. Stillwell. 2017. Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the National Academy of Sciences of the United States of America.

Corrado Monti, Luca Maria Aiello, Gianmarco De Francisci Morales, and Francesco Bonchi. 2022. The language of opinion change on social media under the lens of communicative action. Scientific Reports, 12(1):1–11.

OpenAI. 2023. GPT-4 Technical Report.

Amalie Brogaard Pauli, Isabelle Augenstein, and Ira Assent. 2024. Measuring and benchmarking large language models’ capabilities to generate persuasive language. arXiv.org.

Heinrich Peters and Sandra Matz. 2024. Large language models can infer psychological dispositions of social media users. PNAS nexus.

Yujin Potter, Shiyang Lai, Junsol Kim, James Evans, and Dawn Song. 2024. Hidden persuaders: LMs’ political leaning and their influence on voters. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallootti, and Robert West. 2024. On the conversational persuasiveness of large language models: A randomized controlled trial.

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations.

Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of WWW.

Jacob D. Teeny, Joseph J. Siev, P. Briñol, and R. Petty. 2020. A review and conceptual framework for understanding personalized matching effects in persuasion. Journal of Consumer Psychology.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykhov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Bleicher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harthorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rash Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models.
A Data and Methods

A.1 Data
This section provides additional details about the CMV forum and the filtering process for sociopolitical posts.

A.1.1 CMV Forum
The subreddit CMV functions similarly to other Reddit forums. Users initiate discussions by posting a viewpoint in the title and elaborating on it in the body text, which may include external links to websites, images, or other resources. The platform is designed for opinion exchange: once a post is live, any Reddit user can engage with it by voting or commenting (a Reddit account is required to do so). Users can also reply to comments, forming a tree-structured discussion. A distinctive feature of CMV is the ability for the original poster to award a \( \Delta \) flag to comments that successfully changed their view.

Note that, due to the nature of online discourse, the CMV dataset may contain offensive content. However, this content was not created by us, nor do we employ any methods that endorse or promote such material.

A.1.2 Sociopolitical Classifier
The classifier ([1,2,3]-gram logistic regression model) has been trained on 10,000 posts – 50 posts from each 51 sociopolitical subreddit and later filtered between the years from 2011 to the end of 2019 – and validated on an equal size test set with an average performance of 89.5% (more details in Monti et al. (2022)).

A.2 Methods
This section provides additional details on model selection, LLM-based counter-argument generation, social dimension classification, and the metrics.

A.2.1 LLM Selection
Argument quality evaluation Falk and Lapesa (2023) use RoBERTa (roberta-base, Liu et al. (2019)) as the backbone transformer and train an adapter per dimension (20 single-task adapters). Using these adapters, we obtain scores for seven dimensions relevant to our purposes shown in Table 3. These scores help us assess the quality of the generated arguments across models.

Figure 6 displays the argument quality scores of all seven models on 50 subsample of CMV sociopolitical posts (CMV\(^{50}\)). The metrics can be found in Table 3. The higher the score, the better the argument quality. The black bars display the variance in quality across inference settings (nucleus sample decoding with temp=.9 and top_p=.6, and greedy decoding). As mentioned earlier, the maximum number of tokens is 600.

Results LLAMA3-chat models generate substantially lower quality arguments compared to the rest. Out of the remaining top five, the overall winner is the MISTRAL-7B-instruct. The scores from LLAMA2-chat models are not significantly different (t-test with \( \alpha < 0.01 \)). Though significantly different on several dimensions (quality, clarity, cogency, reasonableness, and overall), GPT-4 does not generate significantly dissimilar arguments in effectiveness and impact, which quantify the persuasiveness and the degree of likability of the arguments. Moreover, this model is approx. 15 times more expensive and currently suffers from API-request timeouts due to high demand, making it approx. 9 times slower than GPT-3.5-turbo.

A.2.2 LLM Data
No prompt engineering or optimization were employed in this study. For LLAMA2-7B and MISTRAL-7B-instruct, we use the HuggingFace text generation pipeline at https://huggingface.co/docs/text-generation-inference, and we access GPT-3.5-turbo via OpenAI text completion API at https://platform.openai.com/docs/guides/gpt (accessed between Dec. 8 - 19. 2024). All LLM comments were generated following the nucleus sample decoding introduced in Appendix A.2.1.

A.2.3 Extracting Social Dimensions
Choi et al. (2020) previously developed classifiers to estimate the likelihood that a message conveys a dimension \( d \). Specifically, they trained a binary classifier (an LSTM) per dimension (\( C_d \)) on sentences to predict \( p(d) \), which altogether can be seen as a multi-label classifier, as any sentence might convey several dimensions. For the CMV comments (analogously for LLM-generated text), we feed the input into the model a sentence at a time and obtain scores for each sentence in the text. We take the maximum score as the final output, specifically \( s_d(m) = \max_{S \in m} s_d(S) \), which then means that a message is considered to express a dimension as likely as the most likely sentence in the message.
<table>
  <tr>
    <th>Dimension</th>
    <th>Description</th>
    <th>Score</th>
  </tr>
  <tr>
    <td>Quality</td>
    <td>General argument quality score</td>
    <td>(0-1)</td>
  </tr>
  <tr>
    <td>Clarity</td>
    <td>Is it hard or easy to interpret the argument?</td>
    <td>(0-1)</td>
  </tr>
  <tr>
    <td>Impact</td>
    <td>User likes / recommendations multi-class (3 classes)</td>
    <td>(0-1)</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>General argument quality score</td>
    <td>(1-5)</td>
  </tr>
  <tr>
    <td>Cogency</td>
    <td>Acceptable and sufficient premises to draw a conclusion score</td>
    <td>(1-5)</td>
  </tr>
  <tr>
    <td>Reasonableness</td>
    <td>Contribution to the resolution of issues, the argument is accepted by universal audience score</td>
    <td>(1-5)</td>
  </tr>
  <tr>
    <td>Effectiveness</td>
    <td>Persuasion, rhetorical, emotional appeal score</td>
    <td>(1-5)</td>
  </tr>
</table>

Table 3: Argument quality dimensions and their respective score ranges in models from Falk and Lapesa (2023).

![Bar chart showing normalized argument quality results for seven models across six dimensions](page_374_682_1002_312.png)

Figure 6: Argument quality results from all seven models on CMV^{50}, normalized to range (0, 1) and averaged between the two inference settings (nucleus sampling and greedy decoding) with black bars representing the variance in scores between the decoding strategies.

Binarization and weight discounting For consistency with prior work from Monti et al. (2022), we do not majorly alter the notations in the following equations.

A message is considered to convey a dimension \( d \) if \( s_d(m) \) is above a threshold of \( \theta_d \):

\[
d(m) = \begin{cases}
1, & \text{if } s_d(m) \geq \theta_d \\
0, & \text{otherwise}
\end{cases}
\]

Finally, the weight discounted dimension score (\( d(m) \)) is: \( \frac{1}{1+zlen_d(m)} \) if \( s_d(m) \geq \theta_d \land zlen_d(m) \geq 0 \), \( 2 - \frac{1}{1-zlen_d(m)} \) if \( s_d(m) \geq \theta_d \land zlen_d(m) < 0 \), and 0 if \( s_d(m) < \theta_d \).

A.2.4 Measuring Social Intent and Opinion Change

The length-discounted prior probability of a message (post, human-written, or LLM-generated comment) being labeled with dimensions d is

\[
p(d) = \frac{\sum_{m \in M} d(m)}{2|M|},
\]

where \( M \) is the set of messages, and the factor of 2 is used to limit \( p(d) \) between 0 and 1, as the values of \( d(m) \) range from 0 to 2.

The conditional probability that a comment contains \( d \), given that it received a \( \Delta \) is

\[
p(d|\Delta) = \frac{\sum_{m \in C_\Delta} d(m)}{2|C_\Delta|},
\]

where \( C_\Delta \) is the set of \( \Delta \) comment (analogously for \( \overline{\Delta} \) and \( LLM \)). The odds ratio between \( d \) in \( \Delta \) and \( \overline{\Delta} \) is defined in Equation (1):

\[
\mathrm{OR}(p(d|\Delta), p(d|\overline{\Delta})) = \frac{\mathrm{odds}(p(d|\Delta))}{\mathrm{odds}(p(d|\overline{\Delta}))},
\]

where \( \mathrm{odds} = \frac{p}{1-p} \) and \( p(d|\Delta) \) is the conditional probability that a comment contains \( d \), given that it received \( \Delta \) (similarly for \( \overline{\Delta} \)). To compare human- and LLM-written comments, we use analogous \( \mathrm{OR}(p(d|LLM), p(d|\Delta)) \) as:

\[
\mathrm{OR}(p(d|LLM), p(d|\Delta)) = \frac{\mathrm{odds}(p(d|LLM))}{\mathrm{odds}(p(d|\Delta))}.
\]

The conditional probability of a comment containing dimension \( d_i \), given that its corresponding post contains dimension \( d_j \) in the set of \( \Delta \) (and equivalently for \( \overline{\Delta} \) and \( LLM \)) comments is

\[
p_{\Delta}(d_i|d_j) = \frac{\sum_{c \in C_\Delta(P_{d_j})} d_i(c)}{2|C_\Delta(P_{d_j})|}.
\]
The odds ratio between the conditional probabilities between \( \Delta \) and all CMV comments (\( \Delta \) and \( \overline{\Delta} \)) is defined in Equation (2):

\[
\mathrm{OR}(p_{\Delta}(d_i|d_j), p(d)) = \frac{\mathrm{odds}(p_{\Delta}(d_i|d_j))}{\mathrm{odds}(p(d))},
\]

with a 95% confidence interval defined as

\[
ci = 1.96 \sqrt{\frac{1}{|C_{d,\Delta}|} + \frac{1}{|C_{d,\overline{\Delta}}|}}.
\] (8)

Analogously, the odds ratio between the conditional probabilities LLM-generated and all CMV comments (\( \Delta \) and \( \overline{\Delta} \)) is defined as

\[
\mathrm{OR}(p_{LLM}(d_i|d_j), p(d)) = \frac{\mathrm{odds}(p_{LLM}(d_i|d_j))}{\mathrm{odds}(p(d))},
\] (9)

where \( p(d) \) is the prior probability of dimensions in CMV comments, and the confidence interval used in Equation (9) includes this additional term \( \frac{1}{|C_{d,LLM}|} \) to control for the model-generated texts.

B  Crowdsourcing

This section provides details about the crowdsourcing.

Annotation  Annotators were shown a single annotation instance in each page – a post with its title, a human-written \( \Delta \) comment, and a GPT-3.5-turbo-generated comment. Below each post, we asked “Do you agree with this message?” with options “Yes” and “No”. The preselected “N/A” was not a valid answer, i.e., the annotators could not move to the next page unless they answered the questions. Comments were displayed in collapsible sections, with the same question and answer options below each comment. At the bottom of each page, we asked the annotators “Which of the messages is more likely to change the opinion holder’s view?” Annotators had to select “Message 1” or “Message 2”. Importantly, the comments were displayed in random orders, meaning that we randomly rotated which comment would be displayed first, to avoid biasing the annotators.

As required by the survey platform, the annotators were given two simple attention checks. The crowdworkers who failed the attention checks have been excluded from the study.

<table>
  <tr>
    <th rowspan="2">Agreed with</th>
    <th rowspan="2"># Cases</th>
    <th colspan="2">Preferred</th>
  </tr>
  <tr>
    <th>\( \Delta \) Comment</th>
    <th>GPT Comment</th>
  </tr>
  <tr>
    <td>Both / Neither</td>
    <td>58</td>
    <td>9</td>
    <td>49</td>
  </tr>
  <tr>
    <td>\( \Delta \) Comment</td>
    <td>9</td>
    <td>6</td>
    <td>3</td>
  </tr>
  <tr>
    <td>GPT Comment</td>
    <td>33</td>
    <td>2</td>
    <td>31</td>
  </tr>
</table>

Table 4: Annotator preferences based on their agreement with the comment from \( \Delta \) or GPT-3.5-turbo. Note that the annotator can agree with both arguments in a single instance as they do not choose which one they agree with but indicate whether they agree with each text (post, human-written \( \Delta \) comment, and GPT-3.5-turbo-generated comment).

Compensation  The crowdworkers were compensated at a rate of £12/h, which is well above the federal minimum wage for covered nonexempt employees of £5.76/h (or $7.25 USD).

Annotator demographics  The worker pool consisted of U.S. residents who were eighteen years old or above and fluent in English (with a fluency level of either fluent or native speaker). The annotators were balanced by gender (20 female and 20 male) and consisted of both U.S. nationals and immigrants with diverse ethnic – 6 Asian, 11 Black, 18 White, 3 Mixed, 2 non-disclosed – educational, and employment backgrounds. Countries of birth included predominantly USA, with one or two participants from countries such as Libya, Nigeria, India, Germany, Mexico, Ghana, Algeria, Philippines, and China. They were warned about potentially triggering conversation topics such as violence, discrimination, or suicide before asking for consent to continue the annotation with an option to drop out of the study and withdraw consent at any point.

C  Further Results

This section provides further results.

LLMs vs. CMV Comments  Figure 7 presents the odds ratios comparing the likelihood that LLMs, relative to CMV commenters (\( \Delta \) and \( \overline{\Delta} \)), generate texts expressing a given social dimension \( d \). Notably, the models are substantially more likely than human commenters to express trust toward their interlocutors and are rarely observed to not express any illocutionary intent.

While differences are less pronounced across the remaining dimensions, notable disparities can be observed in support, fun, status, and power. These dimensions exhibit the largest gaps among the remaining dimensions, with some variation across
models. For instance, *support*, *fun*, and *status* are approximately 8% less likely to be generated by the chat-optimized models (LLAMA2-7B and GPT-3.5-turbo), whereas this trend is not as evident for the instruction-tuned model (MISTRAL-7B-instruct). This contrast suggests that alignment training may play a role in shaping the communicative behaviors exhibited by different LLMs.
<table>
  <tr>
    <th></th>
    <th>Odds ratio</th>
    <th>Odds ratio</th>
    <th>Odds ratio</th>
  </tr>
  <tr>
    <td>trust</td>
    <td>73.33</td>
    <td>84.98</td>
    <td>73.33</td>
  </tr>
  <tr>
    <td>identity</td>
    <td>0.99</td>
    <td>1.00</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>conflict</td>
    <td>0.98</td>
    <td>1.00</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>knowledge</td>
    <td>0.97</td>
    <td>0.98</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>similarity</td>
    <td>0.96</td>
    <td>0.98</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>support</td>
    <td>0.93</td>
    <td>0.98</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>fun</td>
    <td>0.92</td>
    <td>0.95</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>status</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>power</td>
    <td>0.90</td>
    <td>0.94</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>nodimensions</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
</table>

(a) LLAMA2-7B vs. CMV        (b) MISTRAL-7B vs. CMV        (c) GPT-3.5-turbo vs. CMV

Figure 7: Odds ratios between probabilities of social dimensions present in texts measured as in Equation (1) – comparison of LLM-generated and CMV comments (\( \Delta \) and \( \overline{\Delta} \)). Error bars represent the 95% confidence intervals (see Equation (8) and details in Appendix A.2.4). The difference between the LLM-generated and all human-written comments is minimal except for the trust and nodimensions.

Reviewer paper summaries: This paper explores the persuasive capabilities of LLMs through Jürgen Habermas’ Communicative Action Theory. Using discussions from the ChangeMyView subreddit, this study simulates conversations between opinion holders and LLMs. They select three LLMs based on the results of the pre-experiment to generate counter-arguments and compare them with human-written opinion-changing comments. By measuring nine social dimensions in the texts, they find that LLMs effectively convey illocutionary intent, often incorporating more social dimensions than human arguments. Crowdsourced workers also prefer LLM-generated arguments. The study shows that LLMs' persuasive power extends beyond argument quality, and LLMs can engage in nuanced communicative actions, potentially shaping human opinions in ways not yet fully understood.

This paper examines the persuasive abilities of Large Language Models through the application of Communicative Action Theory. Using discussions from the persuasive subreddit ChangeMyView, the authors compare human-written and LLM-generated opinion-changing counterarguments. The study finds that LLM-generated texts typically exhibit stronger and more numerous social dimensions compared to human arguments, with GPT-3.5-turbo crafting responses that align closely with the original poster's intent, a strategy strongly associated with changing opinions. Crowd-sourced annotators also favored LLM-generated arguments, finding them more agreeable and persuasive than human-written responses.

The paper characterizes the persuasive capabilities of LLMs through the lens of Communicative Action. The work compares and contrasts the different illocutionary intents or  social dimensions in LLM-generated arguments with those written by humans in the context of social-media comments. The authors also investigate whether LLM-generated responses capture similar social dimensions to the original argument and how it contrasts with human-generated ones. Finally, they experiment whether such AI-generated persuasive arguments are more aligned with humans than their human-written counterparts. Overall, the work delves deep into the characteristics of LLM-generated persuasive texts, highlighting "how" AI systems interact with and adapt to human-discourse, thereby influencing persuasion, and that it extends beyond the surface form of the argument.
Summary of strengths: - This paper presents a faithful and detailed introduction to related works, clearly demonstrating the connections and innovations between this work and related ones.
- The experiments in this paper are comprehensive and the design is reasonable. A pre-experiment is conducted to select the most suitable models, and a large number of experiments are carried out to satisfactorily answer the proposed research questions. The analysis of the relationships among human non-Δ comments, human Δ comments, and LLM-generated comments is remarkable.
- The findings of this paper are enlightening, revealing the characteristics of LLMs persuasion in terms of social dimensions.
- Experiments demonstrate that the proposed method overall outperforms the state of the arts.

1.	The empirical results generated by the authors are interesting.
2.	The dataset used by the authors appears to be high quality and well suited for the author’s experiments.
3.	The theoretical grounding of this work offers an interesting lens to examine LLM generations.
4.	The statistical analyses the authors use are clearly documented and well explained.

1. Grounding the characteristics of AI generated persuasive text in established theoretical frameworks is important and timely. 
2. The paper is well written; the research questions, methodology, and experimental results are discussed succinctly and clearly. 
3. The experimental section is sound and comprehensive.
4. The results outline interesting findings which are relevant in the current discourse of the influence of AI-generated content.
Summary of weaknesses: - The main contributions of the paper lie in its comprehensive experiments and experimental findings, but it lacks innovation in the research perspective. Related works show that compared with Breum et al. (2024), this work innovates mainly in the research scenario and specific research questions, shifting from a synthetic dialogue scenario to a real-world social media scenario without clear prompts. The perspective of social dimensions is not first proposed by this work.
  
- The work uses a classifier to extract social dimensions. The performance of this classifier should be explained.

1.	The biggest weakness of this paper, in my view, is its limited novel contribution. To the best of my understanding, it primarily applies the dataset from Tan et al. and the experimental methodology from Monti et al. and Choi et al. without introducing new data or methodologies. While the empirical results are interesting, the authors may wish to expand the scope of their contribution, for instance, by exploring ways to enhance LLM persuasiveness or shape specific social dimensions through prompt engineering, instruction tuning, etc.
2.	The pilot study is fairly limited in scope, focusing on only two dimensions with a relatively small sample size. Additional details on the selection process for the 100 triples, as well as more comprehensive annotator demographics beyond the general statement of “diverse ethnic, educational, and employment backgrounds,” would be helpful.
3.	More details on the models used in the paper—such as the training hyperparameters for the classifier in Section 3.2—would improve reproducibility.
4.	The theoretical grounding of the paper could be more clearly articulated to strengthen its connection to the empirical findings.

1. Considering the stark contrast of social intent dynamics of the Mistral and LLama-2 model, it would have been interesting to see how well humans align or prefer the outputs generated by these models as opposed to humans, i.e. how does RQ4 change if the models or LLMs chosen was not GPT-3.5-turbo. 

2. Did the study control for the length of the conversation thread? Often-times on CMV, certain comments elicit more engagement from the OP (original poster), leading to a back-and-forth before the delta is awarded or a decision is reached. It seems that the experiments in this study all focus on a single comment in response to a post with respect to the LLM. For the human-generated responses, did you consider only the cases where the length of the conversation thread is only 1, i.e the delta was awarded in the very first post, or was the past previous conversation history discarded?

3. What was the performance of the model that extracted the social dimensions from the text/post? How reliable were the scores attributed by this classifier, i.e. did you do a sanity check to verify the validity of this classifier on the LLM-generated texts?
Comments, suggestions and typos: N/A

See weaknesses section

Considering how distinct the social dimensions are for human-generated and AI-generated text for the persuasive messages, it would be interesting to see if they could serve as a means to distinguish between the two kinds of messages?
Overall assessment scores: 3.5

2

3.5
Metareview score: 3.5
Metareview: This paper proposes to analyse the persuasive capabilities of Large Language Models through a theoretical, socially validated framework called Communicative Action Theory. After initial experiments based on the *ChangeMyView* subreddit, comparing 3 models along social dimensions, the authors experiment whether AI-generated persuasive arguments are more aligned with humans than their human-written counterparts. The study finds that crowdworkers preferred GPT-3.5-turbo responses, making them more persuasive than human-written arguments regardless of argument quality.
Summary of reasons to publish: - Reviewers all agree that the paper is well written, clear, offers a faithful review of related works, and describes experimental design in a comprehensive manner.
- Reviewers agree that the paper proposes well-designed experiments, which are properly grounded on a theoretical framework.  
- Reviewers all agree that the paper proposes very interesting findings.