LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback (2024)

Abstract: Ensuring that online discussions are civil and productive is a major challenge for social media platforms. Such platforms usually rely both on users and on automated detection tools to flag inappropriate arguments of other users, which moderators then review. However, this kind of post-hoc moderation is expensive and time-consuming, and moderators are often overwhelmed by the amount and severity of flagged content. Instead, a promising alternative is to prevent negative behavior during content creation. This paper studies how inappropriate language in arguments can be computationally mitigated. We propose a reinforcement learningbased rewriting approach that balances content preservation and appropriateness based on existing classifiers, prompting an instructionfinetuned large language model (LLM) as our initial policy. Unlike related style transfer tasks, rewriting inappropriate arguments allows deleting and adding content permanently. It is therefore tackled on document level rather than sentence level. We evaluate different weighting schemes for the reward function in both absolute and relative human assessment studies. Systematic experiments on non-parallel data provide evidence that our approach can mitigate the inappropriateness of arguments while largely preserving their content. It significantly outperforms competitive baselines, including few-shot learning, prompting, and humans.

heading: 1 Introduction
text: Creating trusted and safe online spaces where people with different backgrounds and opinions can discuss controversial issues is a major challenge for social media platforms (Salminen et al., 2018). The diversity in opinions, emotional attachments, and the anonymity of the web easily lead to heated discussions, which can quickly turn into toxic environments, even if only one participant behaves inappropriately (Habernal et al., 2018). Avoiding this is a challenging task, often supported by platform
Inappropriate “for everyone who is talking about RAPE in this subject let me ask you one thing!!!!
Improved (Ours) Issue: “Pro choice vs pro life”
“For those discussing rape, I would like to pose another scenario.
If you got into a fight and broke your hand or arm, would you amputate it to remove the physical reminder of the experience?
if you got in a huge fight with someone and ended up breaking your hand or arm... would you cut it off just because it would REMIND you of that expirience???
if your actualy SANE you would say no and if you say yes you need to see a Physiatrist!!!!”
Of course, the answer is no. Although it is understandable to have emotional scars, physical scars can be treated and lived with, even if they remain as a reminder of the experience.”
Figure 1: Example of an inappropriate argument from the corpus of Ziegenbein et al. (2023) and the same argument after applying our approach. The used colors indicate which parts of the original argument were removed (red strikethrough) and which parts were added by our approach in the rewriting process (green).
moderators that check content flagged by users or detection tools. However, the amount of moderation required on the web necessitates automation of the process, as the resources for manual moderation are usually insufficient, and the severity of inappropriate content can negatively affect the moderators’ psyche (Spence et al., 2023).
Multiple concepts and datasets have been proposed to model unwanted behavior in discussions, from simple offensiveness (Borkan et al., 2019) to sophisticated notions such as inappropriateness (Wachsmuth et al., 2017). The latter focuses on the exchange of arguments, their creation of credibility and emotions, and their adherence to the issue. Ziegenbein et al. (2023) argue that appropriateness displays the minimal quality of an argument necessary to be considered valuable in a debate.
This paper is the first to study how to rewrite inappropriate arguments automatically. Prior work studied the detection of unwanted behavior (Wulczyn et al., 2017; He et al., 2023; Ziegenbein et al.,
4455
2023), often using large language models (LLM). While a few methods improve content, they solely transfer the style of texts to be more formal (Rao and Tetreault, 2018; Lai et al., 2021), less subjective (Pryzant et al., 2020; Liu et al., 2021a), or less toxic (Laugier et al., 2021; Logacheva et al., 2022), or they target the quality of arguments in general (Skitalinskaya et al., 2023). This commonly comes with preserving the original content and operating on single sentences. However, if the inappropriate behavior is rooted in the content itself and not only in the style of the text, content modifications on the document level may be necessary. In addition, most existing approaches rely on parallel data, which is unavailable for rewriting inappropriate arguments.
Instead, we propose an LLM-based rewriting approach to inappropriateness mitigation inspired by reinforcement learning from human feedback, RLHF (Christiano et al., 2017). Compared to the typical use of RLHF in NLP (Ouyang et al., 2022), the core ideas of our approach are: (1) We obtain a ‘cheap’ initial policy (an LLM to align) from either few-shot learning or prompting, rather than using supervised learning. (2) We specifically consider the properties on which we want to align the LLM, rather than relying on generic preference information. (3) We evaluate multiple candidate alignments with different weightings of the desired properties, rather than relying on a single alignment obtained from preference information.
After experimentally determining the performance of multiple LLMs using few-shot learning and prompting on the corpus of Ziegenbein et al. (2023), we find prompting the instruction-finetuned LLaMA (Touvron et al., 2023) variant of Taori et al. (2023) (Alpaca) to be best and thus proceed to align it further using our approach. We deem the desired properties for rewriting inappropriate arguments to be semantic similarity to the original argument and appropriateness of the generated argument, and we make use of existing classifiers to learn how to generate texts that fulfill them (Zhang et al., 2020; Ziegenbein et al., 2023).
Exemplarily, Figure 1 shows an inappropriate argument from a “Pro choice vs pro life” debate and the same argument rewritten by our approach. Here, the original argument uses overly excessive emotions, making it hard to understand, and it displays little interest in the opinion of others. The rewritten argument reduces emotions and adds a more open ending, making it more appropriate while keeping the original argument’s gist intact.
For evaluation, we obtain human rewrites for a portion of the data and compare them automatically against our models and a competitive model from the literature. Moreover, we conduct relative and absolute human evaluations of the rewrites of our trained models and the human rewrites. We find that our approach successfully aligns LLMs according to the desired property weighting and produces the best rewrites. Intriguingly, our human annotators prefer appropriate rewrites, even if they are less semantically similar to the original arguments.
Altogether, this paper’s main contributions are:1
• An RLHF-inspired approach for non-parallel data, based on instruction-finetuned LLMs aligned to specific classified properties.
• The first computational approach for rewriting inappropriate arguments.
• Empirical insights into human preferences regarding semantic similarity and appropriateness when rewriting inappropriate arguments.

heading: 2 Related Work
text: The notion of appropriateness in speech and argumentation, tied to cultural norms, social politeness, and context, originates from Aristotle’s work on rhetoric (Aristotle, ca. 350 B.C.E./ translated 2007). It has been examined in various shades across linguistic studies (Hymes et al., 1972; Ranney, 1992; Schneider, 2012; Jdetawy and Hamzah, 2020). In debate, topic adherence and avoidance of offensive or biased language are considered aspects of appropriateness (Andrew, 1996; Blair, 1988; Walton, 1999; Burkett, 2011). Modeled by Wachsmuth et al. (2017) as a dimension of rhetorical argument quality, appropriateness has been partially explored in NLP, focusing on the simultaneous assessment of credibility, emotional engagement, and proportionality to the issue. Computationally, Wachsmuth and Werner (2020) initially attempted appropriateness prediction as a subtask of argument quality assessment. Later, Ziegenbein et al. (2023) refined the notion of appropriateness in argumentation, modeling it in a 14-dimensional taxonomy and predicting it together with its subdimensions in a multilabel setting. Wachsmuth et al. (2024) recently delineated how to instruct LLMs towards more reliable argument quality assessment, not targeting rewriting though.
1The corpus extension and experiment code can be found under: https://github.com/webis-de/ACL-24.
Related to argument rewriting, Skitalinskaya et al. (2023) studied how to improve the general quality of argumentative claims on data similar to what we use, but not focusing on inappropriateness specifically. Closest to our work in the context of style transfer tasks are Nogueira dos Santos et al. (2018), Laugier et al. (2021), and He et al. (2023), where toxic content is mitigated using supervised rewriting approaches. However, their methods are applied on the sentence level, strictly aim to preserve content, and prevent the addition of new content. Unlike them, we focus on document-level rewriting and explicitly consider adding or deleting content. Furthermore, most of these approaches require parallel data and rely on supervised learning. In contrast, our approach is meant for non-parallel data since no parallel dataset is available to learn to rewrite inappropriate arguments.
To this end, we use reinforcement learning as it allows us to train on non-differentiable metrics, such as the outputs of classifiers, which we use to test for the desired properties of our task. Reinforcement learning has been used for a variety of NLP tasks, including dialogue generation (Li et al., 2016), machine translation (Wu et al., 2018), and summarization (Ziegler et al., 2019; Böhm et al., 2019; Stiennon et al., 2020). In the context of style transfer, multiple approaches have aimed to flip the sentiment, stance, or polarity of texts in parallel (Sancheti et al., 2020; Liu et al., 2021b) and nonparallel settings (Xu et al., 2018; Gong et al., 2019; Wu et al., 2019; Luo et al., 2019).
Many works study the related task of formality transfer (Gong et al., 2019; Luo et al., 2019; Sancheti et al., 2020; Liu et al., 2021b; Lai et al., 2021) with diverse techniques on the parallel data of Rao and Tetreault (2018). Three properties are commonly controlled during transfer: fluency, content preservation, and transfer strength. Similar to our work, Madanagopal and Caverlee (2023) propose reinforcement learning to remove subjective bias in Wikipedia texts, modeling the reward as a weighted function of classifiers for style, fluency, and content preservation. However, unlike in formality transfer, parallel data for rewriting inappropriate arguments is neither available nor straightforward to acquire.
Consequently, our work focuses on non-parallel style transfer inspired by reinforcement learning from human feedback. We rely on the more stable proximal policy optimization (PPO) (Schulman et al., 2017) instead of the commonly used REIN-
FORCE Monte Carlo policy gradient (Williams, 1992) due to promising recent results in NLP and the advances in the capabilities of language models to follow instructions (Taori et al., 2023; Wang et al., 2023). To our knowledge, only the work of de Langis et al. (2024), which was published on arXiv after the submission of our work, uses an RLHF-inspired PPO approach for a style transferrelated task in NLP. Our work is the first to investigate the use of prompting as an initial policy, avoiding the need for parallel data through pseudoparallel data and applying RLHF-inspired PPO to non-parallel data.

heading: 2.1 Proximal Policy Optimization in NLP
text: As a basis for the presentation of our approach, we here shortly describe the intuition behind PPO in an NLP context. For a more formal description, we refer the reader to Zheng et al. (2023).
PPO learns a model critique (a value model), which estimates the expected cumulative future reward (a value) of a state (the generated text up to this point), together with a policy (e.g., an LLM). The “real” reward is based on the output of a reward model (e.g., a classifier) and can be any scalar value. The value model estimates the advantage (gain in reward) of performing a specific action in a state (generating a specific word given the text generated so far) over performing the current policy’s suggested action. Specific actions are chosen based on sampling from the current policy (e.g., top-p sampling). The current policy is updated based on the advantages of the specific actions and the KLdivergence between the token-level distributions of the current policy and its updated version. The KLdivergence here improves stability during training, limiting the size of update steps.
In other words, when predicting the next word given a sequence of previously generated words, multiple fitting potential words are considered by sampling from the LLM. The difference in the longterm expected reward between each of them and the word with the current highest probability is used to update the LLM. This way, the LLM can be steered (aligned) to generate text that fulfills the desired properties represented by the reward model.

heading: 3 Approach
text: This section presents the approach that we propose to rewrite inappropriate arguments inspired by reinforcement learning from human feedback. Figure 2
illustrates the main elements of our approach explained in the following.

heading: 3.1 Problem Formulation
text: Let x be an argument and ŷ be an improved version of x. We define the task of rewriting inappropriate arguments as learning a function f : x 7→ ŷ such that ŷ preserves the content of x as much as possible while being more appropriate.

heading: 3.2 Prompting as an Initial Policy
text: Usually, reinforcement learning from human feedback (RLHF) is used only to steer an LLM that has already learned to solve a task to a certain known extent instead of learning the task from scratch. However, we neither have access to such a model, nor data to train it. Instead, we thus propose to use prompting to obtain an initial policy f =̂ πPRT (an LLM to start from) that solves the task to a certain (not immediately quantifiable) extent. We experimentally compare multiple autoregressive pretrained LLMs using zero-shot, few-shot, and instruction learning-based prompting to find an effective initial policy (further details on the models and prompting methods follow in Section 5). Then, we proceed to use the same prompts while learning πRLϕ , where ϕ are the learnable parameters, to obtain a better version of the initial policy πPRT .2
2In the following, we use the terms policy and LLM interchangeably as they refer to the same concept (π) in the context of reinforcement learning (RL) for LLMs.

heading: 3.3 Reward Modeling and Policy Learning
text: We initialize πRLϕ with a pretrained LLM π PRT that is prompted in natural language to generate ŷ given x. Unlike Stiennon et al. (2020), we do not require learning a reward model r from human feedback on preference judgments. Instead, we model relevant properties that we desire to be present in the target output, ŷ. In particular, we assume that the semantic similarity (sim) of ŷ to x and the appropriateness (app) of ŷ are such relevant properties. We use pretrained classifiers c that estimate these properties as our reward model.
The property reward model r is thus defined as:
r(x, ŷ) := α · csim(x, ŷ) + (1− α) · capp(ŷ) (1) Here, α ∈ [0, 1] is a hyperparameter that controls the trade-off between semantic similarity and appropriateness. csim and capp are classifiers that estimate the semantic similarity and appropriateness of ŷ given x. Similar to Stiennon et al. (2020), to obtain the final reward model R, we penalize r with the KL-divergence between the initial policy πPRT and the learned policy πRLϕ to disincentivize moving away too far from πPRT :
R(x, ŷ) := r(x, ŷ)− β log [
πRLϕ (ŷ|x) πPRT (ŷ|x)
] (2)
Here, β ∈ R is a hyperparameter that controls the strength of the KL-divergence. The policy πRLϕ is optimized using the proximal policy optimization (PPO) algorithm (Schulman et al., 2017).
We argue our setup to be beneficial for two reasons: (1) Pretrained classifiers that learned to assess specific properties from human labels are available for a wide variety of tasks; and (2) no training and consequently no data is required to learn an initial policy that solves the task to a certain extent.

heading: 4 Data
text: For our mitigation experiments, we extended the appropriateness corpus of Ziegenbein et al. (2023).

heading: 4.1 Source Data
text: The original corpus contains 2191 arguments and the corresponding discussion titles from three genres (reviews, discussion forums, and Q&A forums). Each argument has been annotated three times using a hierarchical 14-dimensional taxonomy of appropriateness flaws, such as toxic emotions or missing intelligibility. Here, we consider only the parent dimension inappropriateness to develop and
evaluate our approaches. The corpus contains 1182 inappropriate and 1009 appropriate arguments.

heading: 4.2 Extension
text: We extend the given corpus by arguments from its original domains. In particular, we collected 73,703 arguments from the IACv2 Corpus (Walker et al., 2012; Abbott et al., 2016) and the GAQCorpus (Ng et al., 2020). We kept only those 55,290 arguments that have at least 10 and at most 220 words and that do not exceed 1100 characters. This way, we ensure that the arguments have approximately the same length as those in the original corpus. To avoid any topic leakage in the extended part of the corpus, we also remove arguments that belong to a topic already present in the original version of the corpus (49,417 arguments remaining). Finally, we soft-label all arguments in the extended part of the corpus with the five-fold ensemble classifier of Ziegenbein et al. (2023) to obtain appropriateness labels. These labels can then be used to train our approaches. 35,537 of the arguments were labeled as inappropriate and 13,880 as appropriate.

heading: 5 Experiments
text: This section describes the training procedure of our approach from Section 3 on the data fom Section 4 and the experiments we conducted to evaluate it.

heading: 5.1 Experimental Setup
text: As Ziegenbein et al. (2023), we split the data for evaluation into 70% training, 10% validation, and 20% test, ensuring an equal weighting of the 14 corpus dimensions. However, since we are interested in mitigating inappropriateness, we train and evaluate only on inappropriate arguments. During training, we exclusively use the inappropriate arguments from the corpus extension.
The advantage of this setup is two-fold: First, we can train on a large amount of data, which is often crucial for the success of reinforcement learning (RL). Second, it allows us to avoid propagating any selection bias that may arise from the intermediate step of finding the best initial policy to selecting the best checkpoint from our trained policy. We use the training set to select the best initial policy (details below), the validation set to select the bestperforming RL policy checkpoints, and the test set to evaluate the performance of all approaches.
To automatically evaluate the generated rewrites, we employ the five appropriateness classifiers of
Ziegenbein et al. (2023) trained on different folds of the data. As we use the entire appropriateness corpus for evaluation, we ensure the use of the classifier for each argument that has not seen the argument before as part of its training. We use a classifier to predict the appropriateness of the original argument and the generated rewrites. Then, we calculate the following performance values:
• App. Percentage of arguments for which an approach has flipped the prediction from inappropriate to appropriate;
• Sim. Semantic similarity of a rewrite to the input argument in terms of BERTScore (Zhang et al., 2020);
• NES. Normalized word-wise edit similarity (to quantify amounts of edits) (Lopresti, 1996);
• PPL. Fluency in terms of perplexity; • GM. Geometric mean of App., Sim. and 1/PPL
(to compare approaches using a single score).
We use the semantic similarity and normalized word-wise edit similarity to quantify if the generated arguments are indeed rewrites of the original argument and not any probably unrelated but appropriate text. Furthermore, we use perplexity as a measure of text coherence and fluency.

heading: 5.2 Finding an Initial Policy
text: Similar to Stiennon et al. (2020), we start with obtaining the initial policy, πPRT that should have a reasonable performance mitigating inappropriate language. πPRT is then aligned using our RL approach. Since no parallel data is available to train a supervised model, we prompt four LLMs of similar size (6-7 billion parameters) in a few-shot learning and instruction following-based setting:
• OPT (Zhang et al., 2022); • BLOOM (Scao et al., 2023); • GPT-J. (Wang and Komatsuzaki, 2021); • LLaMA. (Touvron et al., 2023).
For the few-shot setting, we use 1, 4, and 9 examples to see how the performance changes with the amount of reference data. This setup is inspired by the hierarchical setup of the taxonomy of inappropriateness (Ziegenbein et al., 2023) having 1, 4, and 9 dimensions on the first, second, and third level respectively. To obtain instruction-finetuned versions of the models, we train each of them following the procedure suggested by Taori et al. (2023), to ensure we select the best base model and do not select
a model because of a specific way it is prompted or the instruction data it was fine-tuned on. This way, we have a single fixed prompt for all models and can control the generation length and other parameters equally well.
Creating Few-Shot Examples Since no rewrites are available for our few-shot learning setup, we collect 14 rewrites (1+4+9) of inappropriate arguments from three NLP experts, none of whom are authors of this paper. To create rewrites that are highly representative of a dimension, we use the appropriateness corpus, sentence transformers (Reimers and Gurevych, 2019), and PageRank (Lawrence, 1998) (details in Appendix B).
Prompting Setup We employ the natural language prompts suggested by Reif et al. (2022) and Zhang et al. (2020) to generate ŷ given x. The full prompts can be found in Appendix A.
Automatic Evaluation Table 1 shows the results of finding an initial policy. We observe BLOOM + Instruct. to create the most appropriate rewrites (0.653). LLaMA + Instruct. achieves the best results in terms of semantic similarity (0.620), and zero-shot LLaMA in terms of needing the minimal amount of edits to create its rewrites (0.528). In
terms of fluency, GPT-J + Instruct. performs best (37.49), closely followed by LLaMA + Instruct. (38.08). Overall, LLaMA + Instruct. seems to be the most stable choice (GM 0.216), so we select it as the initial policy to train our final approaches. Regarding geometric mean, few-shot learning leads to a general increase in performance. However, no clear trend in the number of few-shot examples used is visible. Overall, rewriting based on instruction-finetuning mitigates inappropriateness best across all models.

heading: 5.3 Policy Learning
text: Starting from our initial policy, LLaMA + Instruct., we use PPO to learn a set of candidate policies. We use the rescaled version of BERTScore (Zhang et al., 2020) to estimate the semantic similarity of ŷ to x and the appropriateness classifier from the first fold of Ziegenbein et al. (2023) to estimate the appropriateness of ŷ. Both values are in [0, 1]. We learn four candidate policies, each with a different property weighting α ∈ {0.4, 0.5, 0.6, 1}. Indicating the property weighting used, we refer to the corresponding models as LLaMA + PPOapp<sim, LLaMA + PPOapp=sim, LLaMA + PPOapp>sim and LLaMA + PPOapp respectively. The exact setup of our PPO training and the hyperparameters used are detailed in Appendix C.
Baselines In addition to the learned policies, we collected human rewrites for each argument. We refer to these as Human Baseline. For this purpose, we hired five native English speakers on
Upwork.com, three male and two female. We instructed the annotators with background information about appropriateness and asked them to suggest rewritten versions of arguments flagged as inappropriate in a forum. Each annotator was asked to rewrite 45 of the 225 inappropriate arguments.3
In automatic evaluation, we also compare multiple settings of the CoEdIT model proposed by Raheja et al. (2023), including paraphrasing (CoEdIT + Paraphrase), formality (CoEdIT + Formality), neutrality (CoEdIT + Neutral), and politeness (CoEdIT + Polite) style transfer, to better understand the relationship to these related notions.
During the development of our approaches we also experimented with other common non-parallel style transfer models, such as TAG (Madaan et al., 2020) and LEWIS (Reid and Zhong, 2021), but found them to be unsuitable as they lack the linguistic quality of modern LLMs, putting them at a disadvantage from the get-go.
Automatic Evaluation We evaluate the candidate policies using the same automatic metrics as for the initial policy. Table 2 shows the results. We observe that our approach successfully manages to align LLaMA + Instruct. according to the desired property weighting with LLaMA + PPOapp being best in terms of appropriateness (0.960), and LLaMA + PPOapp<sim in terms of semantic similarity (0.808) among the LLaMA-based models. We find that none of the CoEdIT baselines can successfully mitigate inappropriateness, speaking for the distinctiveness of rewriting inappropriate arguments as a task. Overall, we find LLaMA + PPOapp=sim to perform best (GM 0.237), even outperforming the human baseline (GM 0.175).
Manual Evaluation To enable comparison of various-automatically generated rewrites and human-suggested alternatives, we perform two manual evaluation studies. We again hire native English speakers on Upwork.com (7 female and 8 male) to evaluate the rewrites in absolute and relative terms, such that each rewrite (pair) is evaluated by five annotators. We again instructed annotators with background information about appropriateness. In total, we collected 4050 absolute and 8775 relative judgments.
In the first study, the annotators scored each rewrite regarding three considered quality metrics:
3Annotators were paid 15$ per hour. Guidelines and screenshots of the user interface are provided in Appendix F.
• App. Appropriateneess of the topic discussion in terms of style and content;
• Sim. Meaning preservation of the original arguments;
• Flu. Fluency and adherence to grammar conventions.
We utilized a 5-point Likert scale to evaluate the level of success of a particular rewrite in meeting each quality metric requirement. Here, 5 indicated a strong agreement with the rewrite’s success, and 1 strong disagreement. We calculate the final score for each rewrite using MACE (Hovy et al., 2013).
Table 3(a) presents the results of the conducted annotation study. We note that the obtained human judgments of appropriateness (App.) and semantic similarity (Sim.) across different desired property configurations are consistent with those obtained in the automatic evaluation (Table 2) for the trained models. Specifically, the LLAMA + PPOapp achieves the highest appropriateness rating (3.77), while LLAMA + PPOapp<sim combination attains the highest similarity rating (4.75). This alignment of human and automatic evaluations underscores the effectiveness of the chosen classification models in capturing the desired argument properties when mitigating inappropriateness. Overall, we find LLaMA + Instruct. to be the most balanced model (GM 3.57) following closely behind the human baseline (GM 3.63).
The goal of the second study is to rank the five LLaMA-based models and the collected human rewrites by perceived overall quality and appropriateness. To make the task more manageable for the human annotators, instead of requiring them to rank all six rewrites at once, we transform the annotation task into a pairwise ranking task and ask them to compare only two items at a time. Studies have shown that pairwise ranking tasks can lead to more reliable and consistent annotations compared to direct ranking tasks (Brun et al., 2010; Narimanzadeh et al., 2023) making them an effective and commonly used approach for subjective annotation studies, such as argument quality assessment (Habernal and Gurevych, 2016; Toledo et al., 2019; Skitalinskaya et al., 2021).
While pairwise rankings significantly increase the number of judgments to be made, making the task more time-consuming, Gienapp et al. (2020, 2022) have shown that efficient sampling strategies, such as Skip-window, can notably reduce the number of required pairwise annotations without
compromising the quality of the final ranking. In our study, we employ the Skip-window with λ = 4, which denotes that each rewrite is compared to every fourth rewrite in the set.4 To aggregate the pairwise preferences into a final ranking we apply Bradley Terry Aggregation (Bradley and Terry, 1952), which has shown to be more effective than alternatives such as KwikSort, Additive Aggregation, and PageRank (Gienapp et al., 2022).
Table 3(b) presents the results of the manual evaluation. Overall, we find that the instructionfinetuned model solely focusing on appropriateness (LLaMA + PPOapp) performs best (mean rank of 1.89), even outperforming the more balanced LLaMA + Instruct. (mean rank of 4.32) and the human baseline (mean rank of 3.18). In general, models incorporating text similarity assessments, such as LLaMA + PPOapp=sim and LLaMA + PPOapp<sim were consistently ranked lower. The findings from both studies indicate that human annotators prioritize appropriateness assessments when identifying the best rewrite. Specifically, the annotators tend to favor rewrites generated by instruction-finetuned models, such as LLaMA + PPOapp, which only includes the appropriateness property. It should be noted that the annotations collected during the annotation studies are made from the reader’s perspective and may not always aling with the writer’s viewpoint. We further discuss this point in Section 8. In terms of inter-annotator agreement, we find Pearson’s r of 0.35 for the ranking pre-study used to determine λ and 0.31 for the complete ranking study, which is considered to be moderate agreement and close to other studies of subjective dimensions in the domain of computational argumentation.
4The optimal λ value was found through a prestudy of 45 rewrite sets, futher details found in Appendix D.

heading: 5.4 Qualitative Analysis
text: We conducted a qualitative analysis to better understand the strengths and weaknesses of the rewrites generated. To this end, we manually inspected the subset of the ~2000 comments, which we received voluntarily from the annotators in the manual evaluation studies for rewrites generated by our most preferred model, LLaMA + PPOapp.
Most of the comments are positive, with the annotators expressing satisfaction with the rewrites’ improvement in emotional intensity, clarity, openness, relevance, seriousness, and language. These aspects are indicators of appropriateness, which is the main focus of LLaMA + PPOapp. Appendix E contains a random sample of different appropriateness flaws and the rewrites created by our models ordered by their similarity to the original argument. A list of all comments is provided together with the code and data in the supplementary material.
However, we also find that some annotators express concern in rare cases where the rewrites flip or neutralize the stance of the original argument by either changing single words (e.g., “not” to “is”) or by adding counterarguments and concluding that different point of views on the controversial issues are relevant to be considered. We find this to be particularly relevant for rewrites of short arguments, where the model has less text to work with. This may be an indicator of the limitations of the task of rewriting inappropriate arguments, as it may not always be possible to rewrite an argument if it, for example, solely consists of a single offensive sentence that is irrelevant to a topic. For such cases, it may be more appropriate to remove the sentence entirely. Finally, we also find that the issue discussed by the original argument can be unclear or inappropriate, making it difficult for our model and the human annotators to create a good rewrite.

heading: 6 Conclusion
text: In this paper, we have studied how to mitigate inappropriate language in arguments through rewriting. To this end, we have proposed an approach based on reinforcement learning from human feedback (RLHF), which balances the semantic similarity of arguments with a target style (here, with appropriateness). Our approach resorts to machine feedback instead of human feedback, though, thus enabling full automation.
Our experiments have demonstrated that prompting an instruction-finetuned large language model, combined with a single style classifier and an unlabeled dataset, is sufficient to train a policy that outperforms competitive baselines in terms of appropriateness and semantic similarity. Through manual annotation studies, we have provided evidence that our approach can mitigate the inappropriateness of arguments while preserving their content to a wide extent. Intriguingly, our human annotators prefer approaches that prioritize appropriateness over semantic similarity. Our results suggest that a careful design of the reward function is crucial for the success of RLHF-like approaches, if trained solely in an offline fashion.
We conclude that rewriting inappropriate language in arguments is a challenging problem that, from a reader’s perspective, often requires heavy editing and careful consideration of context. Our approach is a substantial first step to this tack. We hope it will inspire future work in this direction.

heading: 7 Acknowledgments
text: This project has been partially funded by the German Research Foundation (DFG) within the project OASiS, project number 455913891, as part of the Priority Program “Robust Argumentation Machines (RATIO)” (SPP-1999). We would like to thank the participants of our study and the anonymous reviewers for the feedback and their time.

heading: 8 Limitations
text: On the one hand, we inherit the limitations of the corpus used for modeling appropriateness, which includes being limited to the English language and a Western view of sociocultural factors. On the other hand, we find our work restricted in two aspects: (1) The dependence on the performance of the classifiers and initial policy and (2) The readers’ specific view chosen in our evaluation setup:
First, the performance of our RLHF-inspired approach relies on finding an initial policy that can then be improved to better align with the desired properties of semantic similarity and appropriateness. While using zero-shot learning, few-shot learning, or instruction-based learning to obtain the initial policy frees us from the need for parallel data to train a supervised model, its performance on the task is unknown, such that we can rely on automatic metrics only. The effect of the initial policy’s performance on the final performance of our approach is hence more or less unknown. For other tasks, it may be necessary to use a supervised model to obtain the initial policy. The same also holds for the classifier performance. While we observed in the manual evaluation that the considered classifiers successfully aligned the policy with the desired properties, the effect of the classifier’s performance on the performance of our approach remains unclear.
Second, as indicated already in Section 5.3, our evaluation focuses only on the reader’s perspective and not the writer’s. However, especially when we want to prevent a writer from creating inappropriate content, the writer’s perspective is also important because, in practice, changing a writer’s text may be considered ethically doubtful, if not done in agreement with the writer. Thus, this aspect should be considered in future work. We decided to focus on the reader’s perspective since we believe that this perspective is to be prioritzed in content moderation, where the reader is exposed to inappropriate content and may be harmed by it.

heading: 9 Ethical Considerations
text: Since we are dealing with a sensitive issue (content moderation), we believe it is important to discuss the ethical considerations of our work.
As always, in content moderation, there is a trade-off between freedom of speech and protecting individuals from harm. In practice, this trade-off, which is often further affected by the need to protect the platform from legal liability, may speak for removing potentially harmful content in doubt, even if it is not clearly inappropriate. In this regard, our work is a step towards preventing content removal by rewriting it more appropriately.
However, we stress that our approach is not meant for real-life applications yet, as it may not always be able handle the complexity of real-world arguments. This is due to the readers’ specific view
(as discussed in Section 8) and other generationspecific issues, such as hallucinations, whose effects are not investigated in this work.
The addition and deletion of information without permission of the author also raises questions regarding the responsibility of the platform and the author. As in other style transfer-related tasks that target sensitive topics, the idea of our approach could be inverted to make appropriate text inappropriate. However, since both appropriate and inappropriate arguments are crucial to developing the rewriting approach, we see no way around this but to strongly emphasize not using the approach for this purpose. Ultimately, we think that it is better that research topics as the one of this paper are studied openly in an academic environment than somewhere else without transparency.
Finally, since our approach is based on a classifier that could detect inappropriate content on a platform, it could circumvent the content moderation carried out on some web platform by probing the classifier and adapting the content accordingly. Thus, the rewritten content is not guaranteed to be always appropriate, as shown by our evaluation.

heading: A Prompting Setup
text: • Zero-shot/Few-shot Here is some text: {x} Here is a rewrite of the text that is more appropriate and makes only minimal changes: {ŷ}
• Instruction-Tuning Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction: Rewrite the following argument to be more appropriate and make only minimal changes to the original argument.
### Input: x
### Response: ŷ

heading: B Creating Few-Shot Examples
text: For each of the 14 dimensions of inappropriateness, we compute the mean annotator score for each argument x in the Appropriateness Corpus X . If the dimension is a parent of other dimensions, we keep only arguments with a mean annotator score greater than zero for each corresponding dimension. We then filter the set of all arguments for which the mean annotator score is maximal for the corresponding dimension as X
′ dim.
After obtaining the set of the candidate arguments for each dimension (X
′ dim), we embed them
using the all-mpnet-base-v2 sentence transformer (Reimers and Gurevych, 2019) (S) and calculate the cosine similarity between all possible embedding pairs. To this, we apply a variant of the PageRank algorithm (Lawrence, 1998) to compute centrality scores for each argument. The PageRank score P (si) for the i-th argument in X ′ dim is
P (si) := ∑
sj ̸=si
cos(si, sj)∑ sk ̸=sj cos(sk, sj) P (sj),
where d is a damping factor and n is the number of arguments in X
′ dim. The argument with the highest
centrality score in X ′ dim is our few-shot example for the corresponding dimension.
To obtain the corresponding rewrite ŷ from the set of candidate rewrites created by our experts, we use the geometric mean of semantic similarity, normalized edit similarity, perplexity, and appropriateness, as detailed above.

heading: C Hyperparameter Settings
text: We follow the PPO hyperparameter settings of Stiennon et al. (2020) with a few exceptions. Starting with a search over the learning rate and the KLdivergence coefficient using α = 0.5, we use a decay of the learning rate with a cosine schedule starting from 5 · 10−6 and ending at 1.5 · 10−6, and a KL-divergence coefficient of 1.857 · 10−3. We employ a batch size of 4 and train for 25 000 steps, equaling 3.2 million episodes. For generation, we use top-p sampling with p = 0.95 and a temperature of 1.0. For efficiency, we use adapter-based low-rank adaptation (LoRA) (Hu et al., 2021) with r = 8, an amplification factor of 32, and a dropout value of 0.1. Training a single model took around two days on four A100 GPUs.

heading: D Pre-study Details
text: S-Window Sampling To reduce the number of pairwise comparisons that need to be collected, we employ S-window sampling, which can be formaly defined as follows. Given a full set of comparisions Afull, consisting of k2 − k comparisons (no selfcomparisons), we want to sample such a subset A ⊂ Afull. To do so, we introduce a skip-size λ ∈ N+, and each rewrite ri ∈ Rk, we compile comparisons (ri, rj) , such that j = 1+ = 1 + (bmodk) for b ∈ {i+λ−1, i+2λ−1, ..., i+mλ− 1}, where m ≤ k − 1. If j = i the comparison is not included in the sample.
Finding Optimal λ To determine the optimal λ parameter, we conducted a prestudy, where we asked human annotators annotate the full set of comparisons Afull for a subset of 45 arguments, each with 6 rewrites obtained from the approaches outlined in Sections 3 and 5 as well as human generated rewrites.
For each of the 45 arguments, we created three different subsets Aλ using S-window sampling with λ ∈ {2, 3, 4}. Figure 3 illustrates the applied sampling strategies when considering six rewrites by showcasing which pairwise comparisons have been considered in each strategy. To reconstruct the ranking order derived from the complete set of comparisons, Afull, we applied Bradley-Terry Aggregation (Bradley and Terry, 1952) to each sampled subset of data. The Bradley-Terry model employs maximum-likelihood estimation to infer a latent score si ∈ S for each rewrite Ri ∈ R based on the sampled pairwise comparisons.
Table 4 presents the results of the conducted annotation study. The proposed sampling and comparison strategies are able to produce high quality rankings (λ = 2, ρ = 0.93, NDCG@1 = 0.95) using only 12% of the full set of pariwise comparisions and one annotator. However, to ensure consitency with other manual evaluation tasks in our paper, for the full annotation study we settle on using 5 annotators at λ = 4, which allows us to reasonably reconstruct the original ranking (ρ = 0.91, NDCG@1 = 0.95) while significantly reducing the number of pairwise judgements that need to be collected to only 40%.

heading: 1 4 270 7.99 0.85 0.91

heading: 2 4 540 15.99 0.86 0.93

heading: 3 4 810 23.98 0.88 0.94

heading: 4 4 1080 31.97 0.90 0.95

heading: 5 4 1350 39.96 0.91 0.95

heading: 1 3 225 6.66 0.80 0.88

heading: 2 3 450 13.32 0.80 0.91

heading: 3 3 675 19.98 0.82 0.91

heading: 4 3 900 26.64 0.84 0.93

heading: 5 3 1125 33.30 0.84 0.93

heading: 1 2 405 11.99 0.93 0.95

heading: 2 2 810 23.98 0.94 0.97

heading: 3 2 1215 35.97 0.95 0.98

heading: 4 2 1620 47.96 0.96 0.98

heading: 5 2 2025 59.95 0.97 0.99

heading: E Examples
text: Issue: Pro choice vs pro life
F Annotation Interface

heading: Future Work
text: Thus, this aspect should be considered in future work. We decided to focus on the reader’s perspective since we believe that this perspective is to be prioritzed in content moderation, where the reader is exposed to inappropriate content and may be harmed by it