ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer (2024)

Abstract: While state-of-the-art large language models (LLMs) can excel at adapting text from one style to another, current work does not address the explainability of style transfer models. Recent work has explored generating textual explanations from larger teacher models and distilling them into smaller student models. One challenge with such approach is that LLM outputs may contain errors that require expertise to correct, but gathering and incorporating expert feedback is difficult due to cost and availability. To address this challenge, we propose ICLEF, a novel human-AI collaboration approach to model distillation that incorporates scarce expert human feedback by combining in-context learning and model self-critique. We show that our method leads to generation of high-quality synthetic explainable style transfer datasets for formality (E-GYAFC) and subjective bias (EWNC). Via automatic and human evaluation, we show that specialized student models finetuned on our datasets outperform generalist teacher models on the explainable style transfer task in one-shot settings, and perform competitively compared to few-shot teacher models, highlighting the quality of the data and the role of expert feedback. In an extrinsic task of authorship attribution, we show that explanations generated by smaller models fine-tuned on E-GYAFC are more predictive of authorship than explanations generated by few-shot teacher models.

heading: 1 Introduction
text: Attribute style transfer is the task of transforming a given text along a particular style dimension, such as changing its formality, bias, or level of offensiveness (Lample et al., 2019; Sudhakar et al., 2019; Jin et al., 2022). Formality style transfer (e.g., informal→formal) could be useful in any writing assistance system, while neutralizing text that contains subjective bias would be an important tool for
Wikipedia editors (Pryzant et al., 2020) or journalists (Rosenberg and Fischer, 2023).
Style transfer approaches have primarily focused on the text re-writing task (e.g., informal →formal, subjective bias → neutral) using various methods from supervised (Rao and Tetreault, 2018; Pryzant et al., 2020; Zhong et al., 2021) to unsupervised (Krishna et al., 2020) and zero-shot methods using LLMs (Reif et al., 2022) (see also Jin et al. (2022) for a survey on style transfer). However, to our knowledge, no effort has focused on providing textual explanations for the style transfer task. For example, when transforming an informal sentence “I would throw them out asap !” into a formal paraphrase “I would dispose of them promptly” it would be useful to provide an explanation of the informal attributes in the input sentence (e.g., textese (“asap"), colloquialism (“throw out")), and formal attributes for the paraphrase (e.g., lexical sophistication (“promptly" and “dispose"); lack of abbreviations (“I would")). Similarly, for neutralizing subjective bias in “Orbis latinus, integral site of romance language" → “Orbis latinus, comprehensive site of romance language", it would be useful to have an explanation about which word/phrase in the input is biased and why as well as the type of bias (e.g., Framing (“integral" implies a subjective evaluation on the site’s importance)). The model’s explanations could help the user better assess the correctness of the style transfer system, could be used as features in downstream tasks such as authorship attribution (Section 5), or could act as a defense against spurious correlations (Ludan et al., 2023; Camburu et al., 2018) and annotation artifacts (McCoy et al., 2019; Poliak et al., 2018). To enable explainability in style transfer models, we provide the following contributions:
• A new task of explainable style transfer for which, in addition to sentence rewriting, the model needs to generate textual explanations.
16141
• A novel human-AI collaboration framework, In Context-Learning with Expert Feedback (ICLEF) (see Figure 1, Figure 2, and §3.2). The approach combines model distillation for explanation generation (Ho et al., 2022; Magister et al., 2023) with self-critique ability of LLMs (Madaan et al., 2023; Bai et al., 2022b; Saunders et al., 2022; Scheurer et al., 2023, inter alia), where the critic, unlike in prior work, is instantiated with expert demonstrations.
• Using ICLEF, we create for the first time datasets for explainable style transfer by augmenting an existing formality style transfer dataset GYAFC (Rao and Tetreault, 2018) and the neutralizing subjective bias dataset WNC (Pryzant et al., 2020) with textual explanations (§3). We show that the datasets generated with the help of ICLEF, E-GYAFC and E-WNC, are of good quality via automatic and expert evaluation, and that ICLEF-fixed instances are preferred (§3.3).
• Experiments that show that student models outperform teacher models in one-shot setting and perform comparably even with few-shot teacher models in automatic and expert evaluation, confirming the utility and quality of the datasets (§4). Moreover, in an extrinsic evaluation, we show that explanations generated by student models fine-tuned on our data produce a better signal for the authorship attribution task than the explanations produced by few-shot teacher models (§5).
We release the data, models, and code to encourage further research on explainability, learning from
scarce human feedback, and style transfer. 1

heading: 2 Related Work
text: Knowledge distillation and human feedback Model or knowledge distillation is a process of finetuning a smaller student model to imitate the behaviour of a more competent teacher model (Beyer et al., 2022; Buciluundefined et al., 2006; Hinton et al., 2015). Knowledge distillation became a popular technique, allowing to generate datasets of similar quality to the crowd-sourced data (West et al., 2022), especially when combined with a modelin-the-loop approach (Wiegreffe et al., 2022; Bartolo et al., 2022; Chakrabarty et al., 2022). Recent work explores model distillation with natural language explanations (Wang et al., 2023a; Ho et al., 2022; Magister et al., 2023), showing that large language models are capable of generating acceptable enough reasoning steps for student models to learn. Approaches to incorporate human feedback such as RLHF (Stiennon et al., 2020) and DPO (Rafailov et al., 2023) require large amounts of crowdsourced data and have not been generally shown to be effective for expert preferences. Imitation learning from human feedback (ILF) (Scheurer et al., 2023) utilizes human feedback to improve model-generated instances, and then fine-tunes on that data. Unlike these works, we focus on incorporating expert feedback which is naturally scarce and expensive to collect. Unlike other self-critique approaches (Madaan et al., 2023; Bai et al., 2022b; Saunders et al., 2022), we condition the model on expert corrections to incorporate high-quality human feedback.
1github.com/asaakyan/explain-st
Textual Explanations Natural language explanations have been utilized for a variety of tasks Wiegreffe and Marasovic (2021), such as natural language inference (Camburu et al., 2018), commonsense (Rajani et al., 2019; Aggarwal et al., 2021), social norm entailment (CH-Wang et al., 2023). We focus on creating natural language explanations for the style transfer task, which has not been addressed before.
Style transfer Style transfer approaches range from instruction-based methods (Reif et al., 2022) and paraphrasing (Krishna et al., 2020), to approaches focused on learning in low-resource settings (Patel et al., 2022). Much of style transfer work focuses on style representations that decouple style and content (Wegmann et al., 2022; Wegmann and Nguyen, 2021), however most of these methods are not designed to be interpretable. Interpretable approaches rely on constructing interpretable embeddings, such as LIWC (Tausczik and Pennebaker, 2010) or LISA (Patel et al., 2023). Zhong et al. (2021) proposed identifying biased segments in conjunction with neutralizing biased text. Unlike these approaches, we propose to use natural language explanations to further enhance model interpretability.

heading: 3 Building Datasets for Explainable Style Transfer
text: We build two explainable style transfer datasets by first augmenting existing datasets with synthetic textual explanations generated by a teacher model (§3.1), and then improving the generated data using our In-Context Learning with Expert Feedback (ICLEF) framework (§3.2).

heading: 3.1 Augmenting Style Transfer Datasets with Synthetic Textual Explanations
text: Formality style transfer The GYAFC (Rao and Tetreault, 2018) formality style transfer dataset contains parallel formal and informal sentences. The informal sentences are collected from Yahoo Answers, and formal paraphrases were crowdsourced using Amazon Mechanical Turk (AMT). We use ChatGPT-3.5 to generate explanations and formulate the following multi-step generation task: given an informal sentence from GYAFC si, generate a structured explanation of its informal attributes ei, then generate a formal paraphrase sf based on these attributes, then the formal attributes of the resulting paraphrase ef . Generating both ei and
ef allows us to train models in both directions (formal → informal and informal → formal). We use a semi-structured format for the explanations. Specifically, we ask the model to generate a list of attributes followed by an excerpt from the sentence as the evidence: attribute (“evidence”), see examples in Figure 1. These explanations have a consistent format, making it easier to verify and automatically evaluate.
Subjective bias style transfer We focus on the task of neutralizing subjective biased language introduced by Pryzant et al. (2020) to make sentences follow the Wikipeida Neutral Point of View Policy.2 We start with the Wikipedia Neutrality Corpus (WNC) (Pryzant et al., 2020), a parallel corpus of 180,000 sentence pairs originating from Wikipedia edits of subjective biased language. The goal is to generate an explanation (eb) for the type of bias present in the biased sentence (sb), following the scheme proposed by Pryzant et al. (2020) and Recasens et al. (2013): Framing, Epistemological, and Demographic (see definitions in Appendix I). It is estimated by Pryzant et al. (2020) that a small percentage of cases, the instances in the WNC contain noise. We add an additional “No Bias” label for these cases to reduce hallucinated biases for neutral sentences. In this case, we ask the model to output “This sentence does not contain bias” as the explanation (see second WNC example in Table 1). The explanation is structured as Type of Bias ("evidence" reasoning). Then, the teacher model generates an unbiased paraphrase (sn). See Figure 2 for an overview. At the time we developed this dataset, ChatGPT-4 became available, so we use this more powerful model as a teacher, especially since generating explanations for this task might require more reasoning capabilities. We do not generate explanations for the neutrality of the paraphrase as we are not exploring the neutral to biased paraphrase direction due to ethical concerns.

heading: 3.2 In-Context Learning from Expert
text: Feedback (ICLEF)
ChatGPT generations might contain errors (e.g., the generated style attribute "abbreviated language" with the evidence “I would” in Figure 1). To improve the quality of the data, we turn to expert feedback, since previous work has identified that crowdworkers on platforms such as Amazon Mechanical Turk could be unreliable for open-ended generation
2Wikipedia.org
tasks (Karpinska et al., 2021), and might even rely on ChatGPT to provide their answers (Veselovsky et al., 2023). The crux of our approach is to combine in-context learning and self-critique abilities of LLMs by instantiating the LLM-critic model with few-shot expert feedback demonstrations.
E-GYAFC For the formality style transfer task, we hire an expert annotator with a Masters degree in linguistics on Upwork3. Our annotation protocol (see Appendix J) provides a non-exhaustive reference to formality and informality attributes and asks the annotator to provide feedback on which attributes in ei, ef are incorrect if any, among other information. We provide 50 random samples for annotation. The annotation process took each expert only 2-3 hours. We find that the rate of critical errors observed in formality explanations is significantly lower ( (≈ 8% for formality explanations vs ≈ 56% for informality explanations), so we only focus on applying the LLM-critic to incorrect informality attributes (ei). To do so, we instantiate an LLM-Critic model by prompting another LLM (ChatGPT-3.5) with 35 expert human feedback corrections in-context (we find that this number leads to satisfactory correctness of ≈ 87%, see Appendix C for how performance changes given the amount of feedback) and ask it to act as an annotator on the new instances to identify incorrect attributes in them (see prompt in Table 12 in Appendix E). To mitigate the risk of generating new incorrect attributes, we only query the model to identify and remove incorrect attributes in ei, and if there are any, provide a fixed iclef -ei where they are removed. We refer to the resulting model as LLM-Critic (see how abbreviated langauge is removed in Figure 1).
In this way, we fix ≈ 30% of the generated data (2853 instances) – all instances for which the critic
3Upwork.com
has predicted that an improvement is needed. The resulting data (which we refer to as E-GYAFC) contains 9,960 original si, synthetic sf instances with corresponding iclef -ei, synthetic ef attribute explanations. We randomly split the data into 8,000 training instances and 1,960 held-out test instances.
E-WNC For the neutralizing subjective bias task, we hire an expert annotator with a PhD in linguistics on Upwork. We provide 50 random samples ensuring equal representation for each type of bias. To counteract potential annotation biases, we decided to use two annotators, one of the authors acting as the additional annotator, and then randomly sample the instances in equal proportions. The annotation protocol asks to provide a corrected explanation instead of eb if the synthetic explanation contains an incorrect type of bias or wrong justification. We randomly sample 35 distinct instances of the two annotators’ feedback, finding that this number leads to satisfactory correctness of ≈ 93% (see Appendix C). We provide the expert critiques incontext in a similar way to E-GYAFC (see bottom prompt in Appendix E, Table 12). Since new bias attributes might have been introduced, we regenerate the paraphrase sn given the new explanation (see corrected type of bias and a new paraphrase in Figure 2). We fix 8% of synthetic explanations in this manner (the lower rate of errors is explained by the higher quality of initial generations due to the use of a more powerful ChatGPT-4 model). The resulting dataset (E-WNC) contains 3,000 original WNC sb biased sentences, iclef -eb bias explanations, as well as the corresponding iclef -sn neutralized sentences. We randomly split the data into 2,500 training instances and 500 held-out test instances.

heading: 3.3 Dataset quality
text: Automatic evaluation of paraphrase quality We estimate paraphrase quality automatically using Mutual Implication Score (MIS) (Babakov et al., 2022) and Formality Score (see §4.2 for metrics details) between our formal paraphrases and the ones in GYAFC. We find that our paraphrases are of comparable quality with an MIS of 81.30 vs. 83.08 for GYAFC, yet we achieve a higher formality score of 98.43 vs. 89.39 for GYAFC (see Table 2). For example, for the GYAFC example in Figure 1, the formal paraphrase contains kick them out).
Similarly, for the E-WNC dataset we report bias score from an off-the-shelf classifier (see §4.2) along with MIS. The MIS scores are 79.32 for original paraphrases vs. 85.58 for our paraphrases, indicating higher semantic similarity. The neutrality scores are 69.34 vs. 72.64, indicating higher neutrality of our paraphrases (see Table 2).
Human evaluation For E-GYAFC we hire 3 expert annotators, 2 of which performed the annotation, as well as an independent expert annotator with a masters degree in linguistics. We ask their preferences on 100 randomly sampled instances with respect to the explanations (synthetic ei vs. iclef -ei) and paraphrases (original sf in GYAFC
vs. synthetic sf in E-GYAFC). In addition, we ask for acceptability judgments (whether the paraphrase or explanation are correct and complete) for the preferred paraphrase and separately for ef . We report preference or equal rates and acceptability rates.4 Overall, we found that our dataset instances are considered acceptable, with the average acceptability rate for ei, sf , ef being 87%, 77%, 98% respectively (row 1 in Table 3). Synthetic paraphrases are generally preferred to the ones in the GYAFC corpus (average 77%), and iclef -explanations are preferred or are equal in quality with original generations on average in 90% of cases (row 2 in Table 3). We computed the pairwise accuracy between annotator responses for all categories of E-GYAFC evaluation, and found that it averages at 81% across all categories. We provide more details in Appendix A. Table 1 shows qualitative examples of successful edits with ICLEF. Figure 3 shows the top 10 most frequent informality attributes.
For E-WNC, we hire 2 annotators, one of whom performed the ICLEF annotation. To meaningfully evaluate the preference of iclef -explanations compared to the synthetic ones, we ask to provide feed-
4We compute preference or equal preference among acceptable instances. For acceptability, we compute dispreferred instances as unacceptable.
back on 50 instances where the explanation has been updated. The average preference for iclef - eb compared to synthetic eb is 78%. Average acceptability rate of iclef -eb is 73%. The average preference for synthetic sn compared to the neutral sentence in the E-WNC corpus is 76%. Average acceptability rate of synthetic sn is 74%. The pairwise accuracy between the annotators is 77%. More details can be found in Appendix B.

heading: 4 Evaluation of Student and Teacher Models on the Explainable Style Transfer Task
text: We focus on evaluating the performance of select large language models on the explainable style transfer task. We do not evaluate post-hoc rationale systems (generating attributes given the paraphrase pair), since such pipeline models are less likely to reflect the underlying reasons for the model prediction, while models that jointly predict and rationalize exhibit desirable properties for faithful explanations (Wiegreffe et al., 2021). For explainable formality style transfer, we test the generation of ef , si, ei given sf (Formal→Informal) and ei, sf , ef given si (Informal→Formal) on a held-out test set from E-GYAFC. We evaluate how closely the model generated ei, ef match EGYAFC explanations, and we evaluate the semantic closeness and paraphrase quality for si, sf with reference-free metrics. Similarly, we evaluate eb, sn for the neutralizing subjective bias task using a test set from E-WNC. We report F1 scores for bias classification in sn.

heading: 4.1 Models
text: We test two smaller student models fine-tuned on our datasets. We fine-tune LLaMA-7B (Touvron et al., 2023) and Alpaca-7B (Taori et al., 2023) models on our data converted to the Alpaca instruction format. For formality style transfer, we fine-tune in both Formal→Informal and Informal→Formal directions separately (→), as well as in both directions in a multi-task fashion (↔). For subjective bias transfer, we only fine-tune in one direction. In addition, we test the teacher models (ChatGPT-3.5=GPT-3.5 and ChatGPT-4=GPT-4) in few-show setting, which is an ambitious baseline: first, they were used for data generation which biases reference-based metrics, second, they are prompted with improved instances of the data. Since the teacher models are closed models, we also test a representative open instruction-tuned model larger than the student, Vicuna-13B (Chiang et al., 2023) (Vic in tables), in few-shot setting. See detailed description of the models, hyperparameters, prompts, and additional experiments in Appendix F.

heading: 4.2 Automatic Evaluation
text: We use the following metrics:
• BLEU (Papineni et al., 2002): We measure the amount of exactly matched formal and informal attributes and evidences between the generated structured explanation and reference explanation in E-GYAFC and E-WNC.
• Mutual Implication Score (MIS) (Babakov et al., 2022) is a symmetric measure of text semantic similarity based on a RoBERTa (Liu et al., 2019) model fine-tuned for natural language inference and paraphrase detection used in prior work (e.g., Patel et al., 2022).
• Style Accuracy: For E-GYAFC, we use Formality/Informality Score5: RoBERTa (Liu et al., 2019) fine-tuned to predict whether sentences are formal or informal using GYAFC and Online Formality Corpus (OFC) (Pavlick and Tetreault, 2016). It achieves up to 0.98 ROC AUC. For E-WNC, we use Bias Score6: DistilBERT model (Sanh et al., 2020) finetuned for bias classification on the BABE media bias dataset annotated by experts (Spinde
5huggingface.co/s-nlp/roberta-base-formality-ranker 6huggingface.co/social-media-fairness/classifier-bias-sg
et al., 2021), on which it obtaines F1 score of up to 79.
Given that the bias type detection task can be viewed as a classification task (only 3 labels are present), we report F1 scores for bias type classification in the explanation. We also report the average across all metrics.
Results Table 4 shows model performance on the explainable formality style transfer task. While the Vicuna model does well in terms of style transfer (as evidenced by high MIS and Formality scores), it lacks in explanation quality (overall low BLEU scores). Student models perform better than the one-shot teacher model and competitively in 10- shot scenario judging by the Average score. Table 5 shows model performance on the explainable subjective bias style transfer task. Similarly, student models outperform the teacher model in one-shot setting. They also outperform few-shot models weaker than the teacher model (Vicuna and GPT3.5).
As for style transfer performance without considering the explanations, we see a slight decrease in the Informal→Formal (-0.93% avg. MIS and Formality compared to one-shot teacher) and Bias→Unbias (-2.13% avg. MIS and Neutrality) tasks. This is expected as consistent with prior work such as e-SNLI (Camburu et al., 2018).7 We see a sizeable increase in performance for Formal→Informal (+12.60%) direction. The informality scores are much better for the student models, perhaps due to the tendency to generate more formal speech both by teacher models and the other instruct models.
7“...while sacrificing a bit of performance, we get a better trust that when EXPLAINTHENPREDICT predicts a correct label, it does so for the right reasons.”

heading: 4.3 Expert Human Evaluation: Preference Judgments
text: We evaluate how well models’ outputs align with expert preferences. For e-GYAFC, we sample 100 instances from the test set and the corresponding model outputs from Vic1, the teacher model GPT-3.5 in one-shot setting (GPT-3.51), and the best student fine-tuned model. We ask for their preferences in terms of correctness and completeness of explanations, as well as in terms of paraphrase semantic preservation. We find that student model (Alpaca↔) generations are preferred to teacher model GPT-3.51 and Vicuna1 53% of the time by the expert linguist (GPT-3.51 is preferred 42%), indicating the student model is more aligned with expert preferences. See a qualitative example in Table 6. We also evaluate how well do smaller fine-tuned models perform compared to the teacher model in a few-shot setting. We sample 40 outputs and ask for preferences between generations from Vicuna and GPT-3.5 in the 10-shot setting (GPT-
3.510) as well as the student fine-tuned model. The fine-tuned model performs competitively with 49% preference for GPT-3.510 and 43% preference for the small student fine-tuned model.
Similarly, we hire an expert annotator and provide 50 test set instances from E-WNC along with model outputs from GPT-3.510 (weaker than the teacher model), 10-shot Vicuna, and the best student fine-tuned model (LLaMA→). We find that LLaMA→ is preferred at a rate of 56%, followed by GPT-3.510 (24%). See qualitative examples in Table 7. We also measure preferences to one-shot teacher model (GPT-41), finding comparable preference of 42% for LLaMA→ and 40% for GPT-41.

heading: 5 Extrinsic Evaluation of Formality Style Transfer Explanations
text: We use Authorship Verification (Martindale and McKenzie, 1995; Coulthard, 2004; Neal et al., 2017) as an extrinsic task utilizing PAN 2022 (Bevendorff et al., 2022) data. This is a binary classification task of deciding if two texts belong to the same author or not. The two input texts are two raw documents (e.g., paragraphs from blog posts), one of which is written by author A, and another one which is either written by the same author A
or by a different author B. We then derive a representation (features to be used for classification) of these input texts using our explainable style transfer model. We run AlpacaIF→ F on each text (at most 15 sentences per author are considered) and extract explanations containing informality attributes and evidence (see Table 8). In a preliminary evaluation of the usefulness of these features, we compute the similarity between authors by measuring the percentage of overlapping attributes. Note that the evidence fragments corresponding to the attributes are not used in this preliminary experiment. We then use the percentage of overlapping attributes as a classification score, for example, if author A uses colloquialism and textese, and author B uses colloquialism and abbreviation, their similarity score is the number of common attributes divided by the number of unique attributes between the authors, or 13 (0.33). Here, only 1 attribute is common (colloquialism) whereas the total identified attributes is 3 (colloquialism, abbreviation, textese). Following the computation described above, we take the similarity score as a confidence for a binary prediction task. If the similarity is high, there is a high chance the authors are the same (prediction = 1), and if not they are likely not the same (prediction = 0). The
underlying assumption is that it is more likely that the same author would use some of the informality features they used previously (not every sentence in PAN is completely informal, but informality is a very broad category, so when authors do use some informal attributes they can provide a signal for authorship). We compute ROC AUC between the confidence scores (author similarity score) and ground truth predictions from the PAN dataset (1 if authors are the same and 0 if not). We compare explanations from Vicuna10, GPT-3.510 and AlpacaIF→ F by their predictive signal for this task. Explanations by AlpacaIF→ F achieve an AUC of 56.4, whereas explanations from the Vicuna and GPT-3.5 models achieve a score of 50.0 and 47.0 respectively. This indicates a potential application of the explanations generated by the student model fine-tuned on our dataset (E-GYAFC) to be used as interpretable authorship features that can be explored in future work.

heading: 6 Conclusion
text: We propose a framework to augment two style transfer datasets with semi-structured textual explanations. To improve quality of model distillation and incorporate expert feedback, we propose ICLEF (In-Context Learning from Expert Feedback), a novel human-AI collaboration framework leveraging both in-context learning and self-critique abilities of LLMs. We evaluate smaller student models fine-tuned on the resulting datasets compared to large teacher models and conduct expert human evaluation. We also extrinsically evaluate the explanations for the formality style transfer on the downstream task of authorship attribution.

heading: 7 Limitations
text: The GYAFC dataset does not contain all types of informal and formal language, namely, they mostly focus on interpersonal relationships (the subset
used for this paper) and entertainment. Future work could consider extending our approach to other style transfer datasets, including ones more encompassing of formality.
While our methods are intended to produce faithful explanations, there can still be instances when a model does not rely on the attributes in order to complete the paraphrase. We also observed that hallucinations can still be present in our fine-tuned models’ explanations and hope that future work will try to address these issues. We also note that our approach does not replace expert annotation as it heavily relies on LLMs that may still hallucinate. It is only meant to be applicable in scenarios where expert feedback is expensive and/or difficult to gather.
One limitation of our method is that we used a relatively small number of experts to conduct our study. However, we believe that this setting mirrors real-life conditions where experts are usually scarcely available. We hope our approach provides a more general framework for incorporating expert feedback that can be adjusted to experts’ needs (e.g., a forensic linguist may require a different style transfer explanation than a literary critic).
Fine-tuning and running inference on large models requires expensive computational resources. However, we hope that our study presents a convincing argument that fine-tuning a smaller model once may be more efficient and accurate than running a large general-purpose model with elaborate long-context prompts.

heading: 8 Ethics Statement
text: The GYAFC corpus was created using the Yahoo Answers corpus: L6 - Yahoo! Answers Comprehensive Questions and Answers version 1.0. This Yahoo Answers corpus can be requested free of charge for research purposes. Access to our GYAFC dataset will require users to first gain access to this Yahoo Answers corpus. Authors obtained permission to access the dataset.
Our datasets do not include any protected data to the best of our knowledge. All annotators are fairly compensated for their work in accordance with their asking rate (typically over 20 USD per hour).
Our bias style transfer model is only intended for use in a human-in-the-loop fashion and not by itself to adjudicate bias in text. We hope that the explanation generation capacity of our model will
improve upon existing bias classifiers that typically do not provide textual explanations. In Appendix G, we show how style transfer can be used to evade AI-text detectors. Similarly to Krishna et al. (2023), we reiterate that this is not to provide a way to attack such systems, but to bring awareness to the community that current detectors are easy to evade. Moreover, we bring to attention the issue of detecting text on which style transfer paraphrase has been applied. We hope that future work develops systems capable of defending against such attacks, perhaps utilizing explanations generated by our system.
GYAFC and WNC may potentially contain offensive data as they are crowdowdsourced, however, on samples that we saw we did not find alarming ethical issues.

heading: Acknowledgements
text: We thank the annotators for their work and providing detailed feedback. We also would like to thank the reviewers for productive and engaging discussions. This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Human evaluation details: E-GYAFC
text: We hire three annotators for preference evaluation: A1 (bachelors degree in linguistics), A2 (bachelors degree in linguistics and a masters degree in education), A3 (bachelor and master degrees in linguistics) on Upwork. Due to high prevalence of the positive class, there is a high chance of random agreement, hence we provide a more granular look into the expert annotations than the inter-rater agreement in Table 9. Pairwise accuracy between annotator responses for all categories of E-GYAFC evaluation, and found that it averages at 81% across all categories. Annotator A2 expressed concerns that the paraphrases may sound unnatural due to excessive formality (we believe it is due to the context in which the informal expression would be uttered) and that explanations sometimes miss punctuation errors (which, while important, is not critical for model-generated explanations).

heading: B Human evaluation details: E-WNC
text: For E-WNC, we hire two expert annotators A1 and A2. A1 is a professional with over ten years experience in translation and interpretation, data annotation, linguistics and publishing. A2 is a PhD in linguistics with background in psycholinguistics and neurolinguistics and experience in writing, proofreading and editing academic texts

heading: C How does ICLEF performance change depending on the amount of feedback provided?
text: We perform a small-scale study to explore how performance of the self-critique component of ICLEF changes depending on the number of in-context examples provided. We sample 15 instances of synthetic explanations (before applying ICLEF) and evaluate generated critiques for correctness when providing 1, 10, or 35 instances of expert feedback few-shot. We evaluate the critiques for correctness, i.e. they have to both identify the incorrect attribute and not introduce any incorrect attributes. Figure 4 shows how by increasing the amount of feedback to 35-shot, the correctness is raised to ≈ 87% for E-GYAFC and ≈93% for E-WNC, which can be deemed satisfactory.

heading: D Additional estimations for trustworthiness of explanations
text: We conduct expert evaluation of our data in Section 3.3. To further confirm trustworthiness of explanations, we check that every evidence fragment actually occurs in the text. For every data instance, we compute what percentage of evidences was present in the sentence. We report average across the respective datasets and explanations in Table 11 summarizes the results, where it is evident that the explanations have high trustworthiness. We note that some explanations discuss absence of certain words in which case they should not be present in the sentence in the first place. Moreover, our structured explanation format would make it very easy for the end-user to verify the correctness of the explanation and alert them of potentially incorrect paraphrase.

heading: E Prompts
text: ChatGPT Explanation Generation Prompt We provide an instruction as a system prompt (“You are an expert forensic linguist...”) and 6 examples of the task in the OpenAI ChatML format.8
Instruction: You are an expert forensic linguist. Your task is to identify infromal attributes in a setnece, modify them to create a formal sentence, and then output the attributes of the generated formal sentence. Use the following format: attribute (excertp from text in quotation marks). Make sure to provide a complete list of informal and formal atttributes. Focus on what has changed between formal and informal sentences. Informal writing tends to be more casual, personal, and conversational than formal writing. Here are some common features of informal writing: Contractions: Informal writing often uses contractions, such as "I’m," "can’t," "won’t," and "they’ve," which are generally avoided in formal writing. ...
8github.com/openai/openai-python/blob/main/chatml.md
Examples: Informal greetings and sign-offs: Informal writing often uses casual greetings, such as "Hi" or "Hey," and sign-offs like "Cheers" or "Take care." Informal: if ur under 18 u have a BIG PROBLEM. Informal Candidates: ’18’, ’BIG’, ’PROBLEM.’, ’if’, ’u’, ’ur Attributes of Informal Style: textese ("ur", "u"), capitalization ("BIG PROBLEM"), colloquialism ("BIG PROBLEM")...
Prompt for ChatGPT ICLEF generation You are an extremely attentive and critical annotator with background in stylometry and linguistics. You will be provided with an informal sentence. You will also be provided with an explanation of its informality attributes. Decide whether the explanation is incorrect, and if so, describe what attributes were listed incorrectly.
EXAMPLES: Informal Sentence: Look, If you really like this person, just tell her. Informal Attributes: colloquialism ("just tell her"), contraction ("If you"), simple sentence structure. Attributes Listed Incorrectly: contraction ("If you" is not a contraction)...

heading: F Model details, additional models and experiments
text: Below we provide some additional experiments and models used for them as well as experimentation details.
Instruction-tuned Models All instruction-tuned models are provided with the same one-shot prompt (modulo special token requirements) and generation parameters.
• MPT-7B-Instruct: built by finetuning MPT7B (Team, 2023) on a dataset derived from the Databricks Dolly-15k (Databricks, 2023) and the Anthropic Helpful and Harmless (Bai et al., 2022a) datasets.
• Alpaca-7B (Taori et al., 2023) a model finetuned from the LLaMA-7B model on 52K instruction-following demonstrations generated with the Self-Instruct framework (Wang et al., 2023c).
• Vicuna-13B (Chiang et al., 2023): an opensource chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT9. It places first in the Huggingface Open LLM Leaderboard (Beeching et al., 2023) based on human and GPT-4 evaluation as of writing this paper.
• Falcon-40B (Almazrouei et al., 2023) causal decoder-only model trained on 1,000B tokens of RefinedWeb (Penedo et al., 2023) enhanced with curated corpora.
• Tülu-65B (Wang et al., 2023b) a 65B LLaMa model finetuned on a mixture of instruction datasets (FLAN V2 (Longpre et al., 2023), CoT (Wei et al., 2022), Dolly (Databricks, 2023), Open Assistant 1 (Köpf et al., 2023), GPT4-Alpaca (Peng et al., 2023), CodeAlpaca (Chaudhary, 2023), and ShareGPT).
ChatGPT For ChatGPT-3.5 we use gpt-3.5-turbo-1106. For ChatGPT-4 we use gpt-4.
Fine-tuned models We fine-tune below models on E-GYAFC. → indicates fine-tuning two models in each direction, and ↔ indicates fine-tuning on combined data in both directions.
• FLAN-T5-XL↔ (Chung et al., 2022) approximately 3B parameter instruction-tuned model based on the T5 architecture (Raffel et al., 2020).
• LLaMA-7B→ (Touvron et al., 2023) model by Meta trained on 1 trillion tokens.
• Alpaca-7B→,↔ (Taori et al., 2023) a model fine-tuned from the LLaMA-7B model on 52K instruction-following demonstrations. In addition, we test Alpaca-7Bnoexpl as the model fine-tuned for Formal to Informal style transfer with no explanations provided in the finetuning data or in the output.
Fine-tuning hyperparameters We fine-tune all models using the script provided in the Stanford Alpaca repository.10 We use exact same hyperparameters, except for batch size which we adjust to 1 due to memory constraints. We fine-tune our models on 4 A100 NVIDIA 40GB GPUs. We train our models for 3 epochs with learning rate 2e-05 with
9sharegpt.com 10Alpaca GitHub
cosine rate scheduler and warmup ratio of 0.03. We did not perform hyperparameter search. We report results form single runs with random seeds preserved due to computational constraints.
Inference parameters We use the same hyperparameters for generation across all models, that is temperature=0.7, top p=0.9, max new tokens = 256. We use the huggingface library.
One-shot instruction is provided below: Identify informal attributes in a given sentence, modify them to create a formal sentence, and then output the attributes of the generated formal sentence. For example: Informal: how can you tell if a girl
likes you or not? Informal Attributes: direct question
form ("how can you tell"), informal language ("girl", "likes you") Formal: What are some indications that a woman may be interested in you? Formal Attributes: indirect question form ("what are some indications"), lexical sophistication ("woman", "interested in you")
For the following sentence, identify informal attributes in a given sentence, modify them to create a formal sentence, and then output the attributes of the generated formal sentence.
For Tulu, we add the <asistant> and <user> tokens as advised by model developers.
Packages We used transformers (Wolf et al., 2020) for language model inference and NLTK package (Bird and Loper, 2004) for sentence tokenization.
Additional experiments with instruct and finetuned models We additionally perform experiments with MPT-7B, Falcon-40B, Tulu, and nonfine-tuned Alpaca. We also fine-tune an Alpaca model with no explanations. See Table 13. The model fine-tuned without explanations (Alpacanoexpl) achieves comparable performance, indicating that generating explanations does not significantly hurt performance on the standard style transfer task.
Fluency of style transfer Fluency could be used as an additional metric to measure the quality of the style transfer. However, fluency measures negatively correlate with informality, even though the semantic content stays the same. Because of that, we
preferred to use semantic similarity of the output to the input (MIS) as the primary measure across all tasks. Below is the result of an experiment we conducted with perplexity (lower is better), following Suzgun et al. (2022) exactly to compute the fluency metric.
As can be seen from Table 14, for formal and unbiased paraphrase the perplexity is comparable with the teacher in 1-shot and few-shot settings as well as the gold reference, whereas for informality the gold data has the worst perplexity, making this metric unfit when we want to transfer from formal to informal style. We want to add that we performed a human evaluation in Section 4.3 that includes the evaluation of the transferred sentences.
Performance of PromptRerank Baseline We considered including a PromptRerank baseline (Suzgun et al., 2022) as it is one of the previous SOTA approaches. On our observation the models did not perform competitively possibly because the models described in the paper are smaller than the contemporary LLMs. Future work may explore adjustments on this baseline to adapt it to the new models. Performance is depicted in Table 15.
BLEU and BERTScore between transferred text and gold references We opted for reference-free metrics since they would be less biased toward
producing output that most closely matches the reference. However, we now ran the experiment with BLEU and BERTScore between transferred text (output) and gold reference from our e-GYAFC and e-WNC datasets. BERTScore is based on contextualized-embeddings and usually preferred to using BLEU-4 (which is based on 4-gram overlap) for evaluation of text generation and paraphrases. Results are shown below in Table 16.
We see that using BERTScore, the student model is comparable to the teacher model and both are semantically close to the reference. BLUE scores are lower due to more stricter n-gram overlap requirements but that is across all models and scores are again comparable between smaller student models and the larger teacher models.

heading: G Discussion on applications of models fine-tuned for explainable style transfer
text: Explainable formal → informal style transfer is an interpretable adversarial attack on AIgenerated text detection methods, including retrieval Krishna et al. (2023) established that paraphrasing easily evades the detection of AIgenerated text, and proposed a retrieval-based defense. However, we hypothesize that retrievalbased metrics will degrade as similarity between generations becomes more ambiguous, as is the case for formality style transfer. For example, an adversarial agent might generate a post containing misinformation in typical “formal" language generated by a language model like ChatGPT. This text is relatively detectable by current classifiers and 100% detectable by retrieval-based methods. However, the agent might apply a style transfer model to lower the formality of the message. Alarmingly, not only this accomplishes the goal of spreading the AI-generated message more effectively as the result looks more like user-generated text, but, as
we show, it also decreases the chances of being detected as AI-generated by current methods.
We test this in the following setting: we use an online dataset of political tweets 11, and sample 30 of them. We ask ChatGPT to generate a political commentary post on the topic of the tweet (GPTF), as well as an informal paraphrase of the said post (GPT-Inf). We manually annotate the resulting summaries and select those that look like they could be legitimate political messages posted on social media and have valid paraphrases. We then use our AlpacaF→IF model to generate an informal paraphrase of the GPT-F posts sentence-by-sentence. We also verify that these paraphrases are semantically valid and close to the original GPT-Formal post and select 24 high-quality generations. We choose a relatively small sample since we want to verify the paraphrase was still close to the original sentence manually to ensure semantic control for the experiment.
We report detection scores12 from 4 methods surveyed by Krishna et al. (2023): GPTZero (Tian, 2023), OpenAI classifier (OpenAI, 2023), DetectGPT (Mitchell et al., 2023), and their proposed retrieval methods based on BM25 (Robertson and Zaragoza, 2009) or P-SP (Wieting et al., 2022) retrievers. As can be seen in Table 17, the formalto-informal transfer model significantly decreases detection scores of all AI-generated text detection methods, including the retrieval-based one (despite the fact that the retrieval corpus is significantly
11kaggle.com 12Since we do not have the human baseline text, we do not report the performance at 1% FPR, but for our study it is sufficient to show the scores decrease in absolute terms.
smaller than it would be in real-world). Interestingly, for the BM25 retrieval method, the ChatGPT paraphrases are slightly harder to detect than AlpacaF→ IF , whereas it is easier for all other methods. Since we used ChatGPT to generate the original posts, we could not use the watermarking methods (Kirchenbauer et al., 2023), but this can be explored in future work.
This result highlights the need to investigate new methods of detecting style transferred AI-generated text. As formality style transfer remains an effective attack, informality features produced by our model could help improve such classifiers. We leave this investigation for future work.

heading: H E-EGYAFC Statistics
text: We provide the distribution of 50 most frequent informal and formal attributes in E-GYAFC in Figures 5, 6.

heading: I E-WNC Statistics
text: We provide a proportion of classes in E-WNC in Table 18.

heading: J Annotation protocols
text: The screenshots for explanation interfaces are provided below in Figures 7, 8, 9. Similar annotation interfaces were used for the bias task.

heading: K Annotator demographics
text: Annotators are part of a diverse demographic, geographically present in North America, Europe, and Southeast Asia (as reported by Upwork). All annotators indicated at least fluent to native English skill.

heading: L AI assistants
text: The authors used AI assistants such as Co-Pilot and ChatGPT for writing code.

heading: Future Work
text: , 2023), but this can be explored in future work. This result highlights the need to investigate new methods of detecting style transferred AI-generated text. As formality style transfer remains an effective attack, informality features produced by our model could help improve such classifiers. We leave this investigation for future work