Venue: NAACL 2025

# Style Transfer with Multi-iteration Preference Optimization

Shuai Liu &Jonathan May

Information Sciences Institute

University of Southern California

{liushuai, jonmay}@isi.edu

###### Abstract

Numerous recent techniques for text style transfer characterize their approaches as variants of reinforcement learning and preference optimization. In this work, we consider the relationship between these approaches and a class of optimization approaches developed primarily for (non-neural) statistical machine translation, formerly known as 'tuning'. Inspired by these techniques from the past, we improve upon established preference optimization approaches, incorporating multiple iterations of exploration and optimization, and choosing contrastive examples by following a 'hope' vs 'fear' sampling strategy. Cognizant of the difference between machine translation and style transfer, however, we further tailor our framework with a new pseudo-parallel data generation method and a dynamic weighted reward aggregation method to tackle the lack of parallel data and the need for a multi-objective reward. We evaluate our model on two commonly used text style transfer datasets. Through automatic and human evaluation results we show the effectiveness and the superiority of our model compared to state-of-the-art baselines.

## 1 Introduction

Text style transfer aims to rewrite a given text to match a specific target style while preserving the original meaning. This task has drawn significant attention recently due to its broad range of applications, such as text simplification Laban et al. (2021), formality transfer Rao and Tetreault (2018); Liu et al. (2022), text detoxification Dale et al. (2021); Hallinan et al. (2023b), authorship transfer Patel et al. (2023); Liu et al. (2024), and authorship anonymization Shetty et al. (2018); Bo et al. (2021). Recent approaches have focused on pseudo-parallel data generation Krishna et al. (2020); Riley et al. (2021) and policy optimization Gong et al. (2019); Liu et al. (2021). STEERHallinan et al. (2023a) and ASTRAPOP Liu et al. (2024) combine the two and achieve state-of-the-art performance on text style transfer and authorship style transfer, respectively.

In this work, we seek to advance the frontier of text style transfer, drawing inspiration from the optimization techniques developed in the era of statistical phrasal machine translation, in which the lack of correlation between the log-linear model objective and the desired evaluation metric, typically BLEU Papineni et al. (2002), was observed Och (2003). Approaches to align1 the two objectives came to be known as _tuning_,2 beginning with Och (2003), and evolving into online variants Chiang et al. (2008), rank-based approaches Hopkins and May (2011), batch-based approachesCherry and Foster (2012), and several others. Tuning methods follow a generate-and-optimize pattern: a model is used to generate multiple candidate hypotheses per input, and then parameters are adjusted such that the argmax according to the model score also maximizes the evaluation metric. In this regard, tuning methods resemble approaches taken in the application of policy optimization algorithms, such as PPO Schulman et al. (2017), to generative language modeling Ouyang et al. (2022). More recent algorithms, such as DPO Rafailov et al. (2023) and CPO Xu et al. (2024a), which replace reinforcement learning (RL) in PPO with _preference_ optimization (PO), are reminiscent of the pairwise ranking optimization approach to tuning Hopkins and May (2011). Given this close relationship between these approaches, we can consider whether other techniques developed to improve MT tuning could be applied to optimization for style transfer.

Footnote 1: not to be confused with word alignment.

Footnote 2: not to be confused with parameter fine-tuning.

In this work, we propose Style TrAnsfer with Multi-iteration Preference optimization (STAMP), a two-phase PO training framewor

first use supervised fine-tuning to build a reference model from pseudo-parallel data and then train the reference model using PO. STAMP is similar to STEER and ASTRAPOP at a high level but is enhanced with two techniques borrowed from MT tuning and two modifications that further adapt it for text style transfer. First, we include _multiple iterations_ of preference pair generation followed by model optimization Och (2003), which has already been shown to be effective on other Seq2Seq tasks such as mathematical and scientific reasoning Chen et al. (2024); Pang et al. (2024); Song et al. (2024); Yuan et al. (2024). Second, following the hope-and-fear sampling in Chiang (2012), for PO, we over-generate outputs using the reference model and construct preference pairs using samples with high model scores and extreme (high or low) task objective scores, in order to avoid dangerous generation and encourage reachable good generation. To improve the quality of the reference model and the balance across the multiple training objectives, we additionally design a new two-step end-to-end pseudo-parallel data generation method and a dynamic reward aggregation method.

We evaluate our model on two popular text style transfer datasets, Grammarly's Yahoo Answers Formality Corpus (GYAFC) Rao and Tetreault (2018) and the Corpus of Diverse Styles (CDS) Krishna et al. (2020). Extensive experiments show that our model outperforms all state-of-the-art baselines on both datasets in both in-domain and out-of-domain evaluation, and demonstrates a higher training efficiency than the strongest baseline.

Our main contributions are:

* We propose a multi-iteration contrastive preference optimization training framework with hope-and-fear preference pair construction for text style transfer.
* We design a new pseudo-parallel data generation strategy and a dynamic weighted rewarded aggregation method to enhance the training framework for text style transfer.
* With the enhancements, our training framework produces style transfer models that achieve state-of-the-art performance on two popular text style transfer datasets.3 Footnote 3: Code and models sufficient for a reproducibility study are available at [https://github.com/isi-nlp/STAMP](https://github.com/isi-nlp/STAMP).

## 2 Methodology

In this section, we formalize the text style transfer task and introduce our training framework, STAMP.

### Task Definition

Given a source text **x** and a desired target style \(s\), the goal of text style transfer is to generate a fluent rewrite of **x**, denoted as \({\textbf{x}^{\rightarrow s}}\), that has the same meaning as **x** but is in style \(s\). In this work, we focus on high-resource text style transfer in which we have access to a reasonable number of texts4 for each target style. Specifically, we have a set of texts with style labels, denoted as \(\mathcal{D}=\{({\textbf{x}_{1}},s_{1}),\cdots,({\textbf{x}_{n}},s_{n})\}\), where \({\textbf{x}_{i}}\) and \(s_{i}\) re

Figure 1: An overview of STAMP, in which we first train a unified style transfer model using supervised fine-tuning on pseudo-parallel data generated from non-parallel data, and then further train the model using multi-iteration preference optimization on preference pairs constructed with hope-and-fear sampling.

 

fer to the \(i^{\text{th}}\) text and its style, respectively. For convenience, we adopt notations from Hallinan et al. (2023a) and denote the **fluency** of a text \(\textbf{x}_{i}\) as F(\(\textbf{x}_{i}\)), the **meaning similarity** between two texts \(\textbf{x}_{i}\) and \(\textbf{x}_{j}\) as MS(\(\textbf{x}_{i},\textbf{x}_{j}\)), and the **target style strength** of a text \(\textbf{x}_{i}\) w.r.t. a target style \(s\) as TSS(\(\textbf{x}_{i},s\)). Thus, given \(\mathcal{D}\), we aim to build a text style transfer system that maximizes three independent objectives: F(\(\textbf{x}^{-s}\)), MS(\(\textbf{x},\textbf{x}^{-s}\)), and TSS(\(\textbf{x}^{-s},s\)).5

Footnote 5: For brevity, we omit the arguments where unambiguous.

### Framework Overview

STAMP is a preference optimization-based training framework that contains two main stages, a supervised fine-tuning (SFT) stage and a multi-iteration preference optimization (PO) stage. In the SFT stage, we first generate a dataset \(\mathcal{D}_{\text{trf}}\) of end-to-end pseudo-parallel style transfer pairs from the (non-parallel) dataset \(\mathcal{D}\) and then train a style transfer model \(f_{\text{SFT}}\) on \(\mathcal{D}_{\text{trf}}\) using supervised fine-tuning. In the PO stage, we train a model initialized to \(f_{\text{SFT}}\) using multi-iteration PO6 to directly maximize the three objectives, TSS, MS, and F, and obtain our final transfer model \(f_{\text{PO}}\).

Footnote 6: See § 3.4 for details on the choice of PO used here.

### Supervised Fine-tuning

Due to a lack of parallel data, we adopt the technique described by Krishna et al. (2020), in which style-oriented paraphrasing is used to generate pseudo-parallel transfer data for each target style. Specifically, we paraphrase the texts in \(\mathcal{D}\) using a general paraphraser \(f_{\text{para}}\) similar to Krishna et al. (2020) and Hallinan et al. (2023a). To ensure meaning similarity preservation of the paraphrases, we generate \(k_{\text{para}}\) paraphrases for each text \(\textbf{x}_{i}\in\mathcal{D}\) and select the one with the highest meaning similarity to the original text, denoting it \(\textbf{p}_{i}\). We then obtain a dataset of paraphrases \(\mathcal{D}_{\text{para}}=\left\{\textbf{p}_{1},\cdots,\textbf{p}_{n}\right\}\). For each target style \(s\), we train a Seq2Seq model \(f_{\text{inv}}^{\rightarrow s7}\) on \(\left\{(\textbf{p}_{i}\rightarrow\textbf{x}_{i})~{}|~{}0\leq i\leq n\text{ and }s_{i}=s\right\}\) to maximize

\[p(\textbf{x}~{}|~{}\textbf{p})=\prod_{i=1}^{|\textbf{x}|}p(\textbf{x}[i]~{}| ~{}\textbf{p},\textbf{x}[<i]) \tag{1}\]

where \(\textbf{x}[i]\) and \(\textbf{x}[<i]\) represent the \(i^{\text{th}}\) token in **x** and tokens preceding the \(i^{\text{th}}\) token in **x**, respectively.

Following Krishna et al. (2020), we can transfer the style of a text **x** to a style \(s\) through

\[\textbf{x}^{-s}=f_{\text{inv}}^{\rightarrow s}(f_{\text{para}}(\textbf{x})) \tag{2}\]

where \(\textbf{x}^{-s}\) is the transferred text. However, the two-step generation breaks the gradient connection between **x** and \(\textbf{x}^{-s}\) which is needed in the PO stage to maximize the meaning similarity between **x** and \(\textbf{x}^{*\approx s}\). Therefore, we need an end-to-end pseudo-parallel dataset \(\mathcal{D}_{\text{trf}}\) to train a model that directly transfers a source text to each target style with no intermediate step.

To obtain \(\mathcal{D}_{\text{trf}}\), we transfer the texts in \(\mathcal{D}\) using \(f_{\text{para}}\) and \(f_{\text{inv}}^{\rightarrow s}\) for each target style \(s\). Specifically, for each target style \(s\), we transfer the texts in other styles in \(\mathcal{D}\) using Eq. 2 and obtain a dataset of style transfer pairs \(\mathcal{D}_{\text{tf}}^{\rightarrow s}=\left\{(\textbf{x}_{i}\rightarrow\textbf {t}_{i},s)~{}|~{}(\textbf{x}_{i},s_{i})\in\mathcal{D}\text{ and }s_{i}\neq s\right\}\), where \(\textbf{t}_{i}=f_{\text{inv}}^{\rightarrow s}(f_{\text{para}}(\textbf{x}_{i}))\) is a transfer of \(x_{i}\) in style \(s\). To obtain high-quality transferred texts, we generate \(k_{\text{sf}}\) transfers for each source text and select the one with the highest F \(\cdot\) MS\({}^{\tau_{\text{max}}}\cdot\) TSS, where \(\tau_{\text{ms}}>1\) is a temperature hyperparameter incorporated into the MS term to emphasize meaning similarity. We then construct \(\mathcal{D}_{\text{uf}}\) by combining \(\mathcal{D}_{\text{trf}}^{\rightarrow s}\) for all target styles and train an end-to-end style transfer model \(f_{\text{SFT}}\) on the combined data \(\mathcal{D}_{\text{trf}}\) to maximize

\[p(\textbf{t}~{}|~{}\textbf{x})=\prod_{i=1}^{|\textbf{t}|}p(\textbf{t}[i]~{}|~{} \textbf{x},\textbf{t}[<i],s) \tag{3}\]

Note that unlike Eq. 2, the probability in Eq. 3 is also conditioned on \(s\) because we adopt the unified model setting in Hallinan et al. (2023a). That is, we have a single transfer model for all target styles and control the target style with control codes.

### Multi-iteration Preference Optimization

We further train the SFT model \(f_{\text{SFT}}\) from the previous stage with multi-iteration PO to directly optimize the model on the style transfer objectives: F, MS, and TSS. To apply PO Rafailov et al. (2023); Xu et al. (2024) we first generate paired preference data from a _reference model_\(f_{\text{ref}}\) and then train a model on this offline preference data in a contrastive manner starting from the reference model. Inspired by Och (2003) and recent studies in iterative PO, such as Yuan et al. (2024) and Chen et al. (2024), we perform PO for multiple iterations to improve over the offline-only training, updating the reference model between iterations. Specifically, in iteration \(i\), we construct preference dataset \(\mathcal{D}_{\text{PO}}^{i}\) by transferring texts drawn from \(\mathcal{D}\), using reference model \(f_{\text{ref}}^{i}\). We use PO Rafailov et al. (2023); Xu et al. (2024) to train a model initialized to \(f_{\text{ref}}^{i}\) to match the preferences in \(\mathcal{D}_{\text{PO}}^{i}\); we 

call the resulting model \(f^{i}_{\text{PO}}\). We define \(f^{1}_{\text{ref}}\) to be \(f_{\text{SFT}}\) and in all other cases we define \(f^{i}_{\text{ref}}\) to be \(f^{i-1}_{\text{PO}}\). We next detail how the preference pairs in \(\mathcal{D}^{i}_{\text{PO}}\) are constructed and the reward function used in this process.

#### 2.4.1 PO Data Generation

We construct the preference dataset from \(\mathcal{D}\) using the hope-and-fer sampling strategy in Chiang (2012), which can encourage the model to generate "reachable" outputs with high reward scores and prevent the model from generating "reachable" outputs with low reward scores. While that work used BLEU Papineni et al. (2002) as a preference metric, we instead use our style transfer reward \(\mathcal{R}\) which is detailed in SS 2.4.2. Specifically, for each style \(s\), we generate \(k_{\text{PO}}\) rewrites of each text \(\mathbf{x}_{i}\) in \(\mathcal{D}\), whose initial style \(s_{i}\neq s\), into style \(s\) and select the preference pair from the rewrites based on both the reward scores \(\mathcal{R}\) and the model scores \(\mathcal{M}\) of the rewrites, where \(\mathcal{M}\) is the average token-level probability w.r.t. \(f_{\text{ref}}\). We select the rewrite with the highest \(\mathcal{M}^{\tau_{\mathcal{M}}}+\mathcal{R}\) as the "winning" rewrite \(\mathbf{t}^{w}_{i}\) and the rewrite with the highest \(\mathcal{M}^{\tau_{\mathcal{M}}}-\mathcal{R}\) as the "losing" rewrite8\(\mathbf{t}^{l}_{i}\), where \(\tau_{\mathcal{M}}\) is the temperature controlling the weight of model score.9 We then obtain a new dataset \(\mathcal{D}^{\to s}_{\text{PO}}=\left\{(\mathbf{x}_{i}\rightarrow(\mathbf{t}^{ w}_{i},\mathbf{t}^{l}_{i}),s)~{}|~{}(\mathbf{x}_{i},s_{i})\in\mathcal{D}\right\}\) for each style \(s\). Combining \(\mathcal{D}^{\to s}_{\text{PO}}\) for all styles, we finally obtain the PO dataset \(\mathcal{D}_{\text{PO}}\).

Footnote 8: also called “chosen” and “rejected” rewrites in PO literature (e.g., Rafailov et al., 2023).

Footnote 9: In practice, we find using model score does not benefit performance, so we drop this term for STAMP, which reduces the preference pair selection criteria to the sample with the highest \(\mathcal{R}\) and \(-\mathcal{R}\); a detailed comparison is shown in § 4.3.

#### 2.4.2 Reward Function

To directly maximize the three objectives, F, MS, and, TSS, we use an aggregation of them as the reward function \(\mathcal{R}\). The most straightforward aggregation is to take the product of the three as in Hallinan et al. (2023a). However, since the three objectives are independent, the probability of generating samples that have high scores in all three objectives is very low. Our preliminary experiments show that samples with high total rewards can also have low single-objective scores, which naturally results in preference pairs in which the "winning" outputs have lower single-objective scores. We refer to these as _reversed single-objective scores_. When the percentage of reversed single-objective scores is high, we observe a degradation in the corresponding objective after PO. To prevent the degradation in any objective, we propose to use a weighted product, which is given by

\[\mathcal{R}=\text{TSS}^{\alpha}\cdot\text{MS}^{\beta}\cdot\mathrm{F}^{\gamma} \tag{4}\]

where \(\alpha\), \(\beta\), and \(\gamma\) are temperature parameters.

We dynamically calculate \(\alpha\), \(\beta\), and \(\gamma\) based on the number of reversed single-objective scores in the preference pairs for each iteration. For convenience, we denote the number of reversed single-objective scores for each objective as \(r_{\text{TSS}}\), \(r_{\text{MS}}\), and \(r_{\text{FE}}\).10 We first set \(\beta=\gamma=1\) and set \(\alpha\) to be the smallest positive integer such that \(r_{\text{TSS}}<r_{\text{MS}}\) and \(r_{\text{TSS}}<r_{\text{F}}\). Then, we fix \(\alpha\) and \(\gamma\) and set \(\beta\) to be the largest positive integer such that \(r_{\text{MS}}>r_{\text{TSS}}\). Finally, we fix \(\alpha\) and \(\beta\) and set \(\gamma\) to be the largest positive integer such that \(r_{\text{F}}>r_{\text{TSS}}\) and \(r_{\text{F}}>r_{\text{MS}}\). We set an upper bound \(\tau_{\max}\) to \(\alpha\), \(\beta\), and \(\gamma\) to prevent \(\mathcal{R}\) from leaning too much to any objective.

Footnote 10: r\({}_{\text{TSS}}\), \(r_{\text{MS}}\), and \(r_{\text{F}}\) are functions of \(\alpha\), \(\beta\), and \(\gamma\), so we recalculate \(r\)s each time we change the value of \(\alpha\), \(\beta\), or \(\gamma\).

## 3 Experiments

We evaluate STAMP on two text style transfer datasets in both in-domain and out-of-domain settings and compare STAMP with the state-of-the-art baseline approaches. In this section, we detail the experimental setup and the model implementation.

### Datasets

We use two style transfer datasets in this work: (1) **Corpus of Diverse Styles (CDS)**Krishna et al. (2020), which contains non-parallel texts in 11 different styles, such as Shakespeare and English Tweets, and (2) **Grammarily's Yahoo Answers Formality Corpus (GYAFC)**Rao and Tetreault (2018), which contains non-parallel formal and informal texts for training and a small number of parallel transfer pairs for tuning and test. In this work, we only use non-parallel texts with style labels for training, validation, and test.

To reduce computational costs, we use a subset of each dataset. Specifically, we sample 2000 texts per style for training, and 200 per style for validation. For CDS we sample 200 per style for test, while for GYAFC we sample 1000 per style. When constructing the end-to-end pseudo-parallel dataset \(\mathcal{D}_{\text{trf}}\), for each target style, we sample 200 and 20 source texts from each of the other styles for training and validation, respectively. In the in-domain testing, we transfer the test texts in each style to all 

other styles in the same dataset and calculate the total average scores and average scores grouped by the target style. In the out-of-domain testing, we transfer all test texts in each dataset to all styles in the other dataset and calculate the same scores. We elaborate on metric scores in SS 4.1.

Besides the style transfer datasets, we also use a paraphrase dataset, **ParaNMT**Wieting and Gimpel (2018) to train the paraphraser used for pseudo-parallel data generation. Specifically, we use the filtered version containing 75k paraphrase pairs in Krishna et al. (2020).

### Reward Models

We have a reward model for each of the three objectives, TSS, MS, and F. For convenience, we use the same notations to refer to the objective functions and the corresponding reward models in this paper.

Target Style Strength (TSS)We use a single style classifier, \(f_{\text{cls}}\) with multiple binary sigmoid classification heads to calculate the TSS for each target style. We train \(f_{\text{cls}}\) from the pre-trained RoBERTa-large model Liu et al. (2019) on the same training and validation splits. We use the sigmoid scores from the classification heads as the TSS scores which range from 0 to 1.

Meaning Similarity (MS)We assess the meaning similarity between the source text and the transferred text using the cosine similarity between the semantic embeddings of the two texts. The semantic embeddings are calculated using SBERT11Reimers and Gurevych (2019). Technically, the cosine similarity of two embeddings ranges from -1 to 1, but negative cosine similarity is very rare in our experiments since we always the similarity between two paraphrases. Following Hallinan et al. (2023), we clip negative values to 0 to ensure that MS ranges from 0 to 1.

Footnote 11: We use the variant with the best sentence embedding performance, which is all-mpnet-base-v2.

Fluency (F)To measure the fluency of a text, we use a text classifier12 trained on the Corpus of Linguistic Acceptability (CoLA) Warstadt et al. (2019). The softmax score of the "grammatical" class is used as the F score which also ranges from 0 to 1.

Footnote 12: [https://huggingface.co/cointegrated/roberta-large-cola-krishna2020](https://huggingface.co/cointegrated/roberta-large-cola-krishna2020)

### Baseline Approaches

We compare STAMP with 4 strong baselines: GPT prompting Reif et al. (2022), STRAP Krishna et al. (2020), STEERHallinan et al. (2023), and ASTRAPOPLiu et al. (2024). The first two are the most widely used training-free and SFT approaches, while the latter two are the two SOTA models.

GPT prompting uses the zero- and few-shot capability of GPT-3.5-turbo to transfer texts to the target style given just the name of the style and 5 target style exemplars (5-shot) or no exemplars (zero-shot).

STRAP transfers a text by paraphrasing the text with a diverse paraphraser followed by an inverse paraphraser trained on pseudo-parallel transfer data generated by the diverse paraphraser.

STEER generates pseudo-parallel data using an expert-guided generation technique Liu et al. (2021), and trains an end-to-end style transfer model on the generated data using a reinforcement learning algorithm Lu et al. (2022).

ASTrapop adopts the same paraphrase-and-inverse-paraphrase pipeline as STRAP but trains the inverse paraphraser using policy optimization or PO to directly maximize the target style strength.

### Implementation Details

We implement all Seq2Seq models in STAMP, including the paraphraser and all transfer models, as decoder-only Seq2Seq models Wolf et al. (2019) based on pre-trained LLaMA-2-7B Touvron et al. (2023). The input and output are concatenated together with a separator token "[SEP]." For the unified transfer model \(f_{\text{SFT}}\), we prepend a style code for the target style (e.g., "[SHAKESPEARE]" and "[FORMAL]") to the input to control the output style. We use CPO Xu et al. (2024) in the multi-iteration PO stage. We choose CPO instead of the most popular PO algorithm, DPO Rafailov et al. (2023), since CPO has been shown to be more efficient and effective Xu et al. (2024); Liu et al. (2024). Also, compared to DPO, CPO has an additional negative log-likelihood term that is found to be significant for multi-iteration preference optimization Pang et al. (2024). We stop PO training at the iteration where the validation TSS starts to decrease and use the model from the previous iteration as the final model. For fairness, all non-GPT baselines are also implemented based on LLaMA-2-7B and use the same paraphraser as STAMP. We use gpt-3.5-turbo-0125 for all GPT-based approaches. See SS B for hyperparameters, training runtime, and GPT zero- and few-shot prompts.

 



## 4 Results

In this section, we present the quantitative experimental results and a qualitative case study. Because of the limited resources, we conduct all experiments for a single run and perform t-tests on the results.13

Footnote 13: See § B.1 for details.

### Automatic Evaluation

Automatic evaluation results on in-domain input are shown in Table 1, using the same reward models introduced in SS 3.2 to calculate TSS, MS, and F. To assess the overall performance, we use a single aggregate score \({\rm Agg}_{\cdot}={\rm TSS}\cdot{\rm MS}\cdot{\rm F}\).14 According to the aggregated score (Agg.), STAMP outperforms all baselines on the overall performance by a large margin on both datasets. Looking at the per-objective scores, STAMP has the best target style strength (TSS) and meaning similarity (MS), but its fluency (F) is relatively lower, and this disadvantage is more obvious on the CDS dataset. STEER has the best overall performance (Agg.) among the baselines on both datasets, while the overall performance of other baselines are mixed across the two datasets. For the breakdown scores on each subset in CDS and GYAFC, please see SSA.4.

Footnote 14: Note that the average Agg. on the test set is the average of Agg. for each transfer pair, not a simple product of average TSS, MS, and F.

Table 2 shows automatic evaluation results of the 'out-of-domain' style transfer experiments, in which we transfer the texts in each dataset to the styles in the other dataset, in order to determine whether our results hold up when transferring between styles of different provenance. They do; the out-of-domain results are generally consistent with the in-domain results. The best model in each column in Table 2 is the same as Table 1, which is also true for the second best model in most columns. Also, STAMP still has the best TSS, MS, and aggregated score (Agg.) among all approaches, and STEER still has the best overall performance (Agg.) among the baselines.

We also show that STAMP models are not overfitted on the training rewards by evaluating them on alternative metrics unseen during training and are robust to different hyperparameters by training the models with perturbed hyperparameters in SS A.1 and SS A.2. Besides, STAMP is more computationally efficient than the strongest baseline, STEER. When compared head-to-head using an identical base model but varying only the core design choices (DExpert data generation and Quark for STEER vs. STRAP data generation and iterative CPO for STAMP), we find that STAMP reaches parity with STEER in 43% and 82% of the training time on CDS and GYAFC, and converges efficiently with stronger performance.15

Footnote 15: See § B.4 for details.

### Human Evaluation

We conduct a human evaluation on the CDS dataset for STAMP, the best-performing baseline (STEER), and the best GPT-prompting baseline (GPT 5-shot) to assess their performance on the three style transfer objectives: \({\rm TSS}_{h}\), \({\rm MS}_{h}\), and \({\rm F}_{h}\).16 For \({\rm TSS}_{h}\), we show 5 exemplars for the style of the input text and 5 exemplars for the target style, and ask the annotator to select the style of the transferred text out of these two styles. The sample gets a score of 1 if the target style is selected, and 0 otherwise. For \({\rm MS}_{h}\) and \({\rm F}_{h}\), we ask whether the transferred text has a similar meaning to the input text and whether the transferred is fluent, respectively, and collect

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \multirow{2}{*}{Approach} & \multicolumn{4}{c}{CDS} & \multicolumn{4}{c}{GYAFC} \\ \cline{2-9}  & TSS & MS & F & Agg. & TSS & MS & F & Agg. \\ \hline \hline GPT zero-shot & 0.189\({}^{\ddagger}\) & 0.705\({}^{\ddagger}\) & 0.803\({}^{\dagger}\) & 0.104\({}^{\ddagger}\) & 0.672\({}^{\ddagger}\) & 0.788\({}^{\ddagger}\) & **0.968** & 0.489\({}^{\ddagger}\) \\ GPT 5-shot & 0.199\({}^{\ddagger}\) & 0.735\({}^{\dagger}\) & 0.805\({}^{\ddagger}\) & 0.112\({}^{\ddagger}\) & 0.667\({}^{\ddagger}\) & 0.800\({}^{\dagger}\) & 0.965 & 0.495\({}^{\ddagger}\) \\ STRAP & 0.382\({}^{\ddagger}\) & 0.626\({}^{\ddagger}\) & 0.759\({}^{\ddagger}\) & 0.158\({}^{\ddagger}\) & 0.618\({}^{\ddagger}\) & 0.735\({}^{\ddagger}\) & 0.913\({}^{\ddagger}\) & 0.409\({}^{\ddagger}\) \\ STEER & 0.654\({}^{\dagger}\) & 0.672\({}^{\ddagger}\) & **0.905** & 0.395\({}^{\dagger}\) & 0.951 & 0.776\({}^{\ddagger}\) & 0.930\({}^{\dagger}\) & 0.686\({}^{\ddagger}\) \\ ASTRAPOP & 0.542\({}^{\ddagger}\) & 0.600\({}^{\ddagger}\) & 0.755\({}^{\ddagger}\) & 0.221\({}^{\ddagger}\) & 0.783\({}^{\ddagger}\) & 0.734\({}^{\ddagger}\) & 0.924\({}^{\ddagger}\) & 0.525\({}^{\ddagger}\) \\ \hline STAMP & **0.746** & **0.801** & 0.801\({}^{\dagger}\) & **0.474** & **0.958** & **0.921** & 0.941\({}^{\ddagger}\) & **0.828** \\ \hline \end{tabular}
\end{table}
Table 1: The automatic evaluation results on in-domain inputs on the CDS and the GYAFC datasets. The best and the 2\({}^{\text{nd}}\) best scores in each column are shown in **bold** and underline, respectively. “\({}^{\ddagger}\)” and “\({}^{\ddagger}\)” indicate the score is significantly (\(p<0.05\)) worse than the best score and the top 2 scores in the same column, respectively, determined by resampling t-test.



the answers using a three-level Likert scale ranging from 0 to 2, which is then halved to fit in the 0 to 1 range. See SS B.5 for the detailed instructions used in the human evaluation. We randomly choose 5 samples from each of the 11 target styles for each of the three models, which yields 165 samples in total, and collect up to three annotations for each sample. Seven volunteer NLP experts are recruited for annotation.

We perform an independent sample t-test on the annotation results and find statistically significant differences in MS\({}_{h}\) and F\({}_{h}\) but not in TSS\({}_{h}\).17 which is in line with our expectation since the style classification has been found to be hard for untrained humans18 Krishna et al. (2020); Hallinan et al. (2023a). Therefore, following Krishna et al. (2020) and Hallinan et al. (2023a), we calculate the quasi aggregated score Agg.\({}_{\sim h}\) using TSS,19 MS\({}_{h}\), and F\({}_{h}\). Formally, Agg.\({}_{\sim h}=\text{TSS}\cdot\text{MS}_{h}\cdot\text{F}_{h}\). As shown in Table 3, STAMP has the best meaning similarity (MS\({}_{h}\)) and overall performance (Agg.\({}_{\sim h}\)), but its fluency is worse than STEER and GPT 5-shot transfer, which is consistent with the automatic evaluation results.

Footnote 17: See § A.3 for the raw human evaluation scores and the result of the t-test.

Footnote 18: We still conduct human study for style because we set up the task as a simpler verification task to see whether we can get meaningful results.

### Ablation Studies

In this section, we demonstrate the effects of our four main contributions in STAMP: multi-iteration PO, hope-and-fear sampling, weighted reward aggregation, and end-to-end pseudo-parallel data generation.

Multi-iteration PO & Weighted \(\mathcal{R}\)We show the performance evolution of STAMP and STAMP with unweighted \(\mathcal{R}\) over the multi-iteration PO training in Figure 2. In general, the overall performance (Agg.) of both models keeps increasing over the iterations, which indicates the effectiveness of multi-iteration optimization. STAMP with unweighted \(\mathcal{R}\) performs slightly better than STAMP, but it has a severe degradation in meaning similarity (MS), and the scores in the three objectives have a substantial difference after training. In contrast, with the weighted reward aggregation, STAMP shows a higher stability in all scores. Only fluency (F) exhibits a slight decrease, and scores in all three objectives converge to a similar value at the end of the training.

Hope-and-fear SamplingThe results of hoped-and-fear sampling ablation are shown in Table 4. As mentioned in SS 2.4.2, we do not use the model score term in hope-and-fear sampling for preference pair construction since it does not improve the performance, which can be observed from the "\(\tau_{\mathcal{M}}=0.1\)" row in Table 4. The last three rows in Table 4 show that both dropping over-generation (\(k_{\text{PO}}=2\)) and using a random other sample (Random \(l^{l}\)) or the sample with the second highest reward (High \(\mathbf{t}^{l}\)) as the "losing" sample undermine the overall performance of STAMP.

Pseudo-parallel Data GenerationWe demon

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \multirow{2}{*}{Approach} & \multicolumn{4}{c}{CDS} & \multicolumn{4}{c}{GYFace} \\ \cline{2-9}  & TSS & MS & F & Agg. & TSS & MS & F & Agg. \\ \hline \hline GPT zero-shot & 0.2461  & 0.6571  & 0.8551  & 0.1381  & 0.6721  & 0.7521  & **0.909** & 0.4551  \\ GPT 5-shot & 0.2891  & 0.7081  & 0.8681  & 0.1751  & 0.7221  & 0.7521  & 0.902 & 0.4861  \\ STRAP & 0.4261  & 0.6291  & 0.8104  & 0.1941  & 0.6921  & 0.6891  & 0.8521  & 0.4021  \\ STEER & 0.6541  & 0.7061  & 0.8021  & 0.4261  & 0.8501  & 0.7341  & 0.875 & 0.5441  \\ ASTRAPO & 0.5791  & 0.6061  & 0.8081  & 0.8025  & 0.8161  & 0.6854  & 0.8631  & 0.4791  \\ \hline STAMP & **0.787** & **0.816** & 0.8771  & **0.562** & **0.964** & **0.864** & 0.8271  & **0.687** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The automatic evaluation results on out-of-domain inputs on the CDS and the GYAFC datasets. The best and the 2nd best scores in each column are shown in **bold** and underline, respectively. “!” and “!” indicate the score is significantly (\(p<0.05\)) worse than the best score and the top 2 scores in the same column, respectively, determined by resampling t-test.

 

strate the superiority of our two-step end-to-end pseudo-parallel data generation method by comparing the STAMP SFT model, \(f_{\text{SFT}}\), with the best-performing baseline SFT style transfer model, STRAP. The overall performance (Agg.) of the two models is shown in Table 5. With our method, the overall performance of \(f_{\text{SFT}}\) is much higher than STRAP on both datasets, which provides a better starting point for PO.

### Qualitative Case Study

We show an example from the CDS test set in Table 6 as a case study. In this example, we transfer a text in the style of music lyrics into the style of Shakespeare using STAMP and all baseline approaches. STAMP (iter. 9) maximally preserves the meaning of the original sentence and accurately reflects the target style using the words "tis", "o'er", and "That" with uppercase "T". Other approaches either fail to generate strong target style indicators or change the meaning of the original sentence to some extent. Moreover, the example also demonstrates that STAMP gradually improves the model performance over the multi-iteration training. Specifically, STAMP (iter. 1) successfully reflects Shakespeare's style by using the word "o'er" but reverses the meaning of the original sentence. STAMP (iter. 5) successfully captures the original meaning but suffers from hallucination. The final model, STAMP (iter. 9) fully preserves the original meaning and demonstrates a stronger Shake-sparean style.

## 5 Related Work

Text Style TransferDue to the lack of parallel style transfer data, only a limited number of studies address this task as a supervised or semi-supervised Seq2Seq task, which requires a certain amount of parallel data for training and/or tuning Zhu et al. (2010); Rao and Tetreault (2018); Wang et al. (2019); Shang et al. (2019); Xu et al. (2019); Zhang et al. (2020); Kim et al. (2022); Raheja et al. (2023). Although these approaches work well when parallel data is available, none generalize well to styles with no parallel data. As a result, most works in this area focus on unsupervised approaches that require only non-parallel data or even no data. These works mainly approach the task via latent representation disentanglement and manipulation Lample et al. (2019); Liu et al. (2019); John et al. (2019); Jin et al. (2020), style-related pattern editing Madaan

Figure 2: The value of iterative CPO on performance in STAMP and STAMP with unweighted \(\mathcal{R}\), shown on the CDS dataset (test split). Iteration 0 refers to the SFT model before PO.

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline \multirow{2}{*}{Approach} & \multicolumn{4}{c}{CDS} & \multicolumn{4}{c}{GYAFC} \\ \cline{2-7}  & TSS & MS & F & Agg. & TSS & MS & F & Agg. \\ \hline \hline STAMP & **0.746** & 0.801\({}^{\ddagger}\) & 0.801\({}^{\dagger}\) & **0.474** & 0.958\({}^{\ddagger}\) & 0.921\({}^{\dagger}\) & 0.941\({}^{\dagger}\) & **0.828** \\ \hline \(\tau_{\mathcal{M}}=0.1\) & 0.720\({}^{\dagger}\) & 0.796\({}^{\ddagger}\) & 0.800\({}^{\dagger}\) & 0.454\({}^{\dagger}\) & 0.965 & 0.910\({}^{\ddagger}\) & 0.943\({}^{\dagger}\) & 0.826 \\ \(k_{\text{PO}}=2\) & 0.745 & 0.688\({}^{\ddagger}\) & **0.816** & 0.411\({}^{\ddagger}\) & **0.970** & 0.878\({}^{\ddagger}\) & **0.947** & 0.804\({}^{\ddagger}\) \\ Random **t\({}^{\ddagger}\)** & 0.640\({}^{\ddagger}\) & **0.836** & 0.780\({}^{\ddagger}\) & 0.412\({}^{\ddagger}\) & 0.950\({}^{\ddagger}\) & 0.924\({}^{\dagger}\) & 0.937 & 0.822 \\ High **t\({}^{\ddagger}\)** & 0.592\({}^{\ddagger}\) & 0.826\({}^{\dagger}\) & 0.796\({}^{\dagger}\) & 0.384\({}^{\ddagger}\) & 0.928\({}^{\ddagger}\) & **0.936** & 0.932\({}^{\ddagger}\) & 0.810\({}^{\ddagger}\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Hope-and-fear sampling ablations, evaluated automatically on in-domain inputs on the CDS and the GYAFC datasets. The best and the 2\({}^{\text{nd}}\) best scores in each column are shown in **bold** and underline, respectively. “\({}^{\ddagger}\)” and “\({}^{\ddagger}\)” indicate the score is significantly (\(p<0.05\)) worse than the best score and the top 2 scores in the same column, respectively, determined by resampling t-test.

 

et al., 2019; Liu et al., 2020; Malmi et al., 2020), pseudo-parallel transfer data construction Krishna et al., 2020; Riley et al., 2021), policy optimization Gong et al., 2019; Liu et al., 2021b; Deng et al., 2022; Hallinan et al., 2023a; Liu et al., 2024), and LLM zero- or few-shot prompting Reif et al., 2022; Suzgun et al., 2022; Patel et al., 2023).

Among these approaches, two of the policy optimization based approaches, STEER Hallinan et al., 2023a) and ASTRAPOP Liu et al. (2024) achieve the best performance on text style transfer and authorship style transfer, respectively. Their high-level training frameworks both combine pseudoraparallel data generation and policy optimization, but their specific approaches differ. For pseudo-parallel data generation, STEER uses a paraphraser guided by an expert and an anti-expert, while ASRrapOP simply paraphrases the texts in the target style and uses these paraphrase-to-target transfer pairs. For policy optimization, STEER uses an RL algorithm, Quark, while ASTRAPOP tries three options: one RL algorithm, PPO Schulman et al., 2017), and two PO algorithms, DPO Rafailov et al., 2023) and CPO Xu et al., 2024a). Our framework shares the same high-level procedure with STEER and ASTRAPOP, but we design a new pseudo-parallel data generation method and also enhance the PO stage with multi-iteration training, weighted reward aggregation, and hope-and-fear preference pair construction, These enhancements dramatically improve the performance of STAMP over STEER and ASTRAPOP.

**Preference Optimization** PO Rafailov et al., 2023; Song et al., 2024a; Xu et al., 2024a) is a class of RL-free policy optimization algorithms which has been broadly applied to train generative language models on direct task objectives instead of the language modeling loss and is closely related to (pre-neural) machine translation objective tuning' Och (2003); Chiang et al., 2008; Hopkins and May, 2011). Rafailov et al. (2023) show that PO is more stable and efficient than traditional RL-based algorithms on sentiment generation and text summarization Rafailov et al., 2023). It has also been successfully applied to many other NLP tasks, such as training helpful and harmless assistants Song et al., 2024a), machine translation Xu et al. (2024a), and authorship style transfer Liu et al. (2024). Later works Xiong et al., 2023; Xu et al., 2024b; Yuan et al., 2024; Chen et al., 2024; Pang et al., 2024; Song et al., 2024b) extend the offline PO algorithms by performing the optimization for multiple iterations and further improve the performance of the models. In this work, we adopt the multi-iteration PO for STAMP and enhance it with weighted reward aggregation and hope-and-fear preference pair construction, which improve the effectiveness of multi-iteration PO training.

## 6 Conclusion

We present STAMP, a multi-iteration preference optimization training framework for text style transfer, in which an end-to-end pseudo-parallel data generation pipeline provides a strong reference model, a preference pair construction strategy improves the effectiveness of PO training, and weighted reward aggregation ensures balance across multiple objectives over multi-iteration training. We evaluate STAMP on two commonly used text style transfer datasets; demonstrating superior performance over all state-of-the-art style transfer approaches.

### Limitations

Although achieving the state-of-the-art performance on two text style transfer datasets, STAMP

\begin{table}
\begin{tabular}{l l} \hline \hline Original & I’m not sorry that it’s over \\ \hline \hline GPT zero-shot & I doth not lament the end that hath befallen us. \\ GPT 5-shot & I doth not lament the ending of this tale. \\ STRAP & I am not sorry That he is gone. \\ ASTRAPOP & Now is the winter of our discontent Made glorious summer by this sun of York. \\ STEER & I do not regret that it is done. \\ \hline STAMP (iter. 1) & I am sorry that’s it is o’er. \\ STAMP (iter. 5) & I am not sorry that it is over, Nor sorry that I did not know it. \\ STAMP (iter. 9) & I am not sorry That ’tis o’er. \\ \hline \hline \end{tabular}
\end{table}
Table 6: A style transfer example from the style of music lyrics to the style of Shakespeare.

 

has two main limitations. First, we observe repetitions and hallucinations in some transferred texts. The potential reason is that PO training increases the peakiness of the model, which means the probability of generating the tokens that are frequent in the target style increases disproportionately (Choshen et al., 2020; Kiegeland and Kreutzer, 2021). The occurrence of repetitions and hallucinations also indicates that our reward model cannot fully capture all aspects of the desired objectives. Two possible solutions are developing PO algorithms that are less vulnerable to the increased peakiness and developing better reward models. These are two promising directions for future studies but are out of the scope of the current work which focuses on the multi-iteration extension of existing preference optimization algorithms and the strategies for preference pair construction.

Second, as discussed in SS 4.3, the weighted reward aggregation method is effective on the CDS dataset but is not very useful on the GYAFC dataset because formality transfer is a relatively easier task, and it is more likely to generate high-quality samples with balanced single-objective scores. It could be useful to add a control mechanism to determine when using the weighted aggregation is beneficial to prevent overbalanced single-objective scores on easy tasks.

## Ethical Considerations

As a general text style transfer framework, STAMP can transfer texts to any target style given an adequate amount of non-parallel data, which means it can potentially be used to generate unethical texts such as transferring normal texts into an offensive or profane style. Moreover, although STAMP is not specifically designed for authorship transfer, it can still serve that purpose by transferring the texts into the style of a particular author, which can be unethical if used without authorization. However, privatization of an author's style can also be used to enable oppressed people to communicate freely without the fear of recrimination. In any case, as we and others show, the state of the art of style transfer is not yet advanced for either privacy or mimicry to be a significant concern in a deployed system. Our work is strictly intended for research and personal use on public or authorized data.

Some texts in the datasets used in this work (though collected and released elsewhere) contain words or ideas that may cause harm to others. We do not generally filter out those texts, so that we may maximally preserve the characteristics of different styles. However, for human studies, we remove all texts with personal identifiable information (PII) to ensure privacy and remove texts that contain profane language to minimize harm to human subjects. We exclude these texts instead of masking out PII or profane tokens, since masks may influence annotators' judgments regarding meaning similarity and fluency. The protocols of our human studies have been approved by an institutional review board.

## Acknowledgments

This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-2207220006, and in part by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112490374. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* Babakov et al. (2022) Nikolay Babakov, David Dale, Varvara Logacheva, and Alexander Panchenko. 2022. A large-scale computational study of content preservation measures for text style transfer and paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, Dublin, Ireland. Association for Computational Linguistics.
* Bo et al. (2021) Haohan Bo, Steven H. H. Ding, Benjamin C. M. Fung, and Farkhund Iqbal. 2021. ER-AE: Differentially private text generation for authorship anonymization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online. Association for Computational Linguistics.
* Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. PreprintarXiv:2401.01335.
* Cherry and Foster (2012) Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In 

ceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 427-436, Montreal, Canada. Association for Computational Linguistics.
* Chiang (2012) David Chiang. 2012. Hope and fear for discriminative training of statistical translation models. _Journal of Machine Learning Research_, 13(40):1159-1187.
* Chiang et al. (2008) David Chiang, Yuval Marton, and Philip Resnik. 2008. Online large-margin training of syntactic and structural translation features. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, pages 224-233, Honolulu, Hawaii. Association for Computational Linguistics.
* Choshen et al. (2020) Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020. On the weaknesses of reinforcement learning for neural machine translation. In _International Conference on Learning Representations_.
* Dale et al. (2021) David Dale, Anton Voronov, Daryna Dementieva, Varara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021. Text detoxification using large pre-trained neural models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7979-7996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Deng et al. (2022) Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3369-3391, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Gong et al. (2019) Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3168-3180, Minneapolis, Minnesota. Association for Computational Linguistics.
* Hallinan et al. (2023a) Skyler Hallinan, Faeze Brahman, Ximing Lu, Jaehun Jung, Sean Welleck, and Yejin Choi. 2023a. STEER: Unified style transfer with expert reinforcement. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7546-7562, Singapore. Association for Computational Linguistics.
* Hallinan et al. (2023b) Skyler Hallinan, Alisa Liu, Yejin Choi, and Maarten Sap. 2023b. Detoxifying text with MaRCo: Controllable revision with experts and anti-experts. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 228-242, Toronto, Canada. Association for Computational Linguistics.
* Hopkins and May (2011) Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 1352-1362, Edinburgh, Scotland, UK. Association for Computational Linguistics.
* Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learning to generate headlines with controlled styles. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5082-5093, Online. Association for Computational Linguistics.
* John et al. (2019) Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 424-434, Florence, Italy. Association for Computational Linguistics.
* Kiegeland and Kreutzer (2021) Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neural machine translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1673-1681, Online. Association for Computational Linguistics.
* Kim et al. (2022) Zae Myung Kim, Wanyu Du, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022. Improving iterative text revision by learning where to edit from other revision tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9986-9999, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Krishna et al. (2020) Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 737-762, Online. Association for Computational Linguistics.
* Laban et al. (2021) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021. Keep it simple: Unsupervised simplification of multi-paragraph text. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6365-6378, Online. Association for Computational Linguistics.
* Lample et al. (2019) Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In 

International Conference on Learning Representations_.
* Liu et al. (2021a) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021a. DE experts: Decoding-time controlled text generation with experts and anti-experts. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6691-6706, Online. Association for Computational Linguistics.
* Liu et al. (2022) Ao Liu, An Wang, and Naoaki Okazaki. 2022. Semi-supervised formality style transfer with consistency training. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4689-4701, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. (2019a) Dayiheng Liu, Jie Fu, Yidan Zhang, Christopher Joseph Pal, and Jiancheng Lv. 2019a. Revision in continuous space: Unsupervised text style transfer without adversarial learning. In _AAAI Conference on Artificial Intelligence_.
* Liu et al. (2024) Shuai Liu, Shantanu Agarwal, and Jonathan May. 2024. Authorship style transfer with policy optimization. _Preprint_, arXiv:2403.08043.
* Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. _Preprint_, arXiv:1907.11692.
* Liu et al. (2021b) Yixin Liu, Graham Neubig, and John Wieting. 2021b. On learning text style transfer with direct rewards. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4262-4273, Online. Association for Computational Linguistics.
* Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Linhuji Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. QUARK: Controllable text generation with reinforced unlearning. In _Advances in Neural Information Processing Systems_.
* Luo et al. (2023) Guoqing Luo, Yu Han, Lili Mou, and Mauajama Firdaus. 2023. Prompt-based editing for text style transfer. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 5740-5750, Singapore. Association for Computational Linguistics.
* Madan et al. (2020) Aman Madan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimal Prabhumoye. 2020. Politeness transfer: A tag and generate approach. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1869-1881, Online. Association for Computational Linguistics.
* Malmi et al. (2020) Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8671-8680, Online. Association for Computational Linguistics.
* Och (2003) Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_, pages 160-167, Sapporo, Japan. Association for Computational Linguistics.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Pang et al. (2024) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sakhbaaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. _Preprint_, arXiv:2404.19733.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Patel et al. (2023) Ajay Patel, Nicholas Andrews, and Chris Callison-Burch. 2023. Low-resource authorship style transfer: Can non-fanous authors be imitated? _Preprint_, arXiv:2212.08986.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Raheja et al. (2023) Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEdIT: Text editing by task-specific instruction tuning. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 5274-5291, Singapore. Association for Computational Linguistics.
* Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for normality style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 1 (Long Papers)_, pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
* Reid and Zhong (2021) Machel Reid and Victor Zhong. 2021. LEWIS: Levenshtein editing for unsupervised text style transfer. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3932-3944, Online. Association for Computational Linguistics 

Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In _Proceedings of the 6th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 837-848, Dublin, Ireland. Association for Computational Linguistics.
* Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Riley et al. [2021] Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh. 2021. TextSETTR: Few-shot text style extraction and tunable targeted restifying. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3786-3800, Online. Association for Computational Linguistics.
* Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _Preprint_, arXiv:1707.06347.
* Shang et al. [2019] Mingyue Shang, Piji Li, Zhenxin Fu, Lidong Bing, Dongyan Zhao, Shuming Shi, and Rui Yan. 2019. Semi-supervised text style transfer: Cross projection in latent space. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4937-4946, Hong Kong, China. Association for Computational Linguistics.
* Shetty et al. [2018] Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2018. AANT: Author attribute anonymity by adversarial training of neural machine translation. In _27th USENIX Security Symposium (USENIX Security 18)_, pages 1633-1650, Baltimore, MD. USENIX Association.
* Song et al. [2024a] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024a. Preference ranking optimization for human alignment. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 18990-18998.
* Song et al. [2024b] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024b. Trial and error: Exploration-based trajectory optimization for llm agents. _Preprint_, arXiv:2403.02502.
* Suzgun et al. [2022] Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022. Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2195-2222, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Touvron et al. [2019] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukasa Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenyn Fu, Brian Fuller, Cynthia Gao, Vedantuj Goswami, Namam Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todd Mihaloy, Pushkar Mishra, Igor Mobbyog, Yixin Nie, Andrew Poulton, Jeremy Rezenstein, Rashi Runta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadu, Sharan Narang, Aurelier Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. _Preprint_, arXiv:2307.09288.
* Wang et al. [2019] Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019. Harnessing pre-trained neural networks with rules for formality style transfer. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3573-3578, Hong Kong, China. Association for Computational Linguistics.
* Warstadt et al. [2019] Alex Warstadt, Anampreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. _Transactions of the Association for Computational Linguistics_, 7:625-641.
* Wieting and Gimpel [2018] John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 451-462, Melbourne, Australia. Association for Computational Linguistics.
* Wolf et al. [2019] Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. _Preprint_, arXiv:1901.08149.
* Xiong et al. [2023] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2023. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In _ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
* Xu et al. [2024a] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024a. Contrastive preference optimization: Pushing the boundaries of 

lm performance in machine translation. _Preprint_, arXiv:2401.08417.
* Xu et al. (2024b) Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2024b. Some things are more crime than others: Iterative preference optimization with the pairwise cringe loss. _Preprint_, arXiv:2312.16682.
* Xu et al. (2019) Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. _Preprint_, arXiv:1903.06353.
* Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. _Preprint_, arXiv:2401.10020.
* Zhang et al. (2020) Yi Zhang, Tao Ge, and Xu Sun. 2020. Parallel data augmentation for formality style transfer. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3221-3228, Online. Association for Computational Linguistics.
* Zhu et al. (2010) Zhenmin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_, pages 1353-1361, Beijing, China. Coling 2010 Organizing Committee.

## Appendix A More Experimental Results

### Alternative Automatic Evaluation

To show that the models trained with STAMP are not overfitted on the rewards used during training, we evaluate all models on a set of alternative metrics. Specifically, for TSS, we train a style classifier on the same data using a different base model, BERT-base-cased Devlin et al. (2019); for MS, we use a different semantic similarity model MIS20 Babakov et al. (2022); for F, we use a different off-the-shelf classifier21 trained on the CoLA dataset Warstadt et al. (2019).

Footnote 20: [https://huggingface.co/s-nlp/Mutual_Implication_Score](https://huggingface.co/s-nlp/Mutual_Implication_Score)

Footnote 21: [https://huggingface.co/textattack/distilbert-base-cased-CoLA](https://huggingface.co/textattack/distilbert-base-cased-CoLA)

The results are shown in Table 7, Table 8, and Table 9. In most cases, the alternative metrics and the main metrics agree on the top two models. Although several disagreements exist on the individual metrics, both sets of metrics agree that STAMP models have the best overall performance (highest aggregated score).

### Perturbed Hyperparameters

Although the STAMP training pipeline contains multiple steps of model training and data generation, it is robust to different datasets and commonly used hyperparameters. In Table 1 and Table 2, we show that, using the same set of hyperparameters, STAMP works well on two different datasets for both in-domain and out-of-domain inputs, which confirms STAMP's generalizability to different datasets without further hyperparameter tuning. Furthermore, we train STAMP models on the two datasets with perturbed hyperparameters. Specifically, for all SFT components, we decrease the learning rate from 5e-5 to 2e-5 and double the batch size; for CPO, we double the learning rate and halve the batch size; for data generation, we change the decoding temperature for \(D_{p\to t}\) and \(D_{s\to t}\) to 0.7 and 0.5, respectively. The results in Table 7, Table 8, and Table 9 show that the STAMP models trained with the perturbed hyperparameters have slightly different individual metric scores but they consistently demonstrate better overall performance (aggregated score) than all baseline models.

### More Human Evaluation Results

The raw scores from the human evaluation and the result of the t-test are shown in Table 10. No significant difference is found between any model pairs in TSS\({}_{h}\)22, but MS\({}_{h}\) and F\({}_{h}\) are generally consistent with the automatic evaluation results. Specifically, STAMP and GPT 5-shot transfer are significantly better than STEER in meaning similarity (MS), and STEER and GPT 5-shot transfer are significantly better than STAMP in fluency (F).

Footnote 22: which is expected since style classification is difficult for human annotators Krishna et al. (2020); Hallinan et al. (2023).

### Subsets Automatic Evaluation Scores

The fine-grained automatic evaluation scores on each subset in CDS and GYAFC are shown in Table 11 to Table 16 and Table 17 to Table 22, respectively.

## Appendix B More Implementation Details

### Statistical Significance Test

We conduct a resampling paired t-test for the automatic evaluation results and an independent t-test for the human evaluation results. For the resampling paired t-test, we randomly select 10 subsets of 100 samples from the test set and perform a paired t-test on the mean scores of the subsets between each pair of models. For the independent t-test, we use all available samples from the human study without resampling.

 



### Hyperparameters

We sample same-sized training and validation subsets for CDS and GYAFC, and use the same hyperparameters to train STAMP on the two datasets to reduce the cost for more hyperparameter searching. We list all hyperparameters for STAMP in Table 23, Table 24, Table 25, Table 31, and Table 32.

### GPT prompt templates

We elaborate on the prompts used for GPT zero- and 5-shot style transfer on CDS and GYAFC in Table 26 and Table 27, respectively.

### Hardware and Runtime

We train all components of STAMP using Nvidia A40-48GB GPUs. The number of GPUs and time used to train each model on each dataset are shown in Table 28. Furthermore, we calculate the total training time including SFT, CPO, and all data generation processes for STAMP and the strongest baseline STEER. The results are shown in Table 33. In general, STAMP is slower than STEER on GYAFC but faster on CDS. However, to ensure fairness, we compare STEER's runtime with the runtime required for STAMP to outperform

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{9}{c}{CDS} & \multicolumn{9}{c}{GYAFC} \\ \cline{2-13} Approach & \multicolumn{2}{c}{TSS} & \multicolumn{2}{c}{MS} & F & \multicolumn{2}{c}{TSS} & \multicolumn{2}{c}{MS} & \multicolumn{2}{c}{F} \\ \cline{2-13}  & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. \\ \hline \hline GTP zero-shot & 0.189 & 0.181 & 0.705 & 0.763 & 0.803 & 0.721 & 0.672 & 0.674 & 0.788 & 0.898 & **0.968** & 0.929 \\ GFP 5-shot & 0.199 & 0.197 & 0.735 & **0.781** & 0.805 & 0.748 & 0.667 & 0.669 & 0.800 & 0.896 & 0.965 & 0.923 \\ STRAP & 0.382 & 0.361 & 0.626 & 0.530 & 0.759 & 0.757 & 0.618 & 0.627 & 0.735 & 0.761 & 0.913 & 0.877 \\ STEER & 0.654 & 0.570 & 0.672 & 0.602 & **0.905** & **0.897** & 0.951 & 0.929 & 0.776 & 0.821 & 0.930 & **0.932** \\ ASTRAPOP & 0.542 & 0.513 & 0.600 & 0.498 & 0.755 & 0.737 & 0.783 & 0.788 & 0.734 & 0.767 & 0.924 & 0.872 \\ \hline STAMP & **0.746** & **0.665** & 0.801 & 0.754 & 0.801 & 0.764 & **0.958** & 0.922 & 0.921 & 0.935 & 0.941 & 0.906 \\ STAMP pert. & 0.699 & 0.644 & **0.821** & 0.777 & 0.829 & 0.771 & 0.957 & **0.934** & **0.930** & **0.938** & 0.948 & 0.886 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Main vs. alternative TSS, MS, and F scores on in-domain inputs on the CDS and the GYAFC datasets. STAMP pert. refers to the STAMP model trained with perturbed hyperparameters. The best and the 2nd best scores in each column are shown in **bold** and underline, respectively.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{9}{c}{CDS} & \multicolumn{9}{c}{GYAFC} \\ \cline{2-13} Approach & \multicolumn{2}{c}{TSS} & \multicolumn{2}{c}{MS} & F & \multicolumn{2}{c}{TSS} & \multicolumn{2}{c}{MS} & \multicolumn{2}{c}{F} \\ \cline{2-13}  & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. \\ \hline \hline GTP zero-shot & 0.246 & 0.227 & 0.657 & 0.818 & 0.855 & 0.777 & 0.672 & 0.663 & 0.752 & 0.816 & **0.909** & 0.895 \\ GPT 5-shot & 0.289 & 0.276 & 0.708 & 0.839 & 0.868 & 0.801 & 0.722 & 0.711 & 0.752 & 0.815 & 0.902 & 0.882 \\ STRAP & 0.426 & 0.413 & 0.629 & 0.624 & 0.810 & 0.798 & 0.692 & 0.690 & 0.689 & 0.622 & 0.852 & 0.849 \\ STEER & 0.654 & 0.589 & 0.706 & 0.741 & **0.927** & **0.904** & 0.850 & 0.822 & 0.734 & 0.714 & 0.875 & **0.899** \\ ASTRAPOP & 0.579 & 0.557 & 0.606 & 0.602 & 0.808 & 0.778 & 0.816 & 0.809 & 0.685 & 0.648 & 0.863 & 0.836 \\ \hline STAMP & **0.787** & **0.711** & 0.816 & 0.840 & 0.877 & 0.825 & **0.964** & 0.917 & 0.864 & 0.853 & 0.827 & 0.814 \\ STAMP pert. & 0.695 & 0.647 & **0.861** & **0.873** & 0.903 & 0.821 & **0.964** & **0.923** & **0.870** & **0.860** & 0.829 & 0.816 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Main vs. alternative TSS, MS, and F scores on out-of-domain inputs on the CDS and the GYAFC datasets. STAMP pert. refers to the STAMP model trained with perturbed hyperparameters. The best and the 2nd best scores in each column are shown in **bold** and underline, respectively.

\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline \hline  & \multicolumn{9}{c}{CDS} & \multicolumn{9}{c}{GYAFC} \\ \cline{2-13}  & \multicolumn{2}{c}{In-domain} & \multicolumn{2}{c}{Out-of-domain} & \multicolumn{2}{c}{In-domain} & \multicolumn{2}{c}{Out-of-domain} \\ \cline{2-13}  & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. & Main & Alt. \\ \hline \hline GPT zero-shot & 0.104 & 0.095 & 0.138 & 0.139 & 0.489 & 0.554 & 0.455 & 0.486 \\ GPT 5-shot & 0.112 & 0.108 & 0.175 & 0.180 & 0.495 & 0.543 & 0.486 & 0.511 \\ STRAP & 0.158 & 0.127 & 0.194 & 0.187 & 0.409 & 0.412 & 0.402 & 0.375 \\ STEER & 0.395 & 0.304 & 0.426 & 0.392 & 0.686 & 0.711 & 0.544 & 0.527 \\ ASTRAPOP & 0.221 & 0.173 & 0.259 & 0.244 & 0.525 & 0.521 & 0.479 & 0.435 \\ \hline STAMP & **0.474** & **0.379** & **0.562** & **0.488** & 0.828 & **0.780** & 0.687 & 0.637 \\ STAMP pert. & 0.469 & 0.378 & 0.538 & 0.458 & **0.842** & 0.773 & **0.693** & **0.644** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Main vs. alternative aggregated scores on the CDS and the GYAFC datasets. STAMP pert. refers to the STAMP model trained with perturbed hyperparameters. The best and the 2nd best scores in each column are shown in **bold** and underline, respectively.



STEER and find that STAMP can achieve better performance than STEER in a much shorter time on both datasets (STEER vs. STAMP op. in Table 33), which indicates that STAMP is a more efficient training framework than STEER.

### Human Evaluation Instructions

The instructions used in the human evaluation for all three objectives are shown in Table 30 including the questions asked and the detailed explanation for each level in the Likert scale.

## Appendix C Scientific Artifacts

### Use of Existing Artifacts

The existing artifacts used in this work and their licenses are listed in Table 29. Our use of the existing artifacts is consistent with their intended use specified by their licenses.

 



\begin{table}
\begin{tabular}{l c c c c} \hline  & TSS & MS & F & Agg. \\ \hline \hline AAE Tweets & 0.806 & 0.889 & 0.788 & 0.561 \\ Bible & 0.643 & 0.640 & 0.830 & 0.312 \\
1810-1830 English & 0.764 & 0.799 & 0.807 & 0.490 \\
1890-1910 English & 0.439 & 0.812 & 0.875 & 0.311 \\
1990-2010 English & 0.920 & 0.819 & 0.873 & 0.660 \\ James Joyce & 0.844 & 0.859 & 0.825 & 0.596 \\ Lyrics & 0.545 & 0.806 & 0.815 & 0.357 \\ Romantic Poetry & 0.776 & 0.806 & 0.766 & 0.470 \\ Shakespeare & 0.740 & 0.792 & 0.686 & 0.392 \\ Switchboard & 0.920 & 0.811 & 0.721 & 0.534 \\ English Tweets & 0.810 & 0.784 & 0.831 & 0.529 \\ \hline Overall & 0.746 & 0.801 & 0.801 & 0.474 \\ \hline \end{tabular}
\end{table}
Table 16: The automatic evaluation results for STAMP on in-domain inputs on all subsets in CDS.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Parameter & \(f_{PO}\) \\ \hline learning rate & 2e-6 \\ \(\beta\) & 0.1 \\ batch size & 32 \\ \# epochs & 16 \\ \(k_{\text{FO}}\) & 10 \\ \(N_{\text{iter}}\) & 10 \\ \hline \hline \end{tabular}
\end{table}
Table 23: Training hyperparameters for all supervised fine-tuned models.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & TSS & MS & F & Agg. \\ \hline \hline Formal & 0.972 & 0.734 & 0.939 & 0.673 \\ Informal & 0.931 & 0.817 & 0.921 & 0.699 \\ \hline Overall & 0.951 & 0.776 & 0.930 & 0.686 \\ \hline \hline \end{tabular}
\end{table}
Table 20: The automatic evaluation results for STEER on in-domain inputs on all subsets in GYAFC.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Parameter & \(f_{pos}\) & \(f_{para}\) & \(f_{p\to t}\) & \(f_{s\to t}\) \\ \hline learning rate & 5e-5 & 5e-5 & 5e-5 \\ batch size & 32 & 32 & 8 & 16 \\ \# epochs & 6 & 10 & 6 & 12 \\ \hline \hline \end{tabular}
\end{table}
Table 22: The automatic evaluation results for STAMP on in-domain inputs on all subsets in GYAFC.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & TSS & MS & F & Agg. \\ \hline \hline AAE Tweets & 0.431 & 0.651 & 0.648 & 0.164 \\ Bible & 0.736 & 0.273 & 0.793 & 0.137 \\
1810-1830 English & 0.401 & 0.659 & 0.823 & 0.212 \\
1890-1910 English & 0.263 & 0.679 & 0.879 & 0.159 \\
1990-2010 English & 0.508 & 0.684 & 0.897 & 0.316 \\ James Joyce & 0.472 & 0.668 & 0.754 & 0.224 \\ Lyrics & 0.628 & 0.637 & 0.820 & 0.317 \\ Romantic Poetry & 0.807 & 0.595 & 0.583 & 0.266 \\ Shakespeare & 0.602 & 0.460 & 0.636 & 0.152 \\ Switchboard & 0.837 & 0.625 & 0.656 & 0.334 \\ English Tweets & 0.275 & 0.673 & 0.810 & 0.153 \\ \hline Overall & 0.542 & 0.600 & 0.755 & 0.221 \\ \hline \hline \end{tabular}
\end{table}
Table 19: The automatic evaluation results for STRAP on in-domain inputs on all subsets in GYAFC.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & TSS & MS & F & Agg. \\ \hline \hline Formal & 0.975 & 0.725 & 0.962 & 0.680 \\ Informal & 0.368 & 0.851 & 0.974 & 0.298 \\ \hline Overall & 0.672 & 0.788 & 0.968 & 0.489 \\ \hline \hline \end{tabular}
\end{table}
Table 17: The automatic evaluation results for GPT zero-shot on in-domain inputs on all subsets in GYAFC.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & TSS & MS & F & Agg. \\ \hline \hline AAE Tweets & 0.431 & 0.651 & 0.648 & 0.164 \\ Bible & 0.736 & 0.273 & 0.793 & 0.137 \\
1810-1830 English & 0.401 & 0.659 & 0.823 & 0.212 \\
1890-1910 English & 0.263 & 0.679 & 0.879 & 0.159 \\
1990-2010 English & 0.508 & 0.684 & 0.897 & 0.316 \\ James Joyce & 0.472 & 0.668 & 0.754 & 0.224 \\ Lyrics & 0.628 & 0.637 & 0.820 & 0.317 \\ Romantic Poetry & 0.807 & 0.595 & 0.583 & 0.266 \\ Shakespeare & 0.602 & 0.460 & 0.636 & 0.152 \\ Switchboard & 0.837 & 0.625 & 0.656 & 0.334 \\ English Tweets & 0.275 & 0.673 & 0.810 & 0.153 \\ \hline Overall & 0.542 & 0.600 & 0.755 & 0.221 \\ \hline \hline \end{tabular}
\end{table}
Table 15: The automatic evaluation results for ASTRAPOP on in-domain inputs on all subsets in CDS.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Parameter & \(f_{pos}\) & \(f_{para}\) & \(f_{p\to t}\) & \(f_{s\to t}\) \\ \hline learning rate & 5e-5 & 5e-5 & 5e-5 \\ batch size & 32 & 32 & 8 & 16 \\ \# epochs & 6 & 10 & 6 & 12 \\ \hline \hline \end{tabular}
\end{table}
Table 23: Training hyperparameters for all supervised fine-tuned models.





\begin{table}
\begin{tabular}{l l} \hline \hline  & Rewrite the following sentence in a(n) (in)formal style. \\ Zero-shot & Original Sentence: [input text] \\  & Rewitten Sentence: \\ \hline  & Here are some examples of sentences in a(n) (in)formal style: \\  & [example 1] \\  & ...... \\
5-shot & [example 5] \\  & Rewrite the following sentence in a(n) (in)formal style. \\  & Original Sentence: [input text] \\  & Rewritten Sentence: \\ \hline \hline \end{tabular}
\end{table}
Table 26: GPT zero- and 5-shot prompts for style transfer on CDS.

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & ParaNMT & \multicolumn{4}{c}{CDS} & \multicolumn{3}{c}{GYAFC} \\ \cline{2-9}  & \(f_{para}\) & \(f_{cls}\) & \(f_{p\to t}\) & \(f_{s\to t}\) & \(f_{PO}\) & \(f_{cls}\) & \(f_{p\to t}\) & \(f_{s\to t}\) & \(f_{PO}\) \\ \hline \# GPUs (A40s) & \(\times\)2 & \(\times\)2 & \(\times\)2 & \(\times\)4 & \(\times\)2 & \(\times\)2 & \(\times\)2 & \(\times\)2 \\ Times (hrs) & 3.4 & 0.4 & 1.1 & 1.0 & 35.2 & 0.1 & 0.2 & 0.2 & 7.4 \\ \hline \hline \end{tabular}
\end{table}
Table 27: GPT zero- and 5-shot prompts for style transfer on GYAFC.

\begin{table}
\begin{tabular}{c l c} \hline \hline Type & Name & \multicolumn{3}{c}{License} \\ \hline Dataset & CDS: Corpus of Diverse Styles & MIT \\  & GYAFC: Grammarly’s Yahoo Answers Formality Corpus & Custom (research-only) \\ \hline  & LLaMA-2-7B (6.7B) & Meta \\  & GPT-3.5-turbo-0125 (-) & MIT \\ Model & RoBERTa-large (355M) & MIT \\  & RoBERTa-large CoLA Classifier (355M) & MIT \\  & SBERT all-mpnet-base-v2 (109M) & Apache-2.0 \\ \hline Library & Transformers & Apache-2.0 \\  & PEFT & Apache-2.0 \\  & TRL & Apache-2.0 \\  & Sentence Transformers & Apache-2.0 \\ \hline \hline \end{tabular}
\end{table}
Table 28: Training hardware and runtime for each component in STAMP on CDS and GYAFC.

\begin{table}
\begin{tabular}{l l} \hline \hline Zero-shot & Rewrite the following sentence in a(n) (in)formal style. \\  & Original Sentence: [input text] \\  & Rewritten Sentence: \\ \hline  & Here are some examples of sentences in a(n) (in)formal style: \\  & [example 1] \\  & ..... \\
5-shot & [example 5] \\  & Original Sentence into the style of [targetstyle]. \\  & Original Sentence: [input text] \\  & Rewritten Sentence: \\ \hline \hline \end{tabular}
\end{table}
Table 29: Datasets, models, and software libraries used in this work. The number of parameters of each model is indicated in the parentheses next to the model name.



is only intended for research purposes, which is compatible with the original access conditions of all existing artifacts used in STAMP.

\begin{table}
\begin{tabular}{l l c} \hline \hline Pass & Question & Based on the examples above, what is the style of the following text? \\ \hline \multirow{4}{*}{MS\({}_{h}\)} & Similar & Most of the meaning (75\% or more) of the two passages is the same. \\ \cline{2-3}  & Somewhat Similar & Large portions (50-75\%) of the passages are the same, but there are significant sections that differ or are present in only one passage. \\ \cline{2-3}  & Not Similar & Only small portions (less than 50\%) of the passages are the same. \\ \cline{2-3}  & Question & How similar are the following two texts? \\ \hline \multirow{4}{*}{F\({}_{h}\)} & Fluent & Very clear, grammatical english (need not be formal); the meaning of the sentence is well understood. A small number of errors are ok. \\ \cline{1-1} \cline{2-3}  & Somewhat Fluent & There are grammatical errors, possibly numerous, but the meaning can be understood. \\ \cline{1-1} \cline{2-3}  & Not Fluent & The grammatical errors make it very difficult to understand the meaning. \\ \cline{1-1} \cline{2-3}  & Question & How fluent is the following text? \\ \hline \hline \end{tabular}
\end{table}
Table 30: Instructions used in the human evaluation.

\begin{table}
\begin{tabular}{l r r} \hline \hline Parameter & \multicolumn{2}{c}{Evaluation} \\ \hline top p & 1.0 \\ temperature & 0.5 & 0.7 \\ \cline{2-3}  & 20 & 90 \\ \cline{2-3}  & - & 8 \\ \hline \hline \end{tabular}
\end{table}
Table 31: Generation hyperparameters for dataset construction.

\begin{table}
\begin{tabular}{l r r} \hline \hline  & CDS & GYAFC \\ \hline STEER & 52.0 hrs \(\times\) 4 A40s & 7.2 hrs \(\times\) 2 A40s \\ STAMP op. & 22.2 hrs \(\times\) 4 A40s & 5.9 hrs \(\times\) 2 A40s \\ STAMP & 43.2 hrs \(\times\) 4 A40s & 10.8 hrs \(\times\) 2 A40s \\ \hline \hline \end{tabular}
\end{table}
Table 33: Total runtime (including dataset generation and training) for our reproduction of STEER and STAMP on CDS and GYAFC using identical models and architecture. STAMP op. indicates the training runtime point at which STAMP outperforms STEER.