STEER: Unified Style Transfer with Expert Reinforcement
Authors: Skyler Hallinan
Faeze Brahman
Ximing Lu
Jaehun Jung
Sean Welleck
Yejin Choi
Venue: EMNLP 2023 Findings (2023)
Keywords: style transfer
natural language generation
reinforcement learning
controllable decoding
TL;DR: We introduce STEER, a flexibile and efficient method for unified style transfer that overcomes the lack of parallel data via offline data generation followed by online, off-policy reinforcement learning.

Abstract: While text style transfer has many applications across natural language processing, the core premise of transferring from a single source style is unrealistic in a real-world setting. In this work, we focus on arbitrary style transfer: rewriting a text from an arbitrary, unknown style to a target style.

We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified frame-work developed to overcome the challenge of limited parallel data for style transfer. STEER involves automatically generating a corpus of style-transfer pairs using a product of experts during decoding. The generated offline data is then used to pre-train an initial policy before switching to online, off-policy reinforcement learning for further improvements via fine-grained reward signals. STEER is unified and can transfer to multiple target styles from an arbitrary, unknown source style, making it particularly flexible and efficient.

Experimental results on a challenging dataset with text from a diverse set of styles demonstrate state-of-the-art results compared to competitive baselines. Remarkably, STEER outperforms the 175B parameter instruction-tuned GPT-3 on overall style transfer quality, despite being 226 times smaller in size. We also show STEER is robust, maintaining its style transfer capabilities on out-of-domain data, and surpassing nearly all baselines across various styles. The success of our method highlights the potential of RL algorithms when augmented with controllable decoding to overcome the challenge of limited data supervision.

STEER: Unified Style Transfer with Expert Reinforcement
Skyler Hallinan Jaehun Jung Sean Welleck 9 Yejin Choi
Faeze Brahman 9 Ximing Lu
Paul G. Allen School of Computer Science & Engineering, University of Washington Allen Institute for AI Language Technologies Institute, Carnegie Mellon University hallisky@cs washington. edu
Abstract
Standard Style Transfer
While text style transfer has many applications across natural language processing, the core premise of transferring from a single source style is unrealistic in a real-world setting. In this work, we focus arbitrary style transfer: rewriting text from an arbitrary, unknown style to target style.
In-oomain
He doth loves me
loves me
Outsel-
#PeasAreYUMMY
???
X
domain
Style Transfor Model
Out-of- domain
Where you at?
???
We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified frame- work developed to overcome the challenge of limited parallel data for style transfer. STEER involves automatically generating corpus of style-transfer pairs using product of experts during decoding. The generated offline data is then used pre-traid tre-train initial policy be- fore switching to online, off-policy reinforce- ment learning for further improvements via tineigrinedd reward signals. STEER is unified and can transfer to multiple target styles from an arbitrary, unknown source style, making particularly flexible and efficient.
Unified Style Transfer
In-iiomain
doth loves me
loves me
In-iomainn
#PeasAroYUMMY
Poas taste good
Style Transfer Model
In-domain
Where you at?
Wher' yo' ba?
An overview of unified style transfer. In standard style transfer, models can only transfer from a single source style specified target style, struggling to transfer from out-of-domain texts. In contrast, unified style transfer models can transfer from an arbitrary source style to multiple target styles.
Figure 1:
Experimental results on a challenging dataset with text from a diverse set of styles demon- strate state-of-the-art results compared to com- petitive baselines. Remarkably, STEER out- performs the 175B parameter instruction-tuned GPT-3 on overall style transfer quality, despite being 226 times smaller in size. We also show STEER robust, maintaining its style transfer capabilities on out-of-domain data, and surpass- ing nearly all baselines across various styles. The success of our method highlights the po- tential of RL algorithms when augmented with controllable decoding to overcome the chal- lenge of limited data supervision."
increasing politeness (Madaan et al., 2020; Mukher- jee al., 2023), anonymizing authorship (Shetty et al., 2017: Patel et al., 2022). Previous work has mostly focused on one-to-one style transfer which involves rewriting text from one specific style to another while preserving meaning and flu- ency (Li et al., 2018; Sudhakar et al., 2019; Shen et al., 201 However, this approach may be less practical in real-world scenarios, where there are multiple and often unknown source styles user wishes transfer from.
1 Introduction
Style transfer has been widely explored in the NLP field due to its practical applications, such as mak- ing text more formal (Rao and Tetreault, 2018),
We focus on arbitrary style transfer, many-lo- one style transfer task, where the goal is to transfer text from an arbitrary, unknown style to target style using a single model (Reif et al., 2021; Kr- ishna et al., 2020). This a challenging task mainly
We release our code publicly at https://githon com/shallo shall 1 inan 1aaaa/ss/EEtttt eTrans
com/

---

due to the lack of large-scale, human-curated cor- pora for training. Furthermore, we design frame- work for training unified, many-lo-many style transfer model, which can do arbitrary style trans- fer to multiple target styles, as shown in Figure 1. To circumvent the lack of supervised data, recent approaches (Suzgun et al., 2022: Patel et al., 2022) heavily rely on large language models like GPT- Neo (Black al., 2022) and GPT-3 (Brown et al., 2020) in zero or few-shot settings. Though promis- ing and convenient, these approaches are limited by the high cost of API calls (OpenAI, 2023) and lack of reproducibility due to over-reliance on LLMs (Dean, 2023). Our method enhances the effectiveness of smaller, more accessible models for style transfer, broadening their adaptability and utility for the wider community.
styles that are unseen during training, where STEER consistently outperforms almost all baselines for every target style. The success of STEER demon- strates the effectiveness of reinforcement learning abetted by a high-quality, offline dataset in lieu of good initial policy.
Task: Unified Style Transfer
Conventionally, the goal of style transfer is to take an input text in a known source style Xs; and rewrite into some known target style X while preserv- ing meaning and fluency. However, this setting is unrealistic and may not cover real-world use cases where there are multiple and often unknown source styles. The goal of arbitrary style transfer is to instead transfer text from an arbitrary, unknown style to a text in the target style with meaning and fluency preservation. Formally, given S as the set of all possible style choices, this amounts to finding function f X x S - X, which takes input text and desired target style 8j. and outputs a modified text in the target style Xsy.
In this work, we present Unified Style Transfer with Expert Reinforcement Th STEER), novel, unified framework for many-to-one style transfer without supervision. Starting with a non-parallel corpus of text with various styles and a general paraphraser model, STEER first creates a diverse, sennoo-paallled dataset of style transfer pairs using product-of-expert decoding (Hinton, 2002; Liu al., 2021). This makes our framework efficient by eliminating the need for costly human-curated datasets. Next, STEER uses offline reinforcement learning (RL) with this data before switching to online, pollcy RL for further improvement. To reflect the varied properties of style transfer, we adapt the QUARK algorithm (Lu et al., 2022), incor- porating multiple reward models associated with different aspects such as style strength, fluency, and meaning similarity. Our framework is both practi- cal and flexible, enabling single model transfer arbitrary source styles to multiple target styles.
3 Unified Style Transfer with Expert Reinforcement
We introduce STEER, a novel two-stage framework for unsupervised unified style transfer. Our frame- work is illustrated in Figure and is composed of 1) expert-guided data generation to circumvent the challenge of obtaining supervised datasets at scale, and 2) offline reinforcement learning followed by online reinforcement learning to effectively align an initial policy with multiple reward functions related to the style transfer task.
In expert guieed data generation ($3.1), the goal is to automatically collect a diverse high-quality dataset D, of style transfer pairs using only a gen- eral paraphraser M, and corpus of diverse styles C. To this end, we follow an overgenerate-and filter approuch: we first generate large pool of candidate pairs from the paraphraser guided by style expert models a product-of-expert fashion (Hinton, 2002), then leave only pairs that qualify for the style transfer task (i.e., accurately trans- ferred style and semantically similar pairs). In online off- reinforcement learning ($3.2), we first update the paraphraser Mp as an initial policy using supervised learning on the collected dataset and then switch to online, off-policy learning for further data exploration and model improvements (Ramamurthy et al., 2022; Lu et al., 2022).
We apply STEER diverse dataset of 11 styles (Krishna et al., 2020), developing a unified style model capable of transferring text from any of the 11 styles to any other style in the corpus. Our fi- nal model is effective at transferring style while preserving fluency and semantic similarity for all source and target styles, beating strong baselines across a suite of automatic metrics for style transfer. In particular, across all styles 775M parameter model beats all baselines in overall style transfer quality, including the instruction-tuned 175B pa- rameter GPT-3 model (Ouyang et al., 2022). Fi- nally, we showcase the robustness of our model through evaluation on two oft-oi-domain source

---

(I) Expert-guided Data Generation
(2) Reinforcement Learning
Genoration
Over-generate
Step Offline
Dola Poot
Inputi R.
Quark
Machinc generatee, palrwisc cata
Stoor tawards sty and away from
#PaasYUMM/Y
PPaaYYUMMY
Stop i: Online RL
Peas taste
atee
Exploration
Reward
Use the policy generate new data prot.. D, Then, set D/ D.u.D
Scote the new data based Style
Paraphraser Banee Model
Style S, Expart
Style (AnbrtExpert
Filter
Elecnder
Decodde
Encender
Styee
5. Semiar mesning
Similar mearing
Similrr menning
X
Training
Logio
Sy Expert laghts
AmthSipert
Selected
Selected
Optimize pollcy max Ex,, spell (X.,,
6"
Ousput S4
Figure 2: An overview STEER. first use expert-guided data generation automatically generate candidate mtle-transfer pairs Xs, X., mapping from input of arbitrary style 2a, to rewrite 2a, in target style, by decoding with product of experts using a paraphraser MP and style-expert LMs. After filtering by quality metrics, we have diverse, high-quality dataset Dy. then train unified, many-to- to--aay style transfer model, using D for offline RI. before switching to online, off-policy RI to further optimize style transfer quality.
3.1 Expert-guided Data Generation
obtained by a product-of-experts:
We first leverage expert LMs to generate a high- quality, pecudo-parallel style transfer corpus.
Pe(xol<ee))
P(x)(ee)) x Pa(xe(
(1)
(e xee))
Generation For each target style S, first massively generate a diverse set of candidate style transfer pairs x,, + x., for all 8; that we collect pairs of transfers from each possible source style to the target style. To do so, we first- pass text Xs; from a candidate source style through a general (style-agnostic) paraphraser Mp, typi- cally resulting in normalized text X = (M,,), with little or no stylistic features (Krishna et al., 2020). To ensure that the X belongs the desired target style, we steer the paraphraser Mp gener- ation towards the target style and away from the source style during decoding. Intuitively, we ex- ploit the inherent capability of the paraphraser to faithfully rewrite input texts, while injecting stylis- tic control through guided-decoding."
where is hyperparameter controlling the strength of control over the base model P..
{ }. such
Within our problem setting, consider the gen- eral paraphraser Mp p as the base model, and two language models finetuned on texts belonging target style s, and source style s; as our expert and anti-expert models respectively. Given a text in candidate source style Xs,, we generate text in the target style Xw, via sampling from the probabil- ity distribution obtained in Eq. I. We repeat this expert-guided decoding for all the source and tar- get styles, resulting in dataset Dinit- practice, we over ever-generate data by repeating the generation procedure above with vast sweep hyperparam- eters, such as multiple sampling temperatures and decoding algorithms, so we can eventually filter and attain as many high-quality rewrites possible.
To do this, we leverage DEXPERTS decoding (Liu et al., 2021), a controllable text generation paradigm that enables steering towards and away from distinct attributes. DEXPERTS combines the distribution of a base autoregressive model Pe with those of an "expert" P. and/or "anti-expert" Pa model in a product of experts, which are trained desirable and undesirable attributes respectively. Given a prompt E<t. t the next token probability is
Filtering Not of the expert-guided generations in D Dinit are high-quality. We thus filter Dinit and retain the pairs that best represent the task of style transfer. We assess the quality each candidate style transfer pair in Dinat with three standard style transfer metrics:
Target Style Strength (TSS) of the genera- tion x,, is measured by the probability of the target class 8; with a RoBERTa-large classi- fier (Liu et al., 2019) trained on text from all
II practice, this procedure can be repeated to add any new target style to the datasct.

---

the styles in the corpus C. Both style strength and style accuracy have been used in previous work (Reif et al., 2021: Krishna et al., 2020); we opt for style strength, as it is more fine- grained than a binary measurement of accu- racy. Accordingly, we train our classifier in multi-label setup, such that the prediction probability of each target style can be inde- pendently evaluated.
Recently, online policy-based RL algorithms (Lu et al., 2022: Schulman al., 2017; Ramamurthy et al., 2022) have been shown effective opti- mizing language models towards given objective function. In the RI. framework, we refer to the model Ma as the policy and the objective function V as the reward. Generally, online RL algorithms conduct policy optimization with model-generated outputs while assuming reasonable degree of alignment between the output distribution of the initial policy and the optimal reward distribution. This alignment is necessary to produce generations with meaningful signals for RL training.
2. Fluency (F) of the generation Xs. is measured by the probability of being grammatically ac- ceptable via binary RoBERTa-large classi- fier trained on the CoLA dataset (Warstadt et al., 2018).
Due the absence of supervision, the closest ini- tial policy for our unified style transfer task would be the style-agnostic paraphraser Mp. However, this initial policy is still far away from the optimal reward distribution as the style transfer task falls beyond the capabilities of the paraphraser MM, making it unable to produce useful generations for RL optimization. To overcome this challenge, we propose first conducting offline RL training and then progressing to online RL training. Specifi- cally, prior to optimizing Mp with its own genera- tions, we first perform RL optimization on the style transfer data D, generated through expert-guine guiddncce ($3.1) I). Intuitively, the offline stage equips the ini- tial policy with a certain degree of style transfer capability before online stage further optimizes it towards generating rewrites with better quality.
3. Meaning Similarity (MS) between the in- put Xx; and rewritten text Xa; is measured via Sentence Transformess embedding distance (Reimers and Gurevych, 2019).
Following previous work (Krishna et al., 2020), for each candidate style transfer pair, we aggregate the three style metrics above into joint metric V that captures the overall quality:
(a,,,) st) TSS(E....) 8() F(w..) MS x..)
All three individual metrics are scalar values in the interval [0, 1)², which ensures also that V [0, 1].
Next, we filter our data create high-quality pool of training data D, for subsequent model train- ing. For each target style in Dinit, we sort the style- transfer pairs by their combined score V, then take the top-k examples. This sampling method ensures that the examples in the resulting dataset are the highest quality possible, but may also lead to lower diversity, it excludes scorrng generations. In practice, with multiple target styles in the initial pool of pairs Dinit, filtering done for each style separately, and the filtered data from each target style combined to form Dy.
In practice, we employ and adapt the RL algo- rithm QUARK (Lu et al., 2022) to accomplish the two-stage RL training. QUARK is an online, off- policy RL algorithm that has proven effective in var- ious text generation tasks. Notably, the off off-policy nature+ makes it possible be adapted for the offline RL stage. QUARK optimizes a reward func- tion through reward conditioning. Concretely, the algorithm alternates between 1) collecting samples with the current language model, 2) sorting them into quantiles based on their reward, with each quantile identified by a reward token prepended to the language model's ''s input, and 3) using stan- dard language modeling loss samples from each quantile conditioned on their reward token.
3.2 Reinforcement Learning
Next, we train a unified style transfer model by leveraging the generated corpus Dy- Concretely, our goal is to attain a rewriting model Ma which accepts an input with arbitrary style Xs, along with target style st and produces nigh-qualit rewrite X as evaluated by the joint metric V. formally:
When adapting QUARK to offline RL, we start by initializing the data pool with the style trans- fer corpus D, generated through expert-guidance rather than gathering generations from the initial
8" arg max Ex., pol (x., (x), 91) VxX,, X..: st)
"Sentence Transformers occasionally outputs negative scores; we set these to 0 to ensure a score in [0.
'Off-oolicy RI. evaluates and improves a policy different from the policy used for action selection (i.c. data generation).

---

GPT-2 Large
GPT-3 (text Savincci-000
Target Style
STEER
STRAP
P-A-R
k == 0
h
k:
10
AAE Twitter
42.6
7.4
3.8
23.2
11.2
25.4
22.7
Bible
44.0
26.9
6.6
5.2
16.0
20.2
21.0
1810-1820s
30.2
11.1
3.5
14.7
15.9
17.4
17.0
1890- 1900s
35.9
12.3
4.4
8.6
9.1
10.4
10.1
1990-2000s
42.3
16.6
4.3
7.9
13.0
17.5
17.2
English Twitter
41.2
8.0
5.5
35.0
23.6
32.0
29.5
James Joyce
20.4
11.8
5,4
3,4
1.3
1.6
2.6
Song Lyries
33.3
20.2
7.7
12.2
15.4
11.2
13.2
Romantic Poetry
20.4
15.7
2.8
1.1
3,4
6.2
4.9
Shakespeare
13.6
9.1
2.5
9.6
10.0
9.7
9.7
Switchboard
52.9
21.1
1.7
0.1
0.3
5.3
13.7
Overall
34.3
14.6
4.4
11.0
10.8
14.3
14.7
Table 1: Comparison of 1-way style transfer the CDS dataset measured by aggregate score V with different methods, including STRAP (Krishna et al., 2020) and P-A-R (Suzgun et al., 2022), using GPT-2 Large (774M), and GPT-3 (175B). Bold and underline denote the highest and the second -highst score respectively in each row.
policy. Afterward, we carry out the quantization and learning steps in the same manner the origi- nal QUARK. After completing the offline RL stage, we proceed with the online QUARK training by al- ternating between data generation with the updated policy, quantization and learning. In both stages, our training objective can be written as:
($4.3), experimental details ($4.4), main results (64.5), ablations ($4.6), and analysis of Dy ($4.7).
4.1 Datasets
We use the following datasets our experiments: 1) the Corpus of Diverse Styles (CDS; Krishna et al., 2020) is non non-parallel, diverse text corpus with 11 distinct styles such as Shakespeare and the Bible, 2) Grammarly's 's Yahoo Answers Formal- ity Corpus (GYAFC; Rao and Tetreault, 2018) is parallel corpus of formal and informal responses collected from the Yahoo Answers forum, and 3) the Yelp Review Dataset (Yelp: Shen al., 2017a) non- non-parallel corpus of user-reviews on various businesses and services from the Yelp with binary sentiment ratings of positive or negative. For more details on the datasets see Appendix B.
8" max E Eis boD log NNsss St, (eee))
where rv(-) denotes the quantized reward token corresponding to the reward score V(-) of the gen- erated rewrite. In online RI, D is expanded with samples from the improved policy each iteration. Additionally, we also explore integrating vec- torized reward function v(Xs; st) into the QUARK algorithm, rather than using the joint mul- tiplied scalar score V as the reward function. In this case, instead of conditioning on one reward token that corresponds to a quantized scalar score, condition reward vector composed of three reward tokens. These reward tokens represent quan- tized scores from the style, fluency and similarity metrics respectively. As we will show in the experi- ment section, we observe noticeable performance boost brought by vectorized QUARK in terms of reward optimization. We believe this is likely be- cause the vectorized reward provides additional fine-graines signals for optimization, which reflect the quality of each generated output with respect to individual evaluation metries.
4.2 Baselines
We use three competitive style-transfer baselines. Method-specific details are located in Appendix C:
Style Transfer via Paraphrasing (STRAP; Kr- ishna al., 2020) is unsupervised approach for arbitrary style transfer, which uses GPT-2 Large (Radford al., 2019) inverse paraphrasers.
Frompt-and-Reram (P-A-R: Suzgun et al., 2022) prompts some language model to generate k candidate style transfer texts, ranks them based quality, and returns the best one. We use P-A-R with GPT-2 Large.
4 Experiments
We detail our experiment setup, including the datasets (64.1), baselines ($4.2), evaluation metrics
GPT-3 (Brown et al., 2020; Ouyang et al., 2022) highly-capable class of decoder-only models,

---

GPTe-Large
GPT-3 -text-davineia
STEER
STRAP
P-A-R
k
k
5
10
Target Style
Inf.
For.
Inf.
For.
Inf.
For:
Inf.
For.
Inf.
For.
Inf.
For.
Inf.
For.
AAE Twitter
44.0
47.7
18.7
13.2
25.6
10.6
31.7
29.2
21.5
17.9
30
28.8
30.2
27.6
Bible
36.1
38.8
22
22.9
0.3
1.6
4,3
4.4
15.7
15.9
18.0
19.0
19.8
19.5
1810-1824 1820s
26.3
29.5
5,9
10.0
1.2
4.7
12.4
13.6
14.3
16,9
17.6
21.6
16.9
20.1
1890-1900s
33.5
34.7
10.0
13.4
4.4
11.0
9.9
11.8
13.9
13,8
14.6
14.4
13.8
13.3
1990-200s
50.2
56.2
22.6
32.1
11.8
31.4
16.7
20.7
28.5
32.5
31.5
34.7
28.4
32.8
English Twitter
46.1
54.1
20.1
22.1
32.4
33.5
37.4
41.8
30.1
29.5
34,9
36.4
32.5
35.0
James Joyce
22.3
22.8
10.9
13.2
3.2
7.9
2.9
3.3
2.7
2.3
3.1
2.5
3.3
2.8
Song Lyries
42.6
40.5
22.1
23.2
10.3
12.4
19.3
12.9
22.3
18.4
19.3
16.2
24.2
20.1
Romantic Poetry
13.5
12.9
8.9
10.8
0.8
0.9
2.0
1.1
5.2
4.3
7.0
4.7
6.0
3.9
Shakespeare
11.8
11.6
11.1
10.4
1.3
4.1
12.9
15.1
15.3
14.7
13.4
15.2
13.8
15.2
Switchboard
54.6
59.3
29.7
35.1
5.2
6.1
0.1
0.1
0.3
0.1
9.7
13.4
15.6
23.0
Overall
34.6
37.1
16.5
18.8
8.8
11.3
13.6
14.2
15.4
15.1
18.1
18.8
18.6
19.4
Table 2: Comparison of style transfer to each of the 11 styles in the CDS dataset measured by aggregate V from two out-of of--an styles from the GYAFC corpus. For. and Inf. denote the formal and informal styles respectively. Bold and underline denote the highest and the second-highest score respectively in each row.
particularly showing strong zero- and few-shot per- formance. We utilize GPT-3 as baseline both in zero-shot and few-shot (k: = 1,5, 10) setting. Specifically, we the instruction-tuned, 175B parameter engine text davinci--o0n 5
Specifically, we train unified model that can trans- fer arbitrary text to each style in the corpus C. We top-200K filtering for each target style, result- ing in | 2.2M. Finally, we evaluate style transfer each target style by transferring from 1000 test-set examples from every other style E C:; this results a total test-set size 10,000 for each target style.
4.3 Evaluation Metrics
To evaluate the quality of each style transfer pair, we use the same metrics introduced in $3.1: target style strength (TSS), fluency (F), meaning similar- ity (MS), and the aggregate metric V. For a set of style transfer pairs (i.e., over an entire data cor- pus), we report the average V. To ensure that the improvement from STEER is meaningful (i.e., to make sure model is not reward hacking), also report evaluation using alternative metries offlnne-hen-online RI., a unified model can outper- form other models of the same or even much larger size. The full results, including individual style transfer metrics, are in Appendix G.I.
Automatic Results We demonstrate automatic results on the CDS in Table 1. Across all tar- get styles, STEER outperforms all baselines V. the aggregate style transfer quality, including GPT-3, model 226 times larger, and STRAP, comparably-sized, non-unified baseline. This shows that with expert-guided data generation and
seen during training in Appendix F; these results corroborate our main findings in $4.5.
4.4 Experimental Details
For all non-GPT-3 baselines, we use GPT-2 large as the base language model. Specifically, for STEER, we use GPT-2 large for the paraphraser and for the expert models. Our main STEER results are with the vectorized QUARK variant (i.e, using fine- grained reward). More details are in Appendix A.3.
GPT-3 has its best relative performance on the Twitter and Shakespeare styles, but struggles other- wise. This shows the limitations of relying on large- scale general-purpose LLMs: in this case, GPT-3 excels transferring styles most likely to be highly prevalent in it's internet text corpus (Brown et al., 2020) However, it is unlikely generalize to more obscure styles unseen during training, even with few-shot examples. The poor performance the GPT-2-based P-A-R reinforces this, showing the un- reliability of prompting general-domain, pretrained LMs for style transfer, especially at smaller scales.
4.5
Style Transfer CDS
To evaluate STEER's ' capability on arbitrary style transfer, we use CDS, it has 11 diverse styles.
Moos capable model during conducting this work. "Though for each style transfer pair, V is equal to the product of TSS, F, and MS, once we take the corpus-average of these metrics we lose this equality guarantce
We also conduct an out-of-domaan evaluation to assess the robustness of each method to unseen

---

Aggrogpte [9 Tage
Flwecyy
Meoring Semisrtty (NSu)
Aggrogatee
Strength
Fluency
52.5
:6.3
290
25
25
193
35.3
32.5
70
01 (5.4
03.4
110
15-
200R 200K
0004
1006 200R
400K
[0CK
200K
40OK
1004 200K
450K
STRAP
STEER
Figure 4: Plots of the style transfer quality on CDS averaged across all 11 styles with varying &; the hyper- parameter used in the Top-k sampling strategy.
Figure 3: Style transfer quality V-H on CDS, averaged across all 1I styles, with fluency and meaning similarity human evaluation. TSS is automatically computed."
Input: Can't sleep at
Transfer: AAE 2000s
STHRK
meon, 't sloep at
GPT-3
unable to ohtain any rest, shaking my head disbelief.
inputs. Specifically, we use text from the two styles in GYAFC as inputs at testing time, employing the previously trained CDS model without further fine- tuning, and transfer each of the 11 styles in CDS. Our results are shown in Table 2: overall, STEER is the most robust method, outperforming all oth- ers in total score, V, for almost all target styles. STEER loses only to GPT-3 on the Shakespeare style; this may be due to the inherent knowledge of Shakespeare stored in GPT-3.
STRAP
don't want sleep.
P-A-R
sleep at all, smh.
Input
Yeah one one way or another
Transfer: switchheard - rom. postry
STEER
One way, or another, or both
GPT-3
Aye, one mayhap will find the way.
STRAP
And one way or the other
P-A-R
Yeah one one way or another
Iupot:
In his fear, he dare not face me
Transfer; lyrics - bible
STeaw
And he will noe dare face me: for fear of me is in his eyes.
GPT-3
And his fear grear, that be could not stand before me.
STRAP
For he that afraid of me is of me; but he that is of me is of him.
P-A-R
In fear he came and hid himself, because God was near to him
Human Evaluation We also conduct a human evaluation to verify the quality of the generations. We use a 3-point Likert Scale to evaluate style transfers (lyyer et al., 2018) on both meaning sim- ilarity (MSn) and fluency (Fi).. Afterwards, scale MSH and F₁ down [0, 1] and multiply them by the automaticcally-comppted TSS to attain V~H E [0, 1]. a partially human aggregate metric. Three NLP experts annotate 10 examples per 11 target styles from 3 models (330 total examples).
Table 3: Examples of style transfer pairs generated by STEER and other methods. GPT-3 is run with 10-shot.
off between style transfer accuracy and meaning preservation, both through empirical results (Suz- gun et al., 2022; Malmi et al., 2020; Wu et al., 2019; Li et al., 2018) and explicit mentions in discussions (Li et al., 2018; Xu et al., 2019; Wu et al., 2019; Hallinan et al., 2023). Intuitively, when transfer- ring from one style to another, some amount of semantic changes is unavoidable; as simple exam- ple, meaning similarity will be maximized when the input is naively copied.
Figure 3 shows our human evaluation results. In terms of individual metrics, STEER has better flu- ency than STRAP and maintains competitive fluency GPT-3, which is known to excel at generating human-like text (Brown et al., 2020). STEER also performs slightly better in meaning similarity than STRAP, but GPT-3 outperforms both of them signif- icantly. However, the TSS of STEER makes up for this and dwarves both the baselines. We think this is a reasonable trade-off: STEER sacrifices much less fluency and meaning preservation for much more style transfer strength.
Overall, human evaluation validates our main findings: STEER still beats both baselines in overall score V. These results show that GPT-3 is excellent at paraphrasing creating fluent and semantically similar rewrites, but not at transferring to multi- ple diverse styles, as it often struggles to convert to the target style. On the other hand, STEER is more versatile, maintaining moderate-to-stromg per- formance on all individual metrics, making it the strongest overall method.
Previous work has also demonstrated this trade-
"We use 1000 examples from each class
Finally, we show qualitative examples of gen- erations from different models in Table 3. In the examples, STEER produces style transfers that op- timize across all dimensions, while other methods
"As in Krishna et al., 2020, we do not conduct human evaluation for target style strength, the task to complex for untrained annotators unfamiliar with the target styles. Ap- pendix D.1 details an experiment we conducted which verifies this task's 's hardness for annotators
"Wo omit P- A-R from human evaluation due to its low performance in automatic metrics. See Appendix D for details.
Pairwise agreements are 94.8% and 98.8% for fluency and similarity respectively.

---

Reward Type
TSS
MS
Vl-rrmm entropy,
Coarse
19.9
42.8
79.0
62.6
3
MS
Fine-grained
27.4
47.0
83.6
71.5
D2 (STEER)
0.912
8.8
15.0
19.3
86.8
67.2
GYAPC
9.1
15.0
17.8
86.2
78.4
Table Style transfer quality on CDS, averaged across 11 target styles using STEER with coarse fine- grained reward. The highest values are denoted in bold.
Yelp CDS
0.958
9.6
16.4
21.3
87.2
0.946
10.0
17.0
20.4
83.2
Table 5: Data metrics on Dy (STBER) and other datasets.
optimize for only one or two.
control across each desired attribute, resulting much better style transfers overall.
4.6 Ablations
4.7 Analysis of y
We perform two ablation studies to analyze the effect of dataset size and reward design in STEER. All models are compared after 15K training steps:
We analyze Dy. the dataset resulting from the expert-guided dataset generation. First, we com- pare the lexical diversity D, against existing style transfer corpora. Following Gehrmann et al. (2021), I), we gauge the mean segmented token-type ratio over segemented length of N = 10 (MSTTR) and the 1/2/3-gram entropy of the training split of each corpus. We also assess the quality of style- transferred outputs in each corpus by assessing fluency (F) and meaning similarity (MS).
Dataset Size We investigate the effect of differ- ent dataset sizes the performance of STHER. Using the top-k sampling strategy, we vary & with k = 100K, 200K, and 400K and compare style transfer on CDS. Figure 4 shows the average re- sults transferring to 11 target styles in CDS from all other styles.
Interestingly, we do not observe direct scaling of style transfer performance with increasing dataset size; as the top-k value increases, the aggregate score V, target style strength TSS, and fluency all follow reverse U-shape curve.
Table shows comparisons of these metrics. The automatically-creatt DD is comparable to existing human-created datasets in diversity and in fluency. The average meaning similarity is also promis- ing, as it is within 85% of the value of GYAFC. This shows the potential of machine-gene -eeherrteeeee data when aided with creative decoding algorithms.
These results may indicate a trade-off between diversity and quality in the dataset D, used to train STEER: as the k value increases with top-k sam- pling, Dy becomes more diverse, but also includes samples with lower-quality, which may hurt model performance downstream. On the other hand, when k is too small, though the average quality of each example in Dy is higher, fewer diverse examples may hurt generalization. The optimal dataset has examples with sufficient variety and quality, en- abling the model learn a high-quality policy while staying resilient to various inputs.
Related Work
Style Transfer Due the absence of large-scale parallel corpora for text style transfer (TST), prior work has focused on unsupervised methods de- signed for non- non-parallel datascts (Dai et al., 2019; Luo et al., 2019). Most of these efforts focus on disentangling the representation of content and the style of a given text, either through an auxiliary discriminator to classify text attributes (Hu et al., 2018: Shen al., 2017b), by training with a policy gradient (Xu et al., 2018: Gong et al., 2019). Recent work has leveraged the generation capa- bilities of LMs for TST: Krishna et al. (2020), cre- ate a pseudo-parallel corpus by paraphrasing text from style, then training an inverse paraphraser to convert text to that style. Other work automatically align pairs of sentences in different styles, either in the representation-level (Prabhumoye al., 2018) or corpus-level (Liu et al., 2022b).
Coarse vs Fine-grained Reward We also di- rectly compare the use of coarse or fine-grained reward tokens the RL stages of STEER. As men- tioned in $3.2, rather than using a product of the style metrics and a single reward token, we can use a vectorized reward function that outputs each of the three style metrics individually and correspond- ingly condition each of these specific metrics.
Results are shown Table 4. Incorporating fine- one-ggained reward improves performance across all dimensions, including V. This shows that con- ditioning on fine- tne-grained rewards can lead to more
Others have attempted TST by prompting LMs (Reif et al., 2021; Suzgun et al., 2022). How-

---

ever, these approaches often rely on strong initial model, either already fine-tuned on TST-related tasks (e.g. paraphrasing), or a large LM capable of few-shot generalization. In contrast, our framework does not assume strong capabilities of the initial model, making it applicable in a realistic setting.
7 Limitations, Ethical Considerations, and Broader Impacts
While STEER demonstrates promising results for arbirraryto-mmany style transfer, there are several limitations. Firstly, in our experiments, we rely heavily on the availability of a corpus containing text from diverse styles to act as source styles for the expert-guided creation of Dy; however, not every corpus will have diverse set of styles to create a Dj from. Instead, in data-limited set- tings, may be required gather source text from other locations, like other corpora, in order to cre- ate candidate stylc-transfer pairs. Secondly, while we tested the generalization of STEER to out-of- domain source style, adaption to new target styles through continual learning requires further investi- gation and experimentation.
RL for NLP Recent work has shown the poten- tial of RL to align with arbitrary natural language objective functions across areas such summa- rization (Paulus et al., 2017). open-ended text gen- eration (Lu et al., 2022), dialogue (Li et al., 2016; Zhou et al., 2017), muestion-answering (Liu et al., 2022a), machine translation (Nguyen et al., 2017; Wu et al., 2016), and dataset generation (Pyatkin et al., 2022; Kim et al., 2023). For unified style transfer, setting where the desired output can be directly correlated to automatic metrics, RL is promising avenue.
Additionally, like many other natural language systems, STEER could unintentionally introduce harmful stereotypes or engage in malicious content generation. Specially the use of fine-grained re- ward signals during online training may be used to reinforce undesired behaviors potentially leading to the generation of biased or unethical outputs. Fur- thermore, bad actors may try to intentionally utilize style transfer systems like STEER to create harm to harass marginalized communities by using toxic output styles. This is a common misuse case generation (MeGuffie and Newhouse, 2020), and an application which we strongly condemn.
Data Generation with LMs LM-generated data have been increasingly used across wide range of tasks, such as commonsense reasoning (West et al., 2022; Zelikman et al., 2022), NLI (Ye et al., 2022) and dialogue generation (Kim et al., 2023). While previous approaches rely on the task-solving capability of LLMs, recent work show that small LMs also generate high-quality datasets with- out supervision (Jung et al., 2023; Brahman al., 2023). Building on top of these, our work pushes further on machine-generated data by incorporating !) inference-time decoding algorithms and 2) tar- geted filtering, yielding an effective pseudo-parallel corpus to initialize offline reinforcement learning.
On the positive side, STEER allows for training memory and cost-efficient training of unified style transfer models using existing corpora. Our method thus beneficial for somewhat reducing the carbon footprint by reducing the reliance on training large language models (LLMs) to achieve desired results (Strubell et al., 2019).
6 Conclusion
We propose STEER, a unified framework to over- come the challenge of limited parallel data in style transfer, by leveraging expert-guided decoding and two-stage reinforcement learning. We focus on a more realistic use case: rewriting text from an arbitrary, unknown style to a desired target style. Through extensive experiments, we demonstrate the effectiveness and robustness of STEER on both style transfer, outperforming
References
Sid Black, Stella Rose Biderman, Eric Hallahan, Quentin G. Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Martin Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Benqi Wang. and Samuel Weinbach. 2022. Gpt-neox- 20b: An open-source autoregressive language model. ArXiv, abs/2204.06745.
in- and competitive baselines. The success of STEER un- derscores the potential of RL algorithms when com- bined with controllable decoding and encourages future algorithmic innovation that fully unleash the power of RL for rerl-world NLP applications.
Faeze Brahman, Chandra Bhagavatula, Valentina Py- atkin, Jena Hwang, Xiang Li, Hirona Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Choi Yejin. 2023. Plasma: Making small language models

---

better procedural knowledge models for (counterfac- tual) planning.
Hongyu Gong, Suma Bhat, Lingfei Wu, Jinjun Xiong. and Wen mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training cor- pus.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Tric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learnrr. ArXiv, abs/2005.14165.
Skyler Hallinan, Alisa Liu, Yejin Choi, and Maarten Sap. 2023. Detoxifying text with MaRCo: Controllable revision with experts and anti-experts. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 228-242, 242, Toronto, Canada. Association for Computational Linguistics.
Geoffrey E. Hinton. 2002. Training Products of Ex- perts by Minimizing Contrastive Divergence. Neural Computation, 14(8):771--180.
Ning Dai. Jianze Liang. Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 5997- 6007, Florence, Italy. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degener- ation. ArXiv, 19t/9999...751.
Zhiting Hu, Zichao Yang. Xiaodan Liang, Ruslan Salakhutdinov, and Fric P. Xing. 2018. Toward con- trolled generation of text.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1875-1888 New Or- leans, Louisiana. Association for Computational Lin- guistics.
Adam K Dean. 2023. Gpt unicorn: A daily exploration of gpt-4's 's image generation capabilities. Accessed: 2023-05-15.
Nan Ding and Radu Soricut. 2017. Cold-start reinforce- ment learning with softmax policy gradient. In NIPS.
Yuntao Du, Xinjun Zhu, Lu Chen, Ziquan Fang, and Yunjun Gao. 2022. Metakg: Meta-leammig on knowl- edge graph for cold-start recommendation. ArXiv, abs/2202.03851.
Luo Ji, Qin Qi, Bingging Han, and Hongxia Yang. 2021. Reinforcement learning to optimize lifetime value in cold-start recommendation. Proceedings of the 30th ACM International Conference on Information & Knowledge Management.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrcj Dušck, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Manl Simon Mille. Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachin Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang. Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metries (GEM 2021), pages 96-120, Online. Association for Computational Linguistics.
Jaehun Jung. Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2023. Impossible distillation: from low-quality model to high-quality dataset & model for summa- rization and paraphrasing.
Hyunwoo Kim. Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization.
Kalpesh Krishna, John Wieting, and Mohit lyyer. 2020. Reformulating unsupervised style transfer as para- phrase generation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762, Online. Asso- ciation for Computational Linguistics.
Jiwei Li. Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep re- inforcement learning for dialogue generation. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1192- 1202, Austin, Texas. Association for Computational Linguistics.

---

Juncen Li, Robin Jia, He He, and Perey Liang. 2018. Delete, retrieve, generate: a simple approach to senti- ment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Sourabrata Mukherjee, Vojtech Hudecek, and Ondrej Dusek. 2023. Polite chatbot: A text style transfer application. In Conference of the European Chapter of the Association for Computational Linguistics.
Khanh Nguyen, Hal Daumé III and Jordan Boyd- Graber. 2017. Reinforcement learning for bandit neural machine translation with simulated human feedback. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 1464-1474, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding- srroding-timeeeooe con- trolled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics und the 1 1th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6691-6706, Online. Association for Computational Linguistics.
Jekaterina Novikova, Ondřej Dušck, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2241-2252, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Jiacheng Liu. Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022a. Rainier: Reinforced knowledge introspec- tor for commonsense question answering. ArXiv, abs/2210.03078.
OpenAL. 2023. Openai api pricing. https:// openai comm wwt.///c///..a....e prriiigiiiinnn.ii Accessed: 2023-05-15.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida. Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.
Ruibo Liu, Chongyang Gao, Chenyan Jia, Guangxuan Xu, and Soroush Vosoughi. 2022b. Non-parallel text style transfer with self self-parallel supervision.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11692.
Ajay Patel, Nicholas Andrews, and Chris Callison- Burch. 2022. Low-resource authorship style transfer with in-context learning. ArXiv, abs/22 2.08986.
Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. ArXiv, abs/2205.13636.
Romain Paulus, Caiming Xiong. and Richard Socher. 2017. A deep reinforced model for abstractive sum- marization. arXiv preprint 1
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhut- dinov, and Alan W Black. 2018 Style transfer through tack-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. A dual reinforcement learning framework for unsupervised text style transfer.
Aman Madaan, Amrith Rajagopal Setlur, Tanmay Parekh, Barnabás Póczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W. Black, and Shrimai Prabhumoye. 2020. Politeness transfer: tag and generate approach. In Annual Meeting of the Association for Computational Linguistics.
Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang. Yejin Choi, and Chandra Bhagavatula. 2022. Clarifydelphi: Reinforced clarifi- cation questions with defeasibility rewards for social and moral situations.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Eric Malmi, Aliaksei Severyn. and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 8671-8688 Online. As- sociation for Computational Linguistics.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa. Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. ArXiv, abs/22 10.01241.
Kris McGuffie and Alex Newhouse. 2020. The radical- ization risks gpt-3 and advanced neural language models. ArXiv, abs/2009.0668

---

Sudha Rao and Joel R. Tetreault. 2018. Dear sir or madam, may introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In North American Chapter of the Association for Computational Linguisties.
John Wieting, Taylor KerggKirkpattrck. Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEUntrainin neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 4344- 4355, Florence, Italy. Association for Computational Linguisties.
Emily Reif. Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison- Burchh and Jason Wei. 2021. A recipe for arbitrary text style transfer with large language models. ArXiv, htss/219.003900.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Transformers: State-of-the art natural language processing. In Conference on Empirical Methods in Natural Language Processing.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Mask and infill: Applying masked language model for sentiment transfer. In Proceed- ings of the Twenty-Eighth International Joint Con- ference on Artificial Intelligence, IJCAI-19, pages 5271-5277. International Joint Conferences on Arti- ficial Intelligence Organization.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv, abs/ 0tn//707.06647
Tianxiao Shen, Tao Lei, Regina Barzilay, and T. Jaakkola. 2017a. Style transfer from non-parallel text cross-alignment. ArXiv, abs/1 170.00955.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint 1r.i:: 081408144.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017b. Style transfer from non-parallel text by cross-alignment.
Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2017. A4nt: Author attribute anonymity by adversarial training of neural machine translation. In USENIX Security Symposium.
Jingjing Xu, Xu Sun, Qi Zeng. Xiaodong Zhang, Xu- ancheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment- coweeeetttt--seetiin to-sentiment translation: A cy- cled reinforcement learning approach. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I; Long Papers), pages 979-988, Mclbourne, Australia. Association for Computational Linguistics.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in nlp. ArXiv, abs/1906.02244
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Ma- heswaran. 2019. "transforming" delete, retrieve, generate approach for controlled text style transfer. ArXiv, hts//998.09365.
Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. ArXiv, abs/ 1903.06353.
Mirac Suzgun, Luke Melas-Kyriazi and Dan Juraf- sky. 2022. Prompt-and-res rer-rka A method for zero- shot and few-shot arbitrary textual style transfer with small language models. In Proceedings of the 2022 Conference on Empirical Methods Natural Lan- guage Processing, pages 2195-2222, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu. Jiangtao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653-1 1663-11669,, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2018. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 1:625-641.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning.
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. 2017. End-to-end offline goal-oriented di- alog policy learning via policy gradient. CoRR. abs/1712.02838.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang. Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- cls to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, pages 4602-4625 Seat- tle. United States. Association for Computational Linguistics.
A Modeling Details
A.1 Out-of-the-Box Modeling
For this work, we use the publicly-availabl HuggingFace Transformers library (Wolf et al.,

---

2019), fine-tune GPT2-models and run in- ference. The HuggingFace code is available at https: /github/tttbb..t transformers. and is licensed under the Apache License 2.0. We also use the OpenAT library (documented at https://platfo: openaì. com/ docs//cc/////////ooooo libraries) to make queries to GPT-3 for some of our style transfer baselines.
Besides these hyperparameters, max sequence and generation length as in $6, and use a no-repeat ngram value of avoid repetition. After creating Di we filter the dataset down to 2.2M rows by doing top-200K sampling for each of the 11 target styles in CDS. Next, we use this data to train the unified model:
the
A.3 Training the Unified Model
A.2 Generation of D
We train our online RL for 30,000 steps with early stopping based on the overall score V. We use 4 RTX-A100 GPUs with 80GB of memory, and this takes about 120 hours train. We batch size of 256 and the Adam optimizer with learning rate of le-5. The full hyperparameters are reported Table 8.
We first train GPT2-large experts using the training data from CDS. See Krishna al., 2020 for more details. For each style expert, we train with the following parameters, shown Table 6.
Hyperparameter
Assignment
model
GPTe-large
numher of gpus
Hyperparameter
Assignment
effective batch size total steps
32
50,000
model
GPT2-large
steps evaluation
1000
number of gpus
learning rate optimizer
AdamW
effective batch size total steps
1024
AdamW initial learning rate
lc-05
30,000
AdamW epsilon
lc-05
steps per evaluation learning optimizer
2500
learning rate schedule
linear with warmup
AdamW
weight decay max sequence length
0.0
AdamW initial learning rate
le-05
50
AdamW epsilon
1c-05
generation length
100
learning rate schedule
linear with warmup
padding sequences
seq length
weight decay max sequence length
50
Table 6: Hyperparameters used to finetune the expert models
max generation length
100
padding sequences
to max seq length
QUARK hypers
We also include an early stopping criteria based on repeatedly increasing loss. Each model was trained on a single Nvidia RTX6K with 24GB of memory for less than 24 hours.
sumple steps KL penalty entropy coef policy temp.
2500
0.025
0.00
1.0
vectorized reward token number of buckets
True
After these experts are created, for every pair of styles in CDS, we generate 10,000 candidate rewrite pairs. Each source text is randomly selected from the train set of CDS style, and used as the input text for all other target styles with variety of hyperparameters. Specifically, we employ the DEXPERTS decoding framework for each these generations. DEXPERTS has one hyperparameter, a, which we vary. We attempt the following hy- perparameters for each generation, shown in Table 7:
Table 8: Hyperparameters used for unified model train- ing, including for QUARK
B Dataset Details
B.1 The Corpus of Diverse Styles (CDS)
CDS is collected and ensembled from online archives and previous academic datasets; we list each style and the train/val/test splits in Table 9. The dataset contains parallel data, only text from each respective style. We CDS as the primary dataset for unified style transfer, 11 distinct styles that we can train and evaluate on. See Krishna et al., 2020 for a full explanation the styles.
Hyperparameter
Tested
a temperature
(0.2,0.4.0.6] [0.7. 1.0. 1.3]
has
Table 7: Hyperparameters tested and used for STEER for data over-generation

---

Style
Train
Validation
Test
C.3.2
Few-shot
AAE
720K
7.3K
7.3K
For few-shot style transfer, we randomly show & examples of the target style sampled from the train set, and prepending them before the O-shot prompt as follows.
Bible
31K
1.7K
1.7K
Coha 1810
205K
5.3K
5.3K
Coha 1890
1.2M
32K
32K
Coha 1990
19M
49K
49K
English Tweet
5.2M
40K
Here are some examples of sentences in the style of [target style]
Joyce
37K
2.0K
2.0K
Lyrics
4.5M
252K
252K
Romantic Poetry
27K
1.5K
Shakespeare
25K
1.3K
1.3K
Switchboard
146K
1.5K
1.5K
lexample 1]
[example 2]
Table 9: Styles and dataset sizes in CDS
(example 3]
Baseline Details
D Human Evaluation
C.1
Style Transfer via Paraphrasing
Since automatic metrics alone have been shown in- sufficient for evaluating text generations (Novikova et al., 2017), we conduct human evaluation. An- notators rate meaning similarity of style transfer pair, and the fluency of the style-transferred text.
We retrieve the diverse paraphraser and the inverse style transfer models from the repository in the original paper; please see Krishna et al., 2020 for more details. At inference, we use greedy decoding as this led to the best results in the original paper.
For fluency, annotators choose between: O for not fluent, 1 for somewhat fluent, and for fluent. For meaning similarity, annotators choose between: O for not similar, 1 for somewhat similar, and 2 for similar. We discard annotations where all three annotators disagree on label for either fluency or similarity, resulting in a final human evaluation labeled size of 310 (from an initial size of 330).
C.2 Prompt-and-Reramt
Prompt-and-Rerant has two prompting strategies that do not require knowledge of the source style and are therefore suitable for arbitrary style trans- fer. The two strategies vanilla prompting and contrastive prompting. See Reif et al., 2021 for full details on the exact prompts.
To reduce labor cost, we only run our human evaluations on the top three methods from Table 1, meaning we exclude P-A-R. R. In addition, following previous work, we do not run human evaluation target style strength. Further details are explained in Appendix D.I.
We test out both prompts with small subset data, and find that contrastive prompting works much better, so we use this going forward. We also try generating
3 samples and &o = 5 samples per input, and find that lo = 3 works the best. Following the original paper. we use nucleus sampling (Holtzman et al., 2019) with 0.9 and temperature of 1.0. Finally, for fair comparison with STEER.
D.1 Style Identification Task Difficulty
GPT-2 Large
The target styles in the CDS dataset are extremely complex. Previous work from Krishna et al., 2020 mention that this is too challenging of task, even for experienced annotators.
C.3 GPT-3
We prompt GPT-3 using nucleus sampling (Holtz- man et al., 2019) with = 0.9 and a temperature 1.0. We include further details on zero-shot and few-shot prompting below.
We verify the difficulty of the text style iden- tification task reported Krishna et al., 2020 by performing an additional human evaluation. From the CDS test set, we randomly sample 10 exam- ples from each of the 11 styles (1 (110 total examples with ground truth styles). Next, we use the same three annotators from previous human evalua- tion (NLP experts), and provide them with natural language description of each of the 11 styles and 20 random examples from the train set of each familiarize them with text from different styles. We ask them to assign a style label to each of the 110
C.3.1 Zero-shot
We use the following prompt setup for zero-shot style transfer:
Rewite the following sentence into the style of [target style]
Original Sentence: [original sentence]
Rewritten Sentence:

---

examples, given their knowledge of the styles, and calculate their accuracy and agreement. On aver- age, the annotators only have a 40.0% classification accuracy with an inter-annotator agreement of 0.39 (Fleiss' kappa). In contrast, on the same samples (unseen by the classifier), our classifier obtains a 84.5% classification accuracy. These results vali- date the difficulty of the task and suggest that an automatic classifier is more suited for this task.
similarity, use the embedding based SIM model of Wieting et al., 2019 as used in Krishna et al., 2020. Finally, for the style classifier model, given limited data quantity, we train another RoBERTa-Large classifier with the same CDS data but with different seed. As before, we compute the aggregate metric V by taking the product of the three automatic metrics for each style transfer pair in the corpus, and report the average V value.
Our results using the alternative metrics re- run evaluation on both in-domain style transfer and out-of-domain style transfer using the CDS-traince STEER mode are shown in Table and Table 11 respectively. Overall, this corroborates our main findings by showing that our relative results are largely unchanged: on the in-domain styles, STEER beats all baselines, including impressive gains target style strength as well as improved fluency and meaning similarity. On the out-of-domain lask, STEER continues to excel, once again beating all other baselines other than GPT-3 on Shakespeare.
E
The Cold Start Problem in RL
Reinforcement learning often involves optimizing policy model towards an optimal distribution that muximizes some expected reward. This paradigm works well out-of-the-bol for a variety of tasks in NLP, such as model detoxification and sentiment control (Lu et al., 2022) where the output distribu- tion of the initial policy already aligns, to a reason- able degree, with the optimal reward distribution However, in cold-start, reinforeement-leamiigg setting, the initial policy output distribution dras- tically different than the optimal reward distribu- tion; this may be the case when the reward linked to a specific task outside the capabilities of the orig- inal policy.
G Full Experimental Results
We detail the full experimental results for the main experiments in this section, including all style eval- uation metrics.
Adjusting to cold-start has been mostly explored in the context of recommender systems, where it is difficult to determine user-preferences without any initial data (Ding and Soricut, 2017; Ji al., 2021; Du et al., 2022), but has been sparsely pur- sued in reinforcement learning for NLP. Ding and Soricut (2017) introduce softmax policy gradients for cold-start reinforeement-lerriing but the ap- proach is limited to only one class of reinforcement learning algorithms (policy-gradien) approaches) and includes mathematical assumptions not widely applicable to various NLP applications.
G.1 Main Experiments
We include the full results for the main experiment from Table I, testing out style transfer on the CDS dataset to each target style from all other source styles. Table 12 has the results for STEER, Table 13 has the results for STRAP, Table 14 has the results for A-A-R, and Tables 15-18 have the results for GPT-3 0-shot, I-shot, 5-shot, and 10-shot.
F Alternative Evaluation Metrics
To ensure that the model improvement from STEER is meaningful (i.e., to make sure our model is not reward hacking), we use a set alternative metrics for target style strength, meaning similarity, and fluency, and re-run evaluation on all results from Tables and 2. These are metrics unused during training time for STEER.
For the fluency model, we use different binary CoLA classifier (attps:/hhuggingtt))
co / seeextattacktroberr--bbssse
and again use the raw probability score the linguistically acceptable class. To assess meaning

---

GPT-2 Large
GPT-3 (text Savincii-000
Target Style
STEER
STRAP
P-A-R
k 0
h
k
10
AAE Twitter
38.2
7.8
1.6
18.1
9.4
20.7
17.6
Bible
34.4
22
2.0
4.0
12.3
14.4
13.6
1810-1820s
23.1
9.2
1.0
12.8
11.5
12.3
12.6
1890- 1900s
34.3
13.6
2.2
8.0
8.0
10.3
10.3
1990-200s
39.0
16.1
1.7
8.7
13.1
15.6
15.8
English Twitter
37.7
8.7
2.5
28.2
16.6
24.2
19.8
James Joyce
17.6
10.7
1.9
1.9
1.5
1.5
2.2
Song Lyries
29.3
20.1
2.9
10.3
13.6
9.4
11.7
Romantic Poetry
16.8
14.8
0.9
0.5
2.6
4.2
3.9
Shakespeare
11.4
8.0
0.8
5.9
7.3
6.6
6.9
Switchboard
41.4
15.2
0.5
0.0
0.2
3.0
6.9
Overall
29.4
13.3
1.6
8.9
8.7
11.1
11.0
Table 10: Comparison of 1 1--way style transfer the CDS dataset measured by aggregate score using the alternative evaluation metrics (replication of Table 1). Bold and underline denote the highest and the second-highest score respectively each row.
GPT2-Large
GPT-3 (text davincii 003)
STEER
STRAP
P-A-R
kt
O
5
10
Target Style
Inf.
For.
Inf.
For.
Inf.
For.
Inf.
For.
Inf.
For.
Inf.
For.
Inf.
For.
AAE Twitter
38.7
43.8
18.7
14.1
14.5
5.7
25.7
23.5
16.7
15
25.5
24.8
23.8
22.9
Bihle
29.6
32.4
19.4
20.1
0.5
3.9
3.9
10.8
11.3
13.6
15.5
15.3
15.4
1810-1820s
19.6
22.3
1.5
9.9
12.6
11.2
14.1
12.6
16.7
12.1
15.7
1890-1900s
32.3
10.9
14.2
8.5
10.2
11.3
12.6
12.4
11.4
11.2
1990-200s
44.9
51.9
22.1
31.6
18.9
26.3
30.6
30.1
31.0
English Twitter
51.2
20.6
22.1
23.6
32.3
20.3
24.7
James Joyce
19.0
21.5
11.7
13.2
1.5
3.5
2.0
2.6
2.3
3.0
3.1
Song Lyrics
36.4
36.1
21.3
23.6
4.3
15.3
10.0
19.0
14.6
15.7
13.3
19.2
16.1
Romantic Poctry
11.2
11.4
10.0
11.2
0.3
1.4
0.7
3.7
3.5
4.2
2.8
Shakespeare
10.1
9.7
9.9
9.4
0.3
1.6
10.2
12.4
11.3
12.5
11.1
12.5
11.5
12.8
Switchboard
42.5
47.3
22.6
27.7
2.1
1.7
0.1
0.0
0.3
0.0
4.8
6.6
8.1
12.4
Overall
29.6
32.7
15.7
17.9
5.2
5.9
10.9
11.6
12.2
12.4
14.4
15.1
14.4
15.3
Table 11: Comparison of style transfer to each of the 11 styles in the CDS dataset measured by aggregate score V using the alternative evaluation metrics from two out-of domain styles from the GYAFC corpus (replication of Table 2). For. and Inf. denote the formal and informal styles respectively. Bold and underline denote the highest and the second-highest score respectively in each row.
V
TSS
F
MS
V
TSS
MS
AAE Twitter
42.6
68.2
87.0
72.5
AAE Twitter
7.4
21.8
65.7
66.2
Bible
44.0
80.9
85.9
63.9
Bible
26.9
70.5
79.4
51.7
1810-1820s
30.2
49.8
81.9
74.7
1810-1820s
11.1
22.5
78.6
64.9
1890-19000
35,9
56.6
85.4
74.6
1890-1900s
12.3
22.6
82.6
65.0
1990-200s
42.3
63.0
83.9
77.8
1990-2000s
16.6
29.1
82.8
65.0
English Twitter
41.2
62.6
87.7
74.5
English Twitter
8.0
36.9
76.8
49.8
James Joyce
20.4
34.1
81.8
76.7
James Joyce
11.8
26.4
78.8
66.7
Song Lyrics
33.3
51.6
87.2
74.7
Song Lyries
20.2
39.4
80.2
67.3
Romantic Poetry
20.4
35.3
80.5
74.7
Romantic Poetry
15.7
46.5
64.5
62.1
Shakespeare
13.6
24.7
83.7
72.7
Shakespeare
9.1
27.8
68.0
62.2
Switchboard
52.9
92.0
87.0
66.1
Switchboard
21.1
52.0
69.3
65.2
Overall
34.3
56.3
84.7
73.0
Overall
14.6
36.0
75.2
62.4
Table 12: Full results CDS test with STEER
Table 13: Full results CDS test set with the STRAP

---

V
TSS
MS
AAE Twitter
3,8
10.3
76.1
68.5
Bible
6.6
24
79.1
67.3
1810-1820s
3.5
8.7
74.4
78.9
1890-1900s
4.4
10
76
77.7
TSS
MS
1990-2000s
4.3
9.1
75.4
79.2
AAE Twitter
25.4
79.1
58.4
English Twitter
5.5
12.5
74.1
82.1
Bible
20.2
43.6
78.8
65.6
James Joyce
5.4
13.6
76.9
75.3
1810-1820s
17.4
35.6
80.4
62.4
Song Lyries
7.7
23.2
78.2
65.3
1890-1900s
10.4
17.2
84.6
66.8
Romantic Poetry
2.8
7.6
77.1
76.1
1990-2000s
17.5
26.9
89.1
71
Shakespeare
2.5
8.2
77
72
English Twitter
32
55.9
80.2
68.4
Switchboard
1.7
3.9
75.1
84.5
James Joyce
1.6
10.9
48.6
63.2
Overall
4.4
11.9
76.3
75.2
Song Lyries
11.2
25.8
74.7
66
Romantic Poetry
6.2
28.5
44.6
57.4
Table 14: Full results on CDS test set with PAR
Shakespeare
9.7
28
62.8
68.4
Switchboard
5.3
10.7
85.4
71.2
Overall
14.3
32.9
71.6
65.4
V
TSS
F
MS
AAE Twitter
23.2
58.4
63.1
65.7
Table 17: Full results on CDS test set GPT-3 5-shot.
Bible
5.2
10.4
85
70.4
1810-1820s
14.7
27.6
79.4
67
1890-1900s
8.6
15
83.1
65
1990-2000s
7.9
12.6
83.8
68.8
English Twitter
35
58.3
82
70
James Joyce
3.4
7.4
77.1
66
Song Lyrics
12.2
25.4
73.7
72.2
Romantic Poetry
1.1
3.5
61.7
61.5
Shakespeare
9.6
27.7
64.3
68.6
Switchboard
0.1
92.2
74.5
Overall
11
22.4
76.9
68.2
V
TSS
MS
Table 15: Full results CDS test set for GPT-3 (-shot.
AAE Twitter
22.7
65.4
57.9
64.9
Bible
21
43.8
81.4
65.7
1810-1820s
17
33.1
81
66.2
V
TSS
F
MS
1890-1900s
10.1
16.9
83.1
68.8
1990-2000s
17.2
25.7
88.2
72.1
AAE Twitter
11.2
30.9
67.3
64.8
English Twitter
29.5
51.4
77.8
69.9
Bible
16
40.2
79
59.5
1810-1822
15.9
29.4
82
65.3
James Joyce
2.6
20.9
33.9
62.4
Song Lyrics
13.2
30.8
69.5
69.5
1890-1900s
9.1
16
82.4
63.3
Romantic Poetry
4.9
23.2
43.9
58.9
1990-2000s
13
25.2
86.2
65.3
Shakespeare
9.7
28.7
60.8
69.5
English Twitter
23.6
47
71.3
68
Switchboard
13.7
24.8
82.2
73.5
James Joyce
1.3
11
46.1
63.2
Overall
14.7
33.2
69.1
67.4
Song Lyries
15.4
40.1
71.8
62.4
Romantic Poetry
3.4
15.3
51.7
59.1
Table 18: Full results CDS test set for GPT-3 10-shot.
Shakespeare
10
29.5
61.9
69
Switchboard
0.3
0.6
90.3
75.4
Overall
10.8
25.9
71.8
65
Table 16:
Full results CDS test set for GPT-3 1-shot.


Reviews:
[{"review_id": "DFup2PaGK3", "reviewer": "Reviewer_3tNE", "summary": "", "questions": "1. Can you address the questions posed above related to why the human evaluations skew more towards gpt3.\n2. The question related to if you think the augmented score V is a true indicator of style transfer quality.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and specific feedback! We were happy to read that you appreciated our main points: that our two-stage process to improve a mediocre initial policy is “novel” and could be a “helpful signal to future work in conditional generation”, that our strong experimental results have “prove[d] the model's effectiveness”, and that our work has “merits both conceptually and through experimental results”.\n\nPlease see our response to your valuable questions and feedback below. We would love to address additional questions during the discussion period if anything is unclear.\n\n---\n\n**It is hard to tell if STEER performs better overall than GPT3 since on human evaluation, STEER outperforms GPT3 on style transfer strength, but not on fluency and meaning similarity**\n\nThanks for bringing this up! Prior work demonstrates a **noticeable tradeoff between style transfer accuracy and meaning preservation**; this is often shown **experimentally** (Suzgun et al., 2022; Wu et al., 2019; Malmi et al., 2020; Wu et al., 2019; Xu et al., 2019; Li et al., 2018) and also **explicitly discussed** (Li et al., 2018; Wu et al., 2019; Xu et al., 2019; Hallinan et al., 2023). Intuitively, when transferring from one style to another, some amount of semantic changes is unavoidable; as a simple example, meaning similarity will be maximized when the input is naively copied.\n\nDue to this tradeoff between style transfer accuracy and meaning preservation, it is often impossible to completely optimize for both metrics, and different methods inherently have different advantages. In this case, the **scale of the difference of the metrics** in Figure 4 between STEER and GPT3 is important: STEER has an impressive **44% higher target style strength** while sacrificing **24% decrease in meaning similarity** and a **negligible 1.9% decrease in fluency**. Overall, we think this is a reasonable tradeoff, as STEER sacrifices much less fluency and meaning preservation for much more style transfer accuracy.\n\nFinally, one advantage of STEER is we can **explicitly control the tradeoff** between different style transfer objectives (style transfer accuracy, meaning similarity, fluency) by altering the weights assigned to the metrics when computing the aggregate reward in both the **data generation** and **reinforcement learning** step. This allows one to promote specific qualities in the outputs; for example, we can encourage preserving meaning more by assigning a higher weight to meaning similarity relative to target style strength. We will highlight this advantage more explicitly in our next revision.\n\n---\n\n**Though STEER obtains a high aggregate V score, it may be greedily optimizing for target style strength at the sacrifice of fluency and meaning similarity**\n\nThat’s indeed a potential concern, and one which we considered and were able to avoid! We list the breakdown of automatic metrics (target style strength, meaning similarity, fluency, and the aggregate metric $\\mathcal{V}$) for all methods in Table 1 in the **Appendix, Tables 9-15**. \n\nComparing Table 9, which has the full metrics results from STEER on the CDS test set, to Tables 10-15 (results of other baselines), shows that our target style strength does indeed significantly improve over baselines while also showing that our meaning similarity and fluency improve almost all of the time. This demonstrates that **STEER does not optimize only for target style strength, and in fact improves all three metrics jointly**, resulting in better overall style transfers. In the rare cases in which either fluency or meaning similarity are lower than baseline methods, they are still competitive, while having a much higher target style strength. As an example, for the Bible style, STEER gets a target style strength, fluency, and meaning similarity of 80.9, 85.9, 63.9 respectively; while GPT3 0-shot for the same style has a higher meaning similarity of 70.4, the target style strength is only 10.4 (the fluency is similar at 85.0).\n\nOverall, these results demonstrate that in addition to strong target style strength, our high aggregate metric can be attributed to better or competitive fluency and meaning similarity (ie, we're not optimizing only the target style strength).\n\n---\n\n**Prompting approaches with unsupervised LLMs may be more generalizable because STEER requires a two-phase training** \n\nWhile zero-shot and few-shot methods are promising (Brown et al., 2020; Suzgun et al., 2022), we would like to emphasize that our method has significant merits (1) in making **smaller, more accessible models** available for wider adaptation and broader community usage, and (2) **achieving better or comparable results** than much larger and not widely accessible models, both of which are increasingly important considerations in NLP research (Strubell et al., 2019; Schwartz, 2019; Schick et al., 2020). Also, we would like to clarify that STEER only requires **a one-time fine-tuning step**. \n\nMore specifically, STEER outperforms prompting baselines (Table 1 and 2), on **all in-domain styles**, and on all **out-of-domain** styles, except for GPT3 on Shakespeare, by a significant margin. This is especially impressive as the STEER model is based on GPT2-Large, a 774M parameter model, while the Instruct-GPT3 baseline has 175B parameters, a 226 times disparity in model size. While prompting for style transfer is also possible with smaller models (Suzgun et al., 2022), we find the performance of prompting methods with smaller models (using the method from Suzgun et al, 2022 with GPT2-Large, results shown in Table 1 and 2) to be **extremely poor, especially transferring to lesser known/more obscure styles** such as the switchboard corpus.\n\nAdditionally, STEER only requires **a one-time fine-tuning step** which equips it with style transfer capability from any **arbitrary out-of-domain** source style to several target styles (see Table 2 for specific results). This further enhances the convenience of STEER as a single unified model that can be used for unseen inputs.\n\n---\n\n**Can you address the questions posed above related to why the human evaluations skew more towards gpt3.**\n\nHuman evaluations in meaning similarity may skew toward **GPT3** because it is **known to perform well on paraphrasing and summarization** as shown in multiple literature (Brown et al., 2020; Goyal et al., 2023; Zhang et al., 2023). However, we demonstrate that it **lacks the skill of successfully transferring to styles**. As shown above, though GPT3 outperforms STEER on human evaluation meaning similarity and fluency, STEER performs much better on target style strength.\n\n---\n\n**Is the augmented score V a true indicator of style transfer quality?**\n\nWe follow prior work in style transfer, which suggests aggregating the style transfer accuracy, meaning similarity, and fluency metrics (Krishna et al., 2020; Xu et al., 2018; Pang and Gimpel, 2018; Li et al., 2018). This ensures that all three metrics are optimized and enables a singular comparison between different methods (Krishna et al., 2020). In addition, we also report individual metrics in the Appendix (as discussed above) and conduct human evaluation (lines 413-444) to assess the true quality of the style transfers. \n\n---\n\n**One cannot say for sure that concepts from this work will be re-used in this setting, particularly in style transfer, as they are heavily engineered.**\n\nAlthough our method requires engineering to initially build the model, we think the concepts will be reused both for the taks of arbitrary style transfer and in other potential settings.\n\nAs mentioned above, STEER requires only an **initial** data generation and model training to create a model that can transfer from **arbitrary source styles** to different target styles (see Table 2 for results). Though there is an initial engineering cost to create this model, this is a **one-time** cost and leads to **a capable and robust** downstream model.\n\nAdditionally, the concepts from our method are general and can be potentially applied to other tasks. Specifically, in reinforcement learning, using offline data exploration to supplement an insufficient initial policy could be useful in scenarios where there is a sparse reward signal or where there is a lack of initial, high-quality data, such as in style transfer for styles lacking parallel data.\n\n---\n\n**References**\n\nBrown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T.J., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165.\n\nGoyal, T., Li, J.J., & Durrett, G. (2022). News Summarization and Evaluation in the Era of GPT-3. ArXiv, abs/2209.12356.\n\nHallinan, S., Liu, A., Choi, Y., & Sap, M. (2022). Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts. Annual Meeting of the Association for Computational Linguistics.\n\nKrishna, K., Wieting, J., & Iyyer, M. (2020). Reformulating Unsupervised Style Transfer as Paraphrase Generation. ArXiv, abs/2010.05700.\n\nLi, J., Jia, R., He, H., & Liang, P. (2018). Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer. North American Chapter of the Association for Computational Linguistics.\n\nMalmi, E., Severyn, A., & Rothe, S. (2020). Unsupervised Text Style Transfer with Masked Language Models. EMNLP.\nPang, R., & Gimpel, K. (2018). Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer. Conference on Empirical Methods in Natural Language Processing.\n\nSchick, T., & Schütze, H. (2020). It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. ArXiv, abs/2009.07118.\n\nSchwartz, R., Dodge, J., Smith, N., & Etzioni, O. (2019). Green AI.\n\nStrubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. ArXiv, abs/1906.02243.\n\nSuzgun, M., Melas-Kyriazi, L., & Jurafsky, D. (2022). Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models. ArXiv, abs/2205.11503.\n\nWu, X., Zhang, T., Zang, L., Han, J., & Hu, S. (2019). Mask and Infill: Applying Masked Language Model for Sentiment Transfer. IJCAI.\n\nXu, J., Sun, X., Zeng, Q., Ren, X., Zhang, X., Wang, H., & Li, W. (2018). Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach. Annual Meeting of the Association for Computational Linguistics.\n\nXu, R., Ge, T., & Wei, F. (2019). Formality Style Transfer with Hybrid Textual Annotations. ArXiv, abs/1903.06353.\n\nZhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. (2023). Benchmarking Large Language Models for News Summarization. ArXiv, abs/2301.13848.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This model shows the efficacy of RL to train and finetune conditional generative models. Though it has been explored before, the combination of creating a supervised dataset to train the initial policy and then subsequently improving the policy during the online training seems novel. This might be a helpful signal to future work in conditional generation (not specifically arbitrary style transfer).\n2. Considering a many-to-many style transfer problem seems like the right direction to pursue rather than similar work that focuses on a one-to-one task. The authors have used sufficient datasets to prove the model's effectiveness for this task.\n3. The experiments are well-defined and the explanations are sufficient to get a clear idea of what the model does and how it does it without sacrificing brevity and ease of understanding. The structure of the people can be improved a little bit, however (e.g. analysis of D_f in 4.6 can be added earlier).\n4. Overall this word does have merits both conceptually and through experimental results. However, one cannot say for sure that concepts from this work will be re-used in this setting, particularly in style transfer, as they are heavily engineered.", "reasons_to_reject": "1. The major issue with this work is the claim that it performs better than strong baselines such as gpt3. For example, in Figure 4 showing the results of human evaluation, we can see that gpt3 performs better than steer in meaning similarity and fluency. Whereas steer outperforms gpt3 in only style transfer strength. This indicates that steer might suffer from the inherent flaw that most models do generally in this problem i.e. gaining style transfer strength while compromising on content similarity. It is quite hard to gauge if steer indeed does perform better than gpt3 given these conflicting results.\n2. One might argue that your augmented score definition V consisting of a linear product of fluency, content sim and style transfer metrics offers a singular score to judge a model. Therefore according to Table 2, steer clearly seems like the best-performing model. However, an adversarial situation can be thought of where a model greedily optimises and outperforms in style transfer acc. by a large degree compared to the baselines but sacrifices on content sim., enough to push the augmented score V as high as possible (this is a well-studied issue in the style transfer task where it is still an open problem).\n3. Apart from the two above points, the other consideration is that this model inherently needs two-phase training (granted it performs well in unseen future styles). There are more intermediate steps such as dataset filtering that are also needed. Prompt and LLMs do not require any supervision and therefore are even more generalised approaches for style transfer, although they are indeed more computationally expensive. \n4. To summarise, it is not convincing that steer is the best model due to the conflects between the automated metrics and human evaluations.\n", "questions_to_authors": "1. Can you address the questions posed above related to why the human evaluations skew more towards gpt3.\n2. The question related to if you think the augmented score V is a true indicator of style transfer quality.", "paper_topic_and_main_contributions": "This paper introduces a model that is capable of transferring multiple source styles into multiple target styles by a) first creating a pseudo parallel dataset using the combination of a paraphraser model which can be steered or guided to produce the target style using signals from the product of experts mechanism and b) uses this new dataset D_f as a supervised training problem to train an initial RL policy. Finally c) in online training, this initial policy is used for generation to augment the dataset while also updating the policy alternatively to improve the model's performance in a typical RL fashion. Finally, they show that it also performs well when unseen source-type sentences are introduced.\n\nThe authors claim that this model beats strong baselines significantly through their experiments primarily by arguing that their model on average produces the highest augmented score V across all datasets compared to even larger models like davinci 003. Some side studies are done to show that a vectorised signal consisting of fluency, STS and content sim. work better during RL training.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None come to mind", "Typos_Grammar_Style_and_Presentation_Improvements": "None come to mind"}, "extra_text": {}}, {"review_id": "CbPcNVfOu4", "reviewer": "Reviewer_kVMX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points, including our strong experimental results demonstrating a “[large] improvement” over baselines, and that our method helps overcome the common limitation in style transfer of the lack of parallel data, which is “pushing style transfer closer to real-world applications”.\n\nPlease see our response to your questions and concerns below. We would love to address additional questions during the discussion period if anything is unclear.\n\n---\n**This work still does not realize transferring from arbitrary unknown styles since data generation needs stylistic data as input. Ideally, the data is supposed to be constructed unsupervisedly. The performance on out-of-domain styles is questionable.**\n\nWe would like to clarify our approach - while our method does **initially** require data where the style is known to create an initial data pool (expert-guided data generation), the resulting model trained with this diverse dataset can transfer text both from styles present in the initial data pool, and from **out-of-domain, unseen** styles.  In other words, the resulting model can successfully transfer from any **arbitrary** style. In Table 2 (also lines 403-412), we show performance of the STEER model trained on the CDS corpus transferring from two out-of-domain (OOD) styles. The two OOD styles are unseen by the model and are fed into the CDS-trained model **without additional data generation or model training**. STEER outperforms all baselines transferring to each of the 11 styles in the CDS dataset from the two OOD styles, except to GPT3 on Shakespeare. We will make our out-of-domain experiments and results more clear in the next version of the paper.\n\n---\n\n**Unclear why authors do not perform human evaluation for target style strength**\n\nAs discussed in **Footnote 6 & Appendix E, line 1047**, we follow previous work (Krishna et al, 2020; authored the CDS dataset) in not performing human evaluation on target style transfer accuracy, as this is a challenging task especially for untrained annotators who are unfamiliar with the set of target styles.\n\nWe additionally verify the difficulty of this task by performing a human evaluation. From the CDS test set, we randomly sample 10 examples from each of the 11 styles (110 total examples with ground truth styles). Next, we use the same three annotators from our previous human evaluation (NLP experts), and provide them with a natural language description of each of the 11 styles and 20 random examples from the train set of each to familiarize them with text from different styles. We ask them to assign a style label to each of the 110 examples, given their knowledge of the styles, and calculate their accuracy and agreement. On average, the **annotators only have a 40.0% classification accuracy** with an inter-annotator agreement of 0.39 (Fleiss’ kappa). In contrast, on the same samples (unseen by the classifier), our **classifier obtains a 84.5% classification accuracy**. These results validate the difficulty of the task and suggest that an automatic classifier is more suited for this task.\n\n---\n\n**The trend of NLP is towards zero-shot and few-shot. The approaches requiring heavy finetuning are less convenient in practice.**\n\nWhile zero-shot and few-shot methods are promising (Brown et al., 2020; Suzgun et al., 2022), we would like to emphasize that our method has significant merits (1) in making **smaller, more accessible models** available for wider adaptation and broader community usage, and (2) **achieving better or comparable results** than much larger and not widely accessible models, both of which are increasingly important considerations in NLP research (Strubell et al., 2019; Schwartz, 2019; Schick et al., 2020). Also, we would like to clarify that STEER only requires a one-time fine-tuning step. \n\nMore specifically, STEER outperforms prompting baselines (Table 1 and 2), on **all in-domain styles**, and on all **out-of-domain** styles, except for GPT3 on Shakespeare, by a significant margin. This is especially impressive as the STEER model is based on GPT2-Large, a 774M parameter model, while the Instruct-GPT3 baseline has 175B parameters, a 226 times disparity in model size. While prompting for style transfer is also possible with smaller models (Suzgun et al., 2022), we find the performance of prompting methods with smaller models (using the P-A-R method from Suzgun et al., 2022 with GPT2-Large, results shown in Table 1 and 2) to be **extremely poor, especially transferring to lesser known/more obscure styles** such as the switchboard corpus.\n\nAdditionally, STEER only requires a **one-time fine-tuning step** which equips it with style transfer capability from any **arbitrary out-of-domain** source style to several target styles (see Table 2 for specific results). This further enhances the convenience of STEER as a single unified model that can be used for unseen inputs.\n\n---\n\n**References**\n\nBrown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T.J., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165.\n\nKrishna, K., Wieting, J., & Iyyer, M. (2020). Reformulating Unsupervised Style Transfer as Paraphrase Generation. ArXiv, abs/2010.05700.\n\nSchick, T., & Schütze, H. (2020). It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. ArXiv, abs/2009.07118.\n\nSchwartz, R., Dodge, J., Smith, N., & Etzioni, O. (2019). Green AI.\n\nStrubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. ArXiv, abs/1906.02243.\n\nSuzgun, M., Melas-Kyriazi, L., & Jurafsky, D. (2022). Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models. ArXiv, abs/2205.11503.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work proposes a new way to overcome the lack of parallel data, pushing style transfer closer to real-world applications.\n* The improvement is large compared with previous methods.", "reasons_to_reject": "* This work still does not realize transferring from *arbitrary unknown* styles since data generation needs stylistic data as input. Ideally, the data is supposed to be constructed unsupervisedly. The performance on out-of-domain styles is questionable.\n* I don’t understand why in the human evaluation the authors use an automatic metric TSS rather than a human metric to evaluate the style control. This weakens the convincingness of human evaluation.\n* The trend of NLP is towards zero-shot and few-shot. The approaches requiring heavy finetuning are less convenient in practice.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel approach STEER for text style transfer allowing transfer from a wider range of source styles. Their contribution focuses on an expert-guided pseudo-parallel data generation and a reinforcement learning training strategy adapted from Quark. They use language models finetined on each style dataset as experts and anti-experts to modulate the output logits of the data generation LLM following product of experts. These generated data serve as the initial data pool for reinforcement learning after top-K filtration. The training is alternatively updating the reward model and data pool. Experimental results on 3 datasets with at most 11 styles demonstrate the proposed STEER has significant advantage over previous methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xPrQVeMj0p", "reviewer": "Reviewer_kv8D", "summary": "", "questions": "1. Do I understand correctly that the equation in line 226 is used as the evaluation metric?\n2. Any comments on reasons to reject 1 and 2? I will raise my score if they are addressed properly.\n3. What metrics do the subfigures 2~4 in Figure 3 correspond to?\n4. How is the value $\\mathcal{V}$ in Table 4 calculated? I can not obtain it using the equation in line 226, i.e., $0.428\\times0.626\\times0.790 \\ne 0.1994$.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and specific feedback! We were happy to read that you appreciated our main points: that STEER achieves “good” performance on a diverse dataset which makes our results more “solid”, and that our method results in a model outperforming the much larger Instruct GPT-3 model, showing that we “[do] not rely on large language models”. \n\nPlease see our response below. We would love to address additional questions during the discussion period if anything is unclear.\n\n---\n\n**STEER is optimized directly against the evaluation metric but the baselines are not, which may be unfair**\n\nDefining a task-specific reward function is a common practice in the Reinforcement Learning (RL) field, and a standard procedure adopted across various RL literature (Lu et al., 2022; Ramamurthy et al., 2022; Wu et al., 2023). To this end, we use reinforcement learning to closely, directly align our style transfer model with the desired properties (measured by the evaluation metrics) by using a fine-grained reward. In fact, the baseline we included, Suzgun et al., 2022, devises the same strategy: **reranking** and choosing the best style transfer from multiple candidates based on the evaluation metrics is an important part of the pipeline. \n\nWe additionally verify the quality of generations through a human evaluation on meaning similarity and fluency of style transfer outputs (line 413, and Figure 4). We do not run human evaluation on target style strength following Krishna et al., 2020, given the difficulty of the task with so many target styles. Human evaluation results corroborate our automatic metrics, showing that STEER outperforms baselines on the partially human aggregate metric (the product of the human evaluation fluency and meaning similarity with the automatic evaluation of target style strength).\n\nFinally, we re-run the numbers from Table 1 using alternative fluency, meaning similarity, and target style strength models to give a fair comparison to other baselines, and ensuring that STEER is not optimized on the evaluation metrics. For the fluency model, we use a different binary CoLA classifier (https://huggingface.co/textattack/roberta-base-CoLA), and again use the raw probability score of the linguistically acceptable class. To assess meaning similarity, we use the embedding-based SIM model of Wieting et al., 2018 as used in Krishna et al, 2020. Finally, for the style classifier model, given limited data quantity, we train another RoBERTa-Large classifier with the same CDS data but with a different seed. **We recompute the three metrics for both in-domain and out-of-domain style transfer on CDS** (replicating Tables 1 and 2) and the aggregate metric $\\mathcal{V}$, finding that **our relative results are largely unchanged**: on the in-domain styles, STEER beats all baselines, including impressive gains on target style strength as well as improved fluency and meaning similarity. On the out-of-domain task, STEER continues to excel, once again beating all other baselines other than GPT-3 on Shakespeare. The absolute metrics for fluency and $\\mathcal{V}$ are slightly lower across all methods due to the alternate fluency classifier. We will include these results in the next version of the paper.\n\n---\n\n**The evaluation metric used in this work is different from highly related previous works (Suzgun et al., 2022; Krishna et al., 2020), but the reason is not explained.**\n\nOur evaluation is in fact **very similar to previous work** from Krishna et al., 2020 and Suzgun et al., 2022 with slight modifications. We explain our differences below:\n\nKrishna et al., 2020 assesses overall style transfer quality using a fluency model (RoBERTa trained on the CoLA corpus (Warstadt et al, 2018), a style transfer accuracy model (11-way RoBERTa classifier), and a semantic similarity model (subword embedding-based SIM model of Wieting et al., 2018). For each style transfer input/output, they also take the product of the three three metrics to create an **aggregate** metric (sentence-level aggregation). Importantly, they **binarize** the style transfer accuracy and fluency based on a threshold of 0.5 before this product.\n\nWe follow this evaluation pipeline closely, using the same fluency and style accuracy model, with two modifications designed to enable more fine-grained quality evaluation: 1) Rather than binarizing style transfer accuracy and fluency when calculating the sentence-level aggregate metric V, we use the **raw probability scores** from the respective style classifier and fluency models, a float between 0-1 each. As in Reif et al., 2021, this allows us to represent the **strength** of the target style (as well as fluency), which gives us more fine-grained evaluation and better distinguishes the quality of similar style transfer pairs. 2) Rather than using the SIM model (Wieting et al., 2019), we use Sentence-BERT (Reimers and Gurevych, 2019) since it is more widely-used (5k vs 100 citations), is tested on more benchmarks, and is easier to implement. \n\n**Overall, we closely follow the evaluation from Krishna et al., 2020**. Our minor changes are designed to improve evaluation and are backed by prior work. We will clarify this and include these justifications in the next version of the paper.\n\n---\n\n**Question: Do I understand correctly that the equation in line 226 is used as the evaluation metric?**\n\nCorrect. We use the aggregate metric of target style strength, meaning similarity, and fluency to evaluate the quality of our models. We also report a full breakdown of the three individual metrics in the **Appendix, Tables 9-15** corresponding to all the methods to Table 1, which shows that our method, STEER, increases the performance of **all** metrics over baselines almost all of the time. In addition, we conduct a human evaluation to ensure the quality of the style transfer outputs (line 413, and Figure 4).\n\n---\n\n**Question: What metrics do the subfigures 2~4 in Figure 3 correspond to?**\n\nThis corresponds to the style transfer metrics we define on lines 200-228: target style strength, fluency, meaning similarity, and the product of these metrics, $\\mathcal{V}$, an aggregate metric.  ($\\mathcal{V}$ is the same metric reported in Tables 1 and 2). Figure 3 shows the results of an ablation where we vary the initial data pool used in the reinforcement learning procedure. \n\n---\n\n**Question: How is the value in Table 4 calculated? I can not obtain it using the equation in line 226**\n\nWe calculate the aggregate metric $\\mathcal{V}$ on the **sentence level** (following previous work from Krishna et al., 2020). Each style transfer input/output has the following associated metric values: target style strength (TSS), meaning similarity (MS), output fluency (F), and the product of these metrics, $\\mathcal{V}$ (line 226, as you mentioned). In Table 4, we report the **average** of these metrics across all sentences in the CDS test set. Though for each sentence $\\mathcal{V}$ is equal to the product of the TSS, MS, and F, once we take the corpus-average of these metrics, we lose this equality guarantee. **$\\mathcal{V}$ in Table 4 is a *average of products* while taking the product of TSS, MS, and F (from Table 4) as you suggested would be a *product of averages*.**\n\n---\n\n**Typos, Grammar, Style, And Presentation Improvements**\n\nThanks for the suggestions! We will correct these in the next version of this work, including adding a citation on arbitrary style transfer to the second paragraph of the introduction, correcting the typos, and making the evaluation metrics more clear.\n\n---\n\n**References**\n\nKrishna, K., Wieting, J., & Iyyer, M. (2020). Reformulating Unsupervised Style Transfer as Paraphrase Generation. ArXiv, abs/2010.05700.\n\nLu, X., Welleck, S., Jiang, L., Hessel, J., Qin, L., West, P., Ammanabrolu, P., & Choi, Y. (2022). Quark: Controllable Text Generation with Reinforced Unlearning. ArXiv, abs/2205.13636.\n\nRamamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., & Choi, Y. (2022). Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization. ArXiv, abs/2210.01241.\n\nReif, E., Ippolito, D., Yuan, A., Coenen, A., Callison-Burch, C., & Wei, J. (2021). A Recipe for Arbitrary Text Style Transfer with Large Language Models. ArXiv, abs/2109.03910.\n\nReimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Conference on Empirical Methods in Natural Language Processing.\n\nSuzgun, M., Melas-Kyriazi, L., & Jurafsky, D. (2022). Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models. ArXiv, abs/2205.11503. \n\nWarstadt, A., Singh, A., & Bowman, S.R. (2018). Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7, 625-641.\n\nWieting, J., Berg-Kirkpatrick, T., Gimpel, K., & Neubig, G. (2019). Beyond BLEU:Training Neural Machine Translation with Semantic Similarity. Annual Meeting of the Association for Computational Linguistics.\n\nWu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N.A., Ostendorf, M., & Hajishirzi, H. (2023). Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. ArXiv, abs/2306.01693.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The framework does not rely on large language models but achieves good results.\n2. Experiments are conducted on an 11-style dataset, making the results more solid.", "reasons_to_reject": "1. If I understand correctly, the authors leverage the equation in line 226 as the evaluation metric. However, the metric is directly used to filter data and used as the reward function in reinforcement learning. In other words, this framework is optimized directly against the evaluation metric but the baselines are not, which may be not fair.\n2. The evaluation metric used in this work is different from highly related previous works (Suzgun et al., 2022; Krishna et al., 2020), but the reason is not explained.\n\n[1] Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022. Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2195–2222, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n[2] Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, Online. Association for Computational Linguistics.", "questions_to_authors": "1. Do I understand correctly that the equation in line 226 is used as the evaluation metric?\n2. Any comments on reasons to reject 1 and 2? I will raise my score if they are addressed properly.\n3. What metrics do the subfigures 2~4 in Figure 3 correspond to?\n4. How is the value $\\mathcal{V}$ in Table 4 calculated? I can not obtain it using the equation in line 226, i.e., $0.428\\times0.626\\times0.790 \\ne 0.1994$.", "paper_topic_and_main_contributions": "In this work, the authors propose a unified framework for arbitrary style transfer. It first generates pseudo data using expert language models, then filters these data based on a score function that takes style, fluency, and semantic meaning into consideration, and finally leverages a reinforcement learning algorithm named Quark to train the model. The authors propose to replace the scalar reward with a vectorized reward function. Experiments show that the proposed framework outperforms the compared baselines significantly measured by a specific score.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Citations should be added to the second paragraph of the introduction (Line 50) to make it clear that arbitrary style transfer has already been studied in previous work. Otherwise, readers may misunderstand that this article is the first to propose a solution to the problem.\n2. Line 306: remove the leading space before ``where''.\n3. Line 476: ``3.2'' -> ``\\S 3.2''.\n4. The evaluation metrics used in this work should be made clear."}}]

Author rebuttal:

Meta-review: Using Reinforcement Learning (RL), this work proposes a method for training generative style-transfer models that is particularly useful in settings with limited parallel data. Additionally, this work demonstrates out-of-domain generalization and shows improvement over large language models (LLMs) such as GPT-3, despite being significantly smaller in size. One main concern raised by reviewers is the comparatively complicated method required to train the model, especially when compared to the simplicity of prompting or zero-shot with LLMs. However, I would argue that this isn't really a fair comparison. While LLMs offer the convenience of prompting, they are also computationally expensive. In many cases, fine-tuned, smaller models should outperform zero-shot prompting, as is the case here.

Another point raised by reviewers is whether the evaluation adequately captures general improvement. While STEER has a superior aggregate score, it's important to note that it excels significantly in terms of style strength at the expense of meaning similarity. Therefore, the utility of STEER may depend on whether preserving the original meaning is a requirement or not.
Decision: Accept-Findings