Non-Parallel Text Style Transfer with Self-Parallel Supervision
Authors: Ruibo Liu
Chongyang Gao
Chenyan Jia
Guangxuan Xu
Soroush Vosoughi
Venue: ICLR 2022 Poster (2022)
Keywords: style transfer
non-parallel corpus
imitation learning
language models
political stance transfer

Abstract: The performance of existing text style transfer models is severely limited by the non-parallel datasets on which the models are trained. In non-parallel datasets, no direct mapping exists between sentences of the source and target style; the style transfer models thus only receive weak supervision of the target sentences during training, which often leads the model to discard too much style-independent information, or utterly fail to transfer the style.

In this work, we propose LaMer, a novel text style transfer framework based on large-scale language models. LaMer first mines the roughly parallel expressions in the non-parallel datasets with scene graphs, and then employs MLE training, followed by imitation learning refinement, to leverage the intrinsic parallelism within the data. On two benchmark tasks (sentiment & formality transfer) and a newly proposed challenging task (political stance transfer), our model achieves qualitative advances in transfer accuracy, content preservation, and fluency. Further empirical and human evaluations demonstrate that our model not only makes training more efficient, but also generates more readable and diverse expressions than previous models.

Published as a conference paper at ICLR 2022
NON-PARALLES TEXT STYLE TRANSFER WITH SELF-PARALEY SUPERVISION
Ruibo Liu Chongyang Gao Chenyan Jia
Guangxuan Xu Soroush Vosoughi
Dartmouth College, NNrtthwestern University, University of Texas, Austin
University of California, Los Angeles
[ruibo, gi, soroush. vosoughi soroush.vosoughil
edartmouth.edu
cygaoêu wwwthwesterr..nc edu
chenyar:jia@utexas. edu gxu210cs
ABSTRACT
The performance of existing text style transfer models is severely limited by the non-parallel datasets on which the models are trained. In non-parallel datasets, no direct mapping exists between sentences of the source and target style; the style transfer models thus only receive weak supervision of the target sentences during training, which often leads the model to discard too much style-independent information, utterly fail to transfer the style.
In this work, we propose LaMer, novel text style transfer framework based on large-scale language models. LaMer first mines the roughly parallel expressions in the non-parre non-parallel datasets with scene graphs, and then employs MLE training, followed by imitation learning refinement, to leverage the intrinsic parallelism within the data. On two benchmark tasks (sentiment formality transfer) and a newly proposed challenging task (political stance transfer), our model achieves qualitative advances transfer accuracy, content preservation, and fluency. Further empirical and human evaluations demonstrate that model not only makes training more efficient, but also generates more readable and diverse expressions than previous models.
INTRODUCTION
Text style transfer (TST) models learn how to transfer the style of text from source to tar- get while preserving the style-independent con- tent (John et al., 2019; Fu et al., 2018). Ex- isting TST methods perform well when trans- ferring simple styles, such as sentiment; how- ever, they tend do poor job on more abstract and subtle styles, such as formality and political stance (Lee et al., 2021; Fu al., 2019b).
(a) Exsiting Methods Non- Parailel, Unsupervised
Method:
(Roughly) Parallel, Self- Sati-Supeiiied
LM MLE Training
Style
Mining
0.54
Imitating Learning
Content
Content
The lack of parallel datasets is one of the main bottlenecks for text style transfer tasks, Parallel datasets contain pairwise sentences that only dif- fer in specific styles, while non-pan con-parallel datasets typically contain large corpora of sentences with different styles, without direct one-to-one mapping between sentences (Fu et al., 2019a; Krishna et al., 2020). In real-world settings, is much easier to construct non por-aaallll datasets by collecting large quantities of non non-parallel sentences with different styles; conversely, carefully-aligner parallel TST datasets are much more costly to construct, usually requiring human annotation, and thus scarce.
Red and blue circles represent the source and target texts respectively. (a) Existing methods crucially ignore the inherit parallelism within the data. (b) Our method first mines (roughly) parallel expressions, then learns how to transfer style with the self supervision from the parallel expressions.
Figure 1:

---

Published as a conference paper at ICLR 2022
The lack of parallel datasets for style transfer tasks prompts researchers to develop unsupervised TST methods trained on non non-parat paraallll datasets (Dai et al., 2019; Luo et al., 2019; Shen et al., 2017). Current methods either rely on surface-leve levell split of style- and content-bearing tokens (Li et al., 2018 8a; Castro et al., 2017); or separate content and style vectors in latent space (John et al., 2019; Liu et al., 2020a; Wang et al., 2019) once style and content have been separated, they then swap styles leaving the content unchanged. However, surface-level split is not reliable when the style cannot be extracted at the token level (Sudhakar al., 2019); latent-space disentanglement could also f'il due to a lack of pairwise supervision and results in discarding too much style- pelleindependent content (Reif et al., 2021). However, these methods are less than ideal. As demonstrated in Figure (a), they tend to learn the mapping between source and target styles on the original randomly mapped sentence pairs, but fail to take into account the inherent parallelism within the data (Krishna et al., 2020).
This paper presents LaMer, novel self-supervised text style transfer framework built upon recent text-to-text language models (L.Ms). As illustrated in Figure (b), rather than accepting the random mapping in the given non- non-parallel dataset, we first align sentences from source and target styles based on their scene graphs, and then run conditional MLE training on LM to maximize the probability of generating (roughly) parallel style sentences. As a final refinement step, deploy reinforeed imitation learning with contrastive loss on the aligned data to further reward the best transfer demonstration and penalize the others. In comparison with existing methods, LaMer fully exploits the intrinsic parallelism within non-parallel data and innovatively applies the transfer learning power of lcage-scale LMs to boost text-style transfer performance.
Our contributions can be outlined as follows: First, We present a language model based text style transfer framework, LAMER, that achieves significant advances in two existing benchmark TST tasks and one newly proposed challenging TST task. Human evaluations further confirm that LaMer generation is not only more readable but also richer in diverse expressions than previous methods. Second, We propose a new challenging text style transfer task. Political Stance Transfer, and release a benchmark dataset, which includes news sattence-pairs describing the same fact but from opposite political stances (i.e., liberal and conservative). The stances of the sentences are labeled by expert human editors. Third, As part of LaMer, we introduce a simple but powerful method to create parallel corpora from non-parallel data. We show that this method can be used as general treatment to improve existing TST models. Given the abundance of non- non-parallel corpora for different TST tasks, this method has the potential to greatly impact the field of TST.
Code for LaMer is available at ht https: : //ggghttuub...//// com/ wtt.//oo//aaaaaiie http//DaaaggiuLLMMeeee
APPROACH
LaMer is comprised of three main steps. First, we mine (roughly) parallels target style expressions for each of the source style sentences based on their LM-based sentence embeddings and scene graphs ($2.2). Second, we deploy MLE training on the text-to-text LM BART (Lewis et al., 2020b) to maximize the probability of generating such roughly parallel sentences in target style ($2.3). Finally, we use imitation learning to further refine the generation with the self-supervision from the mined parallel demonstrations ($2.4).
2.1 NOTATION AND PROBLEM STATEMENT
For the task of binary text style transfer (src tgt), We denote the source and target style datasets as DAte and D which contain M and N sentences expressing two different styles respectively (i.c., tet tgt
D=te {s1 sre sre sre 82 .... DiET {si .2 x)). We also denote the tokens in source and target lgt } E s. and (y1, V2.... 31} E 8j for given source and target sentences s and sy Given an input sentence s of the source style, an ideal style transfer model should be able to generate an appropriate, fluent target sentence slg which is of the target style while maintaining the aylle-independent content.
sentences as sre lgt
Existing methods learn a mapping frndim Dh T without considering the self-parallelism between DTd and Dis LaMer, however, works on the assumption that m(s M) sentences in D sec have n parallels in Dist The number of parallels for each of the m sentences is variant but more
2

---

Published as a conference paper at ICLR 2022
than N. Therefore, the goal of LaMer can be framed as first identifying parallel subset of the original non-parallel dataset and learning mapping this dataset: fparallel Pm Di M, N).
2.2 MINING (RoUGHLY) PARALLEL SENTENCES
In this work, we argue that the non-parallel text style transfer datasets have inherent property of parallelism. We outline in the following three jor strategies to mine roughly parallel sentences:
overpriced mons, and mediocre beef.
(they, menu, boof] dgais
donde
Random (RD). For each sentence s E D randomly pick k sentences from D as parallels. This method mainly serves an ablation to show the effectiveness of other strategies.
5-Emb. Allgement
nothing graat on their menu
Rheyy, menu, nothing, beef) discre
donive
Sentence Embedding (S-Emb.). Sentences that only differ in linguistic style should have higher semantic similarity than other unrelated sentences. As more advanced strategy, we leverage off- shelf LMs (e.g., RoBERTa (Liu et al., 2019)) to mine semantically similar sentences. We first en- code every sentence in D and D with their embeddings extracted from an LMM Then, for each sentence 8 D'T, we pick most similar sentences from D through cosine similarity as parallels.
they had such extensive and beef delicious.
Figure An example of aligned parallels by our scene graph based method. We underline the scene entities and annotate how we compute dorder and dxis with slatt and glut.
S-Emb. Scene Alignment Score (S-Emb. + SAS), We add a scene graph based alignment step to the sentence embedding method to leverage shared entities between source and target sentences. Scene graph extraction has been successfully used in computer vision (Venugopalan et al., 2015; Chen et al., 2020b) to guide (Pan et al., 2020) or judge (Anderson et al., 2016) diverse caption generation. Inspired by the CV example and the observation that scene graphs of sentences with similar content but different styles tend to have overlapping nodes, our algorithm highlights the shared entities (nodes) of the sentence scene graph by running the scene graph parser released by Wu et al. (2019)', which outputs mined scenes graphs in rubject-elationnoojjer triplets. In our case, we only consider the end nodes of these triplets (which are subj. and obj.) the scenes entities. Figure shows an example of mined sentences and corresponding scenes entities.
Let le(s "* and |e(s | be the number of scene entities extracted from the source and target style sentence, and ( be the number of their overlaps, we compute Scene Alignment Score (SAS) as:
(1 + B²) precision recall s 8 precision + recall
le(s [e(s
describe the portion of target and source entities that are part of the overlap, and positive real factor 3 is chosen such that recall considered
where precision (=
[e(s and recall (=
times as important as precision in a specific domain. We also normalize SAS by the length of ! since longer target sentence candidates more likely to have higher SAS.
SAS serves as a complementary alignment metric to further filter the parallel sentences picked by the previous sentence-embedding based metric. The complete procedure of S-Emb. + SAS alignment is:
1. Encode each sentence in ah and D as sentence embeddings via a pre-trained LM.
2. Compute cosine similarity between all sentence pairs (y) where 8; E T and 8y E D Pick the i most similar sentences $
(k) from D as parallel candidates for 8;.
TTee scene graph parser can be found here: http://tttt.....aaaaaaa/a////nea
3

---

Published as a conference paper at ICLR 2022
3. For each sentence 8; E D and its parallel candidates 8) compute pairwise SAS by Equation 1. Pick those 8, whose SAS scores (with respect to 8,) are above a threshold p as the final parallel sentences to Si.
From empirical observations, we found that the selection of p and is task-specific and has crucial effects on the performance of LaMer. We explore the outcomes of different choices of k and p in $3.3. Though SAS alignment LaMer relies on entity alignment for mining parallel corpora, other alignment mechanisms, such as (multilingual) LM alignment (i.e., the S-Emb. method) can potentially be used for transfer cases where entities may not be shared across styles (such as ancient v.s. modern English (Xu et al., 2012), or even multilingual text style transfer (Krishna et al., 2021)).
2.3 CONDITIONAL TOKEN-LEVEL MLE TRAINING
After mining parallel sentences for source sentence 8 from our corpus, we take the source sentence ste as the condition, and preform a conditional token-level MLE training on the mined parallels in the manner of standard autoregressive pre- pre-training. Assuming for a certain source sentence
....4) s have n target parallels mined in the previous step (i.e., a {51 82 ...5a J. n N), the MLE loss can be computed as:
1gt lgt
18k
(5) V1 log LM(yh=-1ls*))
JMLE
(2)
!gl where vi E s.. denotes the i-th step ground truth token belonging the i-th target parallel sentence. LM(y)=-1|s is the LM output at f-th step (a distribution on the vocabulary) given the source text and previous generation. We choose BART Lewis et al. (2020b) as the LM, which is already pre-trained on several text-to-text tasks, and concatenate the source and target text with </ />> <>>>>>>>>>>>>>>>>>>>>>>>>>..>< tokens pre-defined by BART input data. At each step, with the supervision of ground truth target (4) tokens Yi we minimize the MLE loss only at the target part. The LM is thus trained to maximize
the probability generating tokens ýt approaching to yr (5)
Though the results produced are reasonable, solely relying MLE training has several defects. First, MLE treats all parallels of certain source equally without penalizing low-quality demonstrations, which not only makes the model less robust noisy data, but also possible to be misled by negative samples during local optimization. Sccond, token-level MLE does not pay special attention to scene entities shared between the source and target, which found take up the main portion in the human written references (as demonstrated in $A.3). We are thus motivated to further improve the performance using the self-supervision from the parallels.
2.4 REINFORCED IMITATION LEARNING BY CONTRAST
For all the mined target style parallels s of sre we denote the one that has the highest similarity score (with S- 5-Eb..) or has the highest SAS (with S-Emb. + SAS) as the expert demonstration get while the others are set as s 1gt-
In the formulation of reinforcement learning, each step the policy To observes the state 91:-1 (partially generated text), takes an action a, Vt (pick token), and transits to the next state Vi.z. In the context of TST tasks, the policy can be interpreted as the LM-based generation model: LM(a, (M((a,//:t-11) // s ) and the of !8t- can be seen as the trajectory derived from an expert policy x while 1gt is produced by an amateur policy x. The goal of this step imitation learning can thus be framed learning policy that has more probability to behave like x rather than म which can be achieved by minimizing the contrastive loss:
Jn. m max
[ [
4.)),
(3)
where (-) is the distance measurement of two policies in terms semantic coherence dsEm() and scene preservation desv (-), such that 16 (.) ddsum()) dpsv and A is hyperparameter
4

---

Published as a conference paper at ICLR 2022
that controls the contrasting margin. The general form is inspired by the triplet loss (Schroff et al., 2015) and recent in- in-batch constrastive learning (Gao et al., )) we encourage the learned policy क to be more similar to the expert one π." than the amatcur one π by at least margin A. dsam (-) and dpsv(2 (+) are detailed below.
Sequence-level Semantic Coherence. For sequence 8 generated under current learning policy ît and a certain target style demonstration o!",, we encode them using bi-directional LM RoBERTa (Liu et al., 2019). which is widely used sentence encoder in related work (Reimers & Gurevych, 2019; Li ct al., 2020). Then, we take their negative cosine similarity as the distance for semantic coherence: emb. dsum ||emb; ||l|emb,,eil Lower value dsuM means the learned policy can produce similar quality sequences in terms of semantics as target style demonstrations. It is sequence-level measurement since the encoding is based on the whole sentence.
Token-level Scene Preservation. In our context, legitimate target style parallels should preserve most scene entities that appear source style sentences. To explicitly promote the content preser- vation performance of the current policy, we introduce a token-level distance dpsv- We first extract scene entities from ger+ and each of the 8 jet- and then measure the differences in both the order of appearance of the entities and the existence of the entities. Taking g as an example (sim- ilar calculation done on each of the 8 )."), we compute: dpsv+ addordr+ + (1 - a)) allē(s [J g+*)) ). [ltew + (1 (1-al) (1-a)]lé(s ()) ( -ē(s set ")) where ē(.) means the extracted scene entities, é(-) is its one-hot encoded vector, || Vll measures Levenshtein distance (Yujian & Bo, 2007), one type of edit distance, and for vectors e(s*)) and e()!)), we compute their hamming distance (Norouzi et al., 2012). Lower donter+ means the appearing order of ē(s resemble e(s (*t).. Similarly, lower dexiu+ means e(s"") and e(s***) are mostly overlapped. Similar procedure applies to dpsy- as well. We present more details about how we compute these distances in $A.4.
2.4.1 POLICY GRADIENT BY REINFORCE
We choose the REINFORCE algorithm (Williams, 1992) to optimize the current policy mg. As shown in Equation 4, we use greedy decoding results as the baseline (Rennie et al., 2017), and sampling decoding for the exploration step, which mimies the actor-critic manner in reinforcement learning (Fujimoto et al., 2018; Haarnoja et al., 2018). The policy gradient can thus be presented as:
V J(0) -E,,
Volog Tog (JiL. (Jummplee(ss (slett i slstt) -Jrroody (s 1stt
(4)
where prample and frreddy can be both calculated by Equation 3. We take the difference of two the mode rewards as the advantage, feeding into the REINFORCE algorithm to optimize our policy my. Finally, the output policy To is able generate style transferred sequences resembling the expert demonstrations g giutt More details about the complete loop of policy gradient can be found in $A.5
Discussion: Imitation Learning, Reward-based RI., and MLL,, Framing token-by-token text generation as a sequential decision-making process, reward-based reinforcement learning (RL) has achieved success in text generation by directly setting indifferentiable evaluation metrics such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) as the reward function (Gong et al., 2019; Liu et al., 2020b). In the context of TST tasks, however, instead of reward- based RL, we argue imitation learning (IL) is a better choice, since: 1) is challenging to design a uniform reward fitting a variety of source style text (Chen et al., 2020a), and 2) it is also unnecessary to calculate designed reward. The roughly parallel target style sentences mined the alignment step can naturally serve expert demonstrations (Li et al., 2018b). They are high-quality and cost-free, which can greatly improve the sampling and learning efficiency of RL optimization.
elaborately
A common obstacle of IL is known covariate shift (Ross & Bagnell, 2010), which means IL can provide a relatively good performance for the test cases similar to the demonstrations (used for policy training), but it could still suffer from bad generalization for cases it never meets during training, as
a: controls the weights assigned denter and deasi set by rumning repeated experiments ranging the a from to by 0. and picking the berf-perf a with respect to GM: a = {0.4, 0.3, 0.1) for the three tasks. "Hamming norm || a is defined as the number of non-zero entries of vector
5

---

Published as a conference paper at ICLR 2022
the number of demonstrations is finite. Similar problems called exposure bias (Bengio al., 2015; Ranzato et al., 2016) also exists in MLE when the supervised data is scarce. We claim our token-level MLE training LMs can somewhat mitigate such problems, since the large-scale LMs are already pre-trained on huge quantities of data across many domains as prior.
3 EXPERIMENTS
3.1 DATASETS AND EXPERIMENTAL SETUP
Sentiment Transfer. We use the Yelp reviews dataset collected by Shen et al. (2017) which contains 250k negative sentences and 380k positive sentences, organized in non-paree non-parallel fashion. Li al. (2018a) released a small test set that has 500 human written references for both target sentiments, and they are already one-on-one aligned with the source sentiment sentences.
Formality Transfer. A more challenging TST task is to modify the formality of given sentence. We use the GYAFC dataset (Rao Tetreault, 2018), which contains formal and informal sentences from two domains. In this paper, choose the Family Relationship domain, which consists of about 52k training sentences, 5k development sentences, and 2.5k test sentences. Different from the Yelp dataset, the test set of GYAFC has four human written references for each source style test case. Our evaluation results for this task are reported as the average against the four references.
Political Stance Transfer. News outlets reporting the same event naturally meet the requirement of style transfer data, which is aligned in content but different in style. We leverage Allsides.com commess.ccom 4 as an aggregator, where outlets reporting the same events with different political stances are already organized and labeled by expert human editors. In total, we collect 2,298 full-length news articles from 6/1/2012 to 4/1/2021 in pairs of liberal-conservative leaning. Though the data seems parallel at the article level, in this work we focus on sentence level TST. We further split the articles into about 56k sentence pairs written by expert writers from liberal- or conservative-lcarnin media outlets, and aligned by the Allsides editors. For ethical consideration, containing sensitive topics (such as hate speech). To adhere to the outlets' TOS, we will release our dataset links the articles (instead of the text), with the appropriate code extract the content for training purposes. A closer comparison of our proposed novel dataset with another political stance TST dataset (Voigt al., 2018) can be found in IA
non-parallel dataset for training (detailed in $A.2). These titles
manually examined the articles to removing those
The detailed statistics of all three benchmark datasets and our experimental setup can be found in SA.3. We have discussed the hyperparameter selection for the best performing LaMer in $3.3.
3.2
MAIN RESULTS ON TEXT STYLE TRANSFER PERFORMANCE
TST models should be able to transfer source style sentence to a target style, while maintaining the syle-indeepndent content as much as possible. Most existing works evaluate their methods with transfer accuracy (ACC; classification accuracy) and content- preservttrss (BLEU-ref: token overlaps with reference). In this work, besides using ACC and BLEU-ref, we also report more advanced metrics- SIM and FL introduced by Krishna et al. (2020) to further evaluate our method.
LaMer has competitive performance. Table shows the main results comparing LaMer with eight popular TST models on the three benchmark datasets. In addition to the independent ACC and BLEU score we also report their geometric mean (GM) for direct comparison with the DualRL and STRAP baselines: however, should be noted that GM has not been well studied as an evaluation metric for TST and is only included for comparison purposes. We also consider inverse PINC scores (Chen& Dolan, 201 2011)-iPPNN--ap a complementary metric of BLEU. i-PINC counts the n-gram overlaps after removing the tokens shared between source and target style sentences, and thus properly penalizes the Input Copy method (i-PINC = 0).
Compared with existing methods, LaMer performs better in terms of overall metric GM, and shows particularly strong net transfer ability (since i-PINC does not count duplicated tokens between
'nttps: // //wwww a ssides. com/ story/admin admin WWe use BLUU-ref which computes average n-gram BLEU (n ranges from 1 4).

---

Published as a conference paper at ICLR 2022
Table 1: LaMer achieves competitive performance in three TST tasks. CAE (Shen al., 2017), DelRetrGen (Li et al., 2018 8a), DualRL (Luo et al., 2019), Style Transformer (Dai et al., 2019). Deep Latent Seq (He et al., 2020) are popular TST models built on LMs. TSF-DelRetGen (Sudhakar et al., 2019), IMaT (Jin et al., 2019), and STRAP (Krishna et al., 2020) are recent LM-based TST models. Input Copy directly copies the source style sentence as the transfer output. GM: Geometric Mean of Transfer Accuracy (ACC) and BLEU. i-PINC measures net transfer ability by not counting words that can be directly copied from the source style sentence, as discussed in $3.2. We color the best results, and underline the second best, not including the Input Copy reference.
Sentiment
Formality
Political Stance
SAS
0.73
SAS = 0.29
SAS 0.30
Existing Methods
ACC
BLEU
GM
i-PINC
ACC
BLEU
GM
i-PINC
ACC
BLEU
GM
iPINC
Input Copy (ref)
7.4
61.7
21.4
0
1.9
54.6
10.2
7.5
17.8
11.6
CAB
70.5
6.0
20.6
1.2
39.5
35.8
37.6
5.9
31.5
5.1
12.7
1.4
DelRetrGen
65.4
11.8
27.8
4.8
61.9
14.8
30.3
6.6
93.8
3.5
18.1
6.2
Dual RL
85.6
21.2
42.5
4.0
71.1
31.9
47.9
7.2
91.8
8.1
27.3
5.7
Style Transformer
93.6
17.1
40.0
5.2
72.2
32.7
48.6
48.6
18.8
1.1
Deep Latent Seq
87.9
18.7
40.5
7.3
80.5
18.5
38.6
6.1
85.8
20.8
42.2
3.0
LM-based
TSF-DelRetten
89.5
31.7
53.3
4.5
79.3
17.4
65.8
39.5
IMaT
22.5
46.5
3.3
72.1
28.2
45.1
12.7
82.9
19.5
40.2
5.3
STRAP
92.3
32.5
54.8
6.4
75.4
32.6
49.6
9.7
84.3
42.4
Ours: LaMer
w/.
97.1
19.8
43.8
2.5
64.2
20.4
36.2
5.5
75.9
27.7
w/. S-Emb.
90.1
[40.6)
660.5
5.6
65.1
33.8
46.9
10.9
78.2
22.1
41.6
9.2
w/. S-Emb, SAS
97.0
34.1
57.5
9.6
76.5
39.2
54.8
13.3
82.7
30.5
50.2
13.6
and )²²)). Among the three mining strategies we proposed in $2.2, the S-Emb. + SAS seems offer the best performance boosting in terms of content preservation, but shows no clear advantage on style control accuracy. When comparing LaMer with the three LM-based models, we find the advantage of LaMer mainly comes from the gain in content preservation (BLEU), which demonstrates the effectiveness of our scene graph alignment. We also find some of the other TST models produce low-fluency (i.e., high perplexity) transferred sentences (see Table 10 in AA.7), though we must note that recent studies have shown perplexity be a poor measure of fluency (Krishna et al., 2020; Mir et al., 2019; Lee et al., 2021).
Table Few-shot performance on Formality TST (only 1% training data available) of LaMer and other LM-based baselines. We also include zero-shot TST method built on GPT-3 (Reif al., 2021) for reference. We annotate standard deviation next to the results as well.
Table 3: Compared with
other baselines, LaMer re- quires neither extra data training additional models.
Training Data (%)
1% for GPT3)
100%
Ext. Resource?
Data Model
Method
ACC
SIM
FL
JASF
ACC
SIM
FL
JASF
ISF-DelRetGen
TSF-DelRetGen
16.5,4
5.5
90.3m.
2.6
54.1
30.5oa
98.9k1
29.7as
IMaT
IMaT
35.9p.
22.1oz
23.0aa
3.2
59.512
39.7..
99.6u1
37.4az
STRAP
/
STRAP
37.Jo.7
12.56g
70.402
5.304
67.7
72.5
90.40,
45.5
Ours: LaMer
48.8o.4
27.4u6
89.8a1
13.5a3
66.405
74.303
97.8k.
53.2g.1
Ours: LaMer
x
GPT-3
prompt 3PT-3ppompire
10.2g6
17.1
98.0gz
6.5ua
GPT-3yyompe IST
X
x
LaMer is robust in few-shot settings. LaMer learns the inherent parallelism within the unsuper- vised data, which could be challenging in data scarce cases. Recent works have explored the limits TST in few-shot settings (Riley et al., 2021; Krishna et al., 2021), we conduct similar studies by limiting the available training data (to 1 1%), aiming to answer whether not LaMer can still provide comparable performance as others. We consider LM--based baselines which show strong results in Table 1. We also compare with zero-shot prompt- based TST baseline built upon GPT-3 (Reif et al., 2021) for reference. For evaluation metrics, we use the recently proposed J-score framework (including ACCuracy, SIMilarity, and Fluency) (Krishna et al., 2020), which solves many of the
7

---

Published as a conference paper at ICLR 2022
Sentiment
Formality
Political Stence
100
(k:1000) pm).4)
(k:500 p.0.4)
(4.200 p.o.6)
*
(0-500 4)
(=500,ppe03)
[e-200, p=O
(0200 p=0.3)
0k-500 p.0.5)
94
(K-200 pi3.7)
0K2200 p-0.4)
72
(e-200, pro
(4-300, pmo.0)
93
100, p.0.6)
(c-50, p=0 4)
(K:50, po0.4)
35.00
BLEU
BL5U
Figure 3: Effect on performance when choosing different is and p for S-Emb. + SAS alignment. We also annotate (with star) the recommended settings which reaches best balance between style control accuracy and content preservation BLEU score.
issues in existing metrics" s. As shown Table 2, we find that LaMer achieves great performance (J-score) in few-shot cases, partially because other baselines rely on external systems trained on ample data;" 7 In extreme data hungry cases, the compromised external modules would propagate errors to following parts as they are coupled. LaMer, instead, requires neither extra data nor additional training on external systems (See Table 3), but fully leverages the self-supervision hidden in the unsupervised data, which achieves robust performance even in few-shot scenarios.
IL refinement erucial for LaMer content preservation. As shown in Table 4, the BLEU score for LaMer drops substantially when the imitation learning refinement stage is totally removed (No IL, MLE only), and we find this drop is higher for challenging style tasks (such as political stance transfer), partially because the quality variance of their demonstrations is higher, i.e., further guidance towards good demonstrations will make a big difference. We also conducted further ablation studies components of IL: In general, removing doeder seems to cause greater performance drop in formality TST, while desist seems important to successful political stance TST. Removing all scene preservation distance sub-components (i.e., IL (dsex)) leads to noticeable performance deterioration across all three TST tasks. We take these ablation study results as evidence that the technologies we propose for LaMer are all crucial.
Table 4: Further ablation on components of LaMer imitation learning (IL). dSEM: semantic coherence distance, donder and dixiu are sub-components of scene preservation distance dpsv. Removing all IL modules (MLE only) will cause severe drop in content preservation (BLEU), especially for challenging TST tasks (Formality and Political Stance TST).
Sentiment
Formality
Political Stance
Ablation LaMer
ACC
BLEU
ACC
BLEU
ACC
BLEUU
IL (csEM dorder + dexis)
97.0
34.1
76.5
39.2
93.4
18.5
IL (dsma dender)
95.5 (11.5)
34.4 (10.3)
74.7 (U1.8)
37.7 (11.5)
93.2 (10.2)
15.8 (12.7)
IL (dseM desisa
96.5 (10.5)
72.6 (10.2)
(13.9)
34.1 (15.1)
92.7 (10.7)
17.1 (J1.4)
IL (dsus)
95.1 (11.9)
33.3 (10.8) 71.9 (14.6)
326.77)67)
89.9 (10.5)
13.9 (14.6)
No II. (MLE Only)
93.1 (13.9) (10.9
68.7 (17.8)
27.2 (112.0)
89.8 (10.6)
8.8 (19.7)
3.3
CONFIGURING LAMER
The filtering parameter p and l are hyperparameters that crucial for the construction of roughly parallel datasets. Thus, for all three TST tasks, we evaluate LaMer performance given different combination of p and k, shown in Figure 3. We find there is approximate trade-off relationship between style control accuracy and content preservation BLEU score: In general, higher & (more parallels) seems to improve accuracy but normally has lower BLEU, while for a fixed k, a proper
'E.g., BLEU score can be artificially high when directly copying source style sentence as the target style generation, and perplexity unbounded and discourages ereative TST generation.
'F.g. TSF-DeRRetGen requires training classifier to locate style-carrying spans of text. STRAP fine-tunes GPT-2 model to learn how to paraphrase. IMaT trains plain Seq2Seq model from scratch L.M.
8

---

Published as a conference paper at ICLR 2022
p can be reached empirically, We annotate the recommended settings of p and & for the three TST tasks we studied in this work.
3.4 HUMAN EVALUATION
We conducted human evaluation to answer: 1) "How good is LaMer's SAS alignment?", and 2) "How is the quality of the style-transferred sentences?" (The exact questions and procedures are details in $A.6.). We recruited 212 participants from MTurk. Most annotators agreed that SAS-aligned pairs share similar style-independent content (Table 5). We conducted several one-way ANOVAs to test the difference among the seven baselines and LaMer (political stance TST results shown in Table 6; see Table 8 for sentiment TST and Table 9 for formality TST in $A.6). LaMer was significantly better than next-best all three perspectives (p 005, and had the most ratings that where 5.
Table 5: Effectiveness of SAS align- Table 6: Human evaluations on the political stance transfer task ment for mining weakly parallel sen- (on point scale). We bold the highest average rating. LaMer tences. Similarity: "How much was rated significantly higher than the next-best method (p < the mined pairs similar in style- 005). Allas: The ratio for which the ratings are 5 for all three independent content?" SAS Align: perspectives.
"How well SAS scores reflect such ilarity?" Ratings are a 7 point Methods scale. All24: The ratio for which the
Political Stance
Style
Content
Readability
Mean
SD
Mean
SD
Mean
SD
Allas
ratings are 2 4 for both questions.
CAE
4.91
1.59
4.85
1.58
5.04
1.68
12%
DelRetrGen
5.14
1.60
4.93
1.63
4.96
1.60
11%
Similarity
SAS
Dual RL
5.15
1.37
4.87
1.59
5.15
1.53
27%
TST Tasks
McanşD
Mcanşp
Allg4
Style Transformer
5.31
1.19
5.34
1.36
5.18
1.40
28%
Deep Latent Seq
5.42
0.57
5.29
1.20
5.30
1.26
25%
Sentiment
5.42
5.56
88%
TSF-DeRRetGen
4.87
0.65
5.23
1.14
5.20
1.13
15%
Formality
5.58
5.74
82%
STARP
5.33
1.25
5.31
1,40
5.43
1.07
21%
Political
5.45
5.43
74%
Ours: LaMer
5.59
1.17
5.44
1.11
5.56
1.05
37%
4 RELATED WORK
Besides the baselines have studies in the paper, notice there other TST methods focusing on either latent vector manipulation (Liu et al., 2020a; Wang et al., 2019; John et al., 2019) or surface level editing (Castro al., 2017; Karadzhov al., 2017; Mansoorizadeh al.). LaMer differs from them as is learning to transfer with self-supervision mined from the unsupervised data, without complicated manipulation or editing. The idea of mining roughly parallel demonstrations LaMer seems to echo the recent findings of retrieval-based generation systems (Lewis et al., 2020a; Lee al., 2019)- retrieving relevant documents from external resources (e.g., Wikipedia) as grounding can improve the quality of generation (Lewis et al., 2020c; Guu et al., 2020). LaMer does not rely on any external resources, but mines parallel demonstrations from the training data itself, and thus is easier to deploy and alleviates potential bias.
CONCLUSION
We have proposed LaMer, self-supervised text style transfer method built on large-scale LMs. LaMer demonstrates the potential of leveraging parallaaal within large non-parallel corpora, which can guide efficient TST models development even I.M design. On three diverse TST tasks, we show that LaMer performs better than eight existing methods while being robust to data scarcity. Human evaluations further validate the advantage of our model compared to existing baselines.
Future work of LaMer could study how to extend to multilingual scenarios, or transfer the style that essentially changes the form of language (e.g., ancient v.s. modern English). One could also investigate combining the alignment module of LaMer with the text2text LM one model, and jointly train to further boost the performance. Another direction is further study the integration of LaMer with larger size foundation models, like GPT-3 175B to understand LaMers potential in few-shot settings.

---

Published as a conference paper at ICLR 2022
6 ETHICS AND REPRODUCIBILITY STATEMENT
The goal of LaMer is to provide a simple but efficient general-purpose text style transfer framework by leveraging large-scale pre-trained LMs. LaMer can generate text in a specified target style given source style input, which can serve as a useful tool for social good. For example, LaMer can assist language formalization (formality transfer), enable data anonymization (writing style transfer), or depolarize politically biased text (Liu et al., 2022; 2021a), via transferring the identifiable style of the input text.
The performance of LaMer depends the pre-trained LM, which means that the generation of LaMer can be affected by certain socictal biases found in the pre-trained LM is based on (e.g.. BART) (Blodgett et al., 2020). Other negatives such abusive language (Hancock al., 2020), hate speech (Kennedy et al., 2018), and dehumanization (Mendelsohn et al., 2020) are also prevalent in the language data for LM pre-training. When deployed in real-world settings, LaMer may increase the risk of reinforcing and propagating polarizing and toxic language in public space through people's dissemination (Waseem et al.. 2017). Moreover, the generated text of LaMer could potentially be included in the next iteration of large-scale LM training data collection, which may amplify offensive stereotypes and overtly abusive views in the future LMs.
Though LaMer was developed as an "neutral" tool mainly for research purposes, it can also be used in malicious ways such as generating sophisticated spam in a particular style "disguise", which we strongly discourage (Rae et al., 2021; ; Liu et al., 2021b). Due to the lack of fact checking systems as a safety measure, LaMer at the risk of generating seemingly coherent but counterfactual text in certain target style, which could further undermine social trust in computer- mediatee forms of communication (Liu et al., 2021c; Petroni et al., 2019). We thus recommend the users of LaMer to integrate necessary safety checks into the actual systems.
In addition, our experiments and analysis are done when source and target style sentences are both in English, and therefore we do not claim that findings will generalize across all languages, although our framework has the potential to be extended to other languages with necessary modifications.
Though it's beyond the scope of this paper and our individual ability to address all mhove-mentioned complications, we strongly encourage users of our system (and other TST models and systems based on pre-trained language models) to acknowledge and beware of these issues.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: image caption evaluation. In European conference on computer vision, pp. 382-398. Springer, 2016.
Semantic propositional
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Informa- tion Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. wttpppppe///.///aaeeeep Abst html.
1171-1179, 2015. URL https: http::/:///poceednng.. e
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454-5476, Online, 2020. Association for Computational Linguistics. doi: 10. 00118653/1/2020.00 -aainn00-.ammm URL https: htt//aaaaantthooeeeeeeeee org/ 2020. ac1-main. 485.
Daniel Castro, Reynier Ortega, and Rafael Muñoz. Author masking by sentence transforma- titn-notebook for pan at clef 2017. In CLEF 2017 Evaluation Labs and Workshop-Wond Working Notes Papers, pp. 11-14, 2017.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Pro- ceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human
10

---

Published as a conference paper at ICLR 2022
Language Technologies, pp. 190- 200, Portland, Oregon, USA, 2011. Association for Computa- tional Linguistics. URL https: : htt.aaatthoooooo..oo11111-----1e
Liqun Chen, Ke Bai, Chenyang Tao, Yizhe Zhang, Guoyin Wang, Wenlin Wang, Ricardo Henao, reinforcement
and Lawrence Carin. Sequence generation with opimal-ranssoor-enhannc learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7512-7520. AAAI Press, 2020a. URL https: //aaai. wwg/oos///oooooooooooooooo....to index. php/ /AAI/ w.A.I//rticl/////.......a...aaae
6249.
Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9959-9968. IEEE, 2020b. doi: 10.1 11000///2222000200000009999999 URL https: / wtt.ooooog/..11.. CVPR42600 2020.002200 00998.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5997-6007, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 181188//1//11--16000 URL https: /ccllaaclccc/ anthology. org/P19-13
Yao Fu, Yansong Feng, and John P. Cunningham. Paraphrase generation with latent bag of words. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- BucheBBu, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 13623-13634, 2019a. URL ht / http://pooceeeeng.ccceece paper/201 9 hhshhahh///// 5e2. (t2222666552222 25d. 2aa.ear7500022222222222222222112 Absttrfffebbbttrat.eccac html.
Yao Fu, Hao Zhou, Jiaze Chen, and Lei Li. Rethinking text attribute transfer: A lexical analysis. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 24-33, Tokyo, Japan, 2019b Association for Computational Linguistics. doi: 10. 10.118/3/1//111100 8604. URL https: wwwaannnnolloooooooooooooocooooa org/ 9-6044
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Explo- ration and evaluation. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18). the 30th innovative Applica- tions of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 663-670 AAAI Press, 2018. URL https: // www. aaai .org/ /ww///oc//////oooo/////////o index, . php/ hwtppppap//.aaa./p..paapappppppp AAATAAAATAAAAAA paper/view/ 17015.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. URL http:/ http::/pproceeeinieeee http///ppeeeenggceeeeeep/ mir, prwss/esvvve /80/ Eu fujimoto18a, html.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sen- tence embeddings. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 18.53/6//vll 1/2021 m5.5lp- URL https://aclanthie
org/ 20220021112 emnlp-mainnnm.eee.e 552.
Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3168-31888001 3180, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 acl aanthooogy... org/N1 9- 9320.
URL https://
11

---

Published as a conference paper at ICLR 2022
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wo Wei Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3929-3938. PMLR, 2020. URL http:/ proceedinggg ings. ngs. mlr w.r..press/vll 9/ gww.0a. html.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockhoimssäässan, Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Reseurch, pp. 1856-1865. PMLR, 2018. URL http: httpp//roceee....... mlr. prwss/esssseeee v80 /ttp/aaaroooo/ ja 8b. html.
Jeffrey T Hancock, Mor Naaman, and Karen Levy. Ai-mediated communication: definition, research agenda, and ethical considerations. Journal of Computer-Mediatte Communication, 2591) 2020.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatriee probabilistic formulation of unsupervised text style transfer. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: //openreview net wowoooodddooolllto
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. IMaT: Unsupervised text attribute transfer via iterative matching and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNIPP) pp. 3097-3109, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D19- aclanthology, otg/D19-13000
URL https://
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 424 434, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/11111-1044) URL https: /atpaattolaaall.y....//..
Drg/P19-1041.
Georgi Karadzhov, Tsvetomila Mihaylova, Yasen Kiprov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. The case for being average: A mediocrity approach to style masking and author obfuscation. In International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 173-185. Springer, 2017.
Brendan Kennedy, Drew Kogon, Kris Coombs, Joseph Hoover, Christina Park, Gwenyth Portillo- Wightman, Aida Mostafazadeh Davani, Mohammad Atari, and Morteza Dehghani. A typology and coding manual for the study of
rhetoric. PsyArXiv. July, 18, 2018.
Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 737-762, Online, 2020. Association for Computational Linguistics. doi: 10. 118653/1120200mmmll 2020. mamnp-mmiin.5.
URL https: https///caaaaaaollaeooeeoo www.hllohhooooo..ooooooooooo org/
Kalpesh Krishna. Deepak Nathani, Xavier Garcia, Bidisha Samanta, and Partha Talukdar, Few-shot controllable style transfer for low-resource settings: A study in indian languages. ArXiv preprint, abs/2110.07385, 2021. URL https: org/ abs 2210.1101.. 07385.
Dongkyu Lee, Zhiliang Tian, Lanqing Xue, and Nevin L. Zhang. Enhancing content preservation in text style transfer using reverse attention and conditional layer normalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference Natural Language Processing ( (Volume 1: Long Papers), pp. 93-102, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 long.. URL https: //acl clnahho ogy..oog/.ooo.o....o acc 1ong. 8.
12

---

Published as a conference paper at ICLR 2022
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086-6096, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/119-166 1612. URL https::::: : / ac anthol ogy. org/p1 9-11111911111 61 2.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida I. Wang. and Luke Zettlemoyer. Pre-training via paraphrasing. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020a. URL https: httpttttt//ttooooto http://pcceeeennggeceeeee/ees nwwriinsss..gnnn...............s cc/ paper / /200 / hash d6f1 aabde 757
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequene pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880 1-7880, Online, 2020b. Association for Computational Linguistics. doi: 10. 10.18653/11/2020.co main.. 703. URL https: / http///aaaaaattohceeo
org/ 2020. manlmmmmin.... 703.
Patrick S. H. Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni. Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis. Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmmented generation for knowledge-intensingg NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020c. URL htttpp::/pproceeeeteeeete 8b493230200 E7 80 711. html.
neurips
Jingjin Li, Zichao Li, Lili Mou, Xin Jiang, Michael R. Lyu, and Irwin King. Unsupervised text generation by learning from search. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neurul Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual, 2020. URL https: /ttp:p:::///roee pwoceedooeeeceooooooooooooooeece iggs neurips..... wttpriis...cpppppeee cc/paper/ 2020/ / htt.aaahhhhh/ /1
477ae2dd33333333 (eeida1dddd611111 922al-992---- -aa.raattdd11e222-attttttttt.. .tmm..
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationall Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1865-18744 New Orleans, Louisiana, 2018a. Association for Computational Linguistics. doi: 10. 12118865/v1 //11 169, URL wwwps:ttp..............t / acc anthol ogy. wrg//o 8- 69.
Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. Paraphrase generation with deep reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3865-3878, Brussels, Belgium, 2018b. Association for Computational Linguistics. doi: 10.18653/v Lt//DD-8-11111 URL https::///// http:///aaaaaatthhoeooeo org/ 2-18-12211
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics.
URL https:/ ac aclanni aclanthology..............o. org/ 1013.
Dayiheng Liu, Jie Fu, Yidan Zhang, Chris Pal, and Jiancheng Lv. Revision in continuous space: Un- supervised text style transfer without adversarial learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Phirty-Second Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educutional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8376-8383, AAAI Press, wttetticeeeeeeweeeeeeeeeeeettttt vwl.:///w..............
2020a. httptts:: /aaaaaaaaaa......a/.aaaaaaa..aaa org/ o js/ winex.ndxx..///// php/
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 9031-9041, Online. 2020b. Association for Computational Linguistics. doi: 1111866/1/22222 emnlp-main. 726. URL https:: : / a/acl antho 1 logy. , org/ 2000 emn 1p-mat i n. 726.
13

---

Published as a conference paper at ICLR 2022
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Mitigating political bias in language models through reinforced calibration. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 35(17):14857-1488) 202 URL httth:////// aaai org/ index. sondex.ppp/ie AAA www.AA//aaiiiii 1777744
Ruibo Liu, Jason Wei, and Soroush Vosoughi. Language model augmented relevance score. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6677-6690, Online, 2021b, Association for Computational Linguistics. doi: 10. 1/22322222221. acl-long.521 URL https: http:sss/aaaantthlopgooo org/ 2022/ weee2aacc-120 -ong- 521.
Ruibo Liu, Guoqing Zheng, Shashank Gupta, Radhika Gaonkar, Chongyang Gao, Soroush Vosoughi, Milad Shokouhi, and Ahmed Hassan Awadallah. Knowledge infused decoding. In International
Conference on Learning Representations, 2021c. URL http::/://///p/viee/e/e/ net/pdf? id-upnDJ7itech.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. Quantifying and alleviat- ing political bias in language models. Artificial Intelligence, 304:103654, 2022. ISSN 0004-3702. doi: https::/doi.org//D0 com/ /ttp://sceeeee/aatttlll/////////
URL https: : /wwwww sciencedirect.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. ArXiv preprint, abs/1907. 1692, 2019. URL https: /ttp/vaxvvvv///v////////////./al
org/ abs / 907. 692.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. A dual reinforcement learning framework for unsupervised text style transfer. In Sarit Kraus (ed.), Proceedings of the Twenty-Fighth International Joint Conference
Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 51 16-5122 ijcai.org, 2019. doi: 10.24963/ijcai. 2019/711. URL https: /ttps://ddo... /tt.i//iiioo///ooo/ooooo org/ 10. 24963 /ttee/ijaaaa..//o 2019/711.
Muharram Mansoorizadeh, Taher Rahgooy, Mohammad Aminiyan, and Mahdy Eskandari. Author obfuscation using wordnet and language modell--notebook for pan at clef 2016.
Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky. A framework for the computational linguistic analysis of dehumanization. Frontiers in artificial intelligence, 3:55, 2020.
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. Evaluating style transfer for text. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 495-504, Minneapolis, Minnesota, 2019, Association for Computational Linguistics. doi: 10.18653/v I/N19-1049, URL https:/ aac.anthooog...or//11 1049.
Mohammad Norouzi, David Fleet, and Ruslan Salakhutdinov. Hamming distance metric learning. In Peter L. Bartlett, Fernando C. N. Percira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Sys- tems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Pro- ceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 1070-1078, 2012. URL https: //tttpp::/ppoooeeeee htt.://bbetttbbbbbbaattttttt html.
Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Car- los Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10867-10876. IEEE, 2020. doi: 10. 01100//CV222000200000000 www.loorg/11.. R01111/////P22220 2020 01088.
URL https:://////.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting the Association for Computational Linguistics, pp. 311-318, Philadelphia. Pennsylvania, USA, 2002, Association for Computational Linguistics. doi: 10.3115/1073083.1073 1073313.. URL https: / ac anthology.
org/PO2-1040.
14

---

Published as a conference paper at ICLR 2022
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pp. 2463-2473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10. (0.16653/1//D11-12555 URL https: htt///aaaaaatttooooeeeeeee o 1250.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann,, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, abs/2112.11446, 2021. URL
https: //rpsi//arxvv.. org/ abs 1///2111..1111 11446.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL www.aaiivvv....e. .or.g.oooooooooooooooo.ooo.oooo. abs / 11111.1110011 06732.
Sudha Rao and Joel Tetreault. Dear sir or madam, may introduce the GYAFC dataset: Corpus. benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pp. 129-140, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/N188 1012. URL /erLatttps//aaaaatthooo.g
org/N18 1711811111 101222
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Buree and Jason Wei. A recipe for arbitrary text style transfer with large language models. ArXiv preprint, abs/2109.03910, 2021. URL https:: arxiv org/ /abs/////////// 22099. 03910.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 1982-3992, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19
URL wwtph:tt///cc..... http:////ccaaathooooo.. wttg 94100.
Steven J. Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1179 119.. IEEE Computer Society, 2017. doi: 00.1109/CVPR.2017..33 URL https: /oi. org/ 10. 1109/CVPS 2017 131
Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh. Tex(SETT:: Few-shot text style extraction and tunable targeted restyling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume Long Papers), pp. 3786-30 -8000 Online, 2021. Association for Computational Linguistics. doi: 10. (0.18653/v/22221.ccm http//22222all-ooonn1...o 293.
URL https:///// / /acl anttootoohooo ogy. org/ 2021
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-666 1668. JMLR Workshop and Conference Proceedings. 2010.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 815-823, IEEE Computer Society, 2015. doi: Ww..L///CVVR..2115772865 URL https: /ooi org/ 1119//11R..2222 2015. 7298682.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non- parallel text by cross-aligmment. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.
15

---

Published as a conference paper at ICLR 2022
6830-6840 2017. URL 2d2
http://ag...eaoaooocccc///////// /
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. "transforming" delete, retrieve, generate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3269 -279, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/1//11-1133 URL https:// ac aalaanhhoogy. org/Di 9322..
Subhashini Venugopalan, Mareus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence video to text. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 4534-4542. IEEE Computer Society, 2015. doi: 10. 2..109//CCV.20155555 URL https: /tt.:/iooo/o/o/oo///o//.oo/ooo// 0r1//0.119/ ICCV. 2015.515.
Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. RiGender: A corpus for studying differential responses gender. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Evaluation (LREC 2018), Miyazaki,, Japan, 2016 8. European Language Resources Association (ELRA). URL https: /////acaanttt aww.anooooooyyy.o org/
1. 8 84445.
Ke Wang. Hang Hua, and Xiaojun Wan. Controllable unsupervised text attribute transfer via editing entangled latent representation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11034- 1044, 2019. URL https: /ttp:p::/prrooeee http///pccceeencceeeeeeeeee neurips cc cc/ppaper///1//eaeeeeeeeeeeeeeee 6balb 2a444444a..bbb800222244444000000 Abstzaaa........ html.
Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. Understanding abuse: typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pp. 78-84, Vancouver, BC, Canada, 2017. Association for Computational Lin- guisties. doi:
URL https: /tttttts//aaaaaaateeeetetaaeaaat /aw.aaaaannnnllooaaaaaaaaaaollla wwtpllooooooooooooooooeeeeoeeeeo prg/W17-3012
Ronald J Williams. Simple statistical wowdlieen--fllooiing algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei- Ying Ma. Unified visual-semante embeddings: Bridging vision and language with structured meaning representa- tions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA. June 16-20, 2019, pp. 6609-6618, Computer Vision Foundation IEEE, 2019, doi: 10.1 N0.1109/VPP.RC 000099900677..7 URL htpp:///ttttttttt////eeeeeee http///ppcccsss.eeeo thecvf com/ com/content__CVRR 2019 9 //ttm/////////mmmmm/eee Wwn_ll/WWunnniiiiie wtwp://wwwiiiineeeeeeeeeeeeeeeee fid_diisual_lliaaeeeaaaaaaaaaaae Visual- Semantic and Language With http:/itthhttuuurreeedeeeee Meanina__aaaaaaae_e 2019 paper html.
httpice_nnneedddng____eedde Vision
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. Paraphrasing for style. In Pro- ceedings of COLING 2012, pp. 2899-2914, Mumbai, India, 2012. The COLING 2012 Organizing Committee. URL httpt::t///aeeeee http::s:s///aaaaahlllloo/ooo//// wttp://rr////////
orgg/12-1177
Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence, 29(6):1091-1999 2007.
16

---

Published as a conference paper at ICLR 2022
A
APPENDIX
A.1 COMPARING WITH ANOTHER POLITICAL STANCE DATASET
Transferring political stance has also been studied in Voigt et al. (2018), where the author also collected a dataset for political stance TST. Comparing with their dataset, we found ours has following differences:
Their dataset is composed of comments on Facebook posts from members of the United States Senate and House, while our dataset consists of sentences from news articles written by professional reporters belonging to well-known media outlets (CNN, Fox News, etc.). The text in the dataset of Voigt et al. (2018) tends to be more casual and oral (e.g.. "congrats lee !"). Conversely, the text in dataset is generally more formal and well-formed.
Language Formality:
Their dataset does not contain the original post to which the comments are responding, which causes some potential ambiguity its political stance labels especially when the comment is generic (e.g.. "happy birthday jerry "") Most of the sentences in our dataset are complete statements of certain facts. We also conducted an extra distillation step (described in Appendix A in the supplementary document) to only keep the sentences that are salient to the article's ' stance label.
Completeness:
The comments in Voigt et al. (2018) mainly from 2017-2018, while our dataset collects nearly 10-year of articles from 6/1/2012 to 4/1/2021, which covers a wider range of societal issues and events. In the released version of our data we will add the time stamps to the sentences for customized filtering.
Coverage:
A.2 SENTENCE DISTILLATION FOR CONSTRUCTING POLITICAL STANCE DATASET
As mentioned in Section 3.1 of the paper, the articles collected from Allsides, manually aligned (based on their political idcology) by Allsides editors. We collected a total of 2,298 full-length news articles from opposite ideologies (i.e., liberal and conservative) grouped by events. To create a sentence-level TST dataset for political stance transfer, we split the articles into sentences and took their article-level ideologies (as assigned by the editors) as their sentence-level stance labels. However, we found that not all sentences explicitly exhibit the stance label inherited from their corresponding articles, we implemented an additional distillation step to select those sentences that are salient with respect to their assigned political stance.
already
Assuming we have an article with N sentences X {a1, I2, ..., EN} } and the title of the article Tuile we aim to find the subset of sentences Dasmil C X which all explicitly exhibit the assigned political stance. To better describe our approach, we first introduce two concepts: (1) stance saliency, and (2) distillation cost.
Stance Saliency. We compute the stance saliency v; for each sentence x; E X Diisiill which is measure of how well rj exhibits a certain political stance. Daisin is the set of sentences already picked our distilled set. We assume that the stance exhibited in Tute matches that of the article (as assigned by the editors), Thus, define stance saliency ii as the similarity between x, and the union of Tutte and D Dasain, which is:
1/ LCS(#;, Tuile Ddisun)
(5)
where LCS(() longest common sequence between the two input sequences. However, this metric is biased towards longer sentences as they can have more tokens matched with the title. We address this issue through the intriduction of a distilation cost, described below.
Distillation Cost. We assign distillation cost w; to each sentence x; in an article. The cost is simply the length of the sentence:
10: := |.pa-
17

---

Published as a conference paper at ICLR 2022
Note that for both concepts computation is based lower-cased and lemmatized sentence tokens. We use À to denote the ratio the sentences to be picked as distilled sentences in an article consisting of N sentences, and define Wmax = A Ei=u i=u Wi as the maximum distillation cost allowed.
Now that for each sentence we have stance saliency and distillation cost, we aim to choose set of sentences as the distillation which have the highest possible sum of saliency for which the sum of distillation costs not greater than Wmax. Given a function ((x;)) {1, 0} where corresponds to sentence z; being picked for the distilled set and O means otherwise, the objective of distillation can be expressed as:
max U oxx,), s.t. Σ w; o(x;) W mix
This is actually NP-complete combinatorial optimization problem, called the Knapsack problem, which we solve using saaamiccrooraammin (DP). In general. the distillation strategy aggressively picks sentences of high stance saliency while keeping the cost of distillation from exceeding the mask cost limitation W max- The procedure is demonstrated Algorithm below.
Algorithm 1: Sentence Distillation for Political Stance Dataset
Input: article title Trite. article sentences X, distillation ratio A.
> Initialization
Init current distillation set Distiil as Ø; Init current distillation cost Waistin with [imel:
> Multiple Rounds of Distillation
while W disui W max do
Compute V2 for each x; DDiaiisuu with Vitte and Dassin (by Equation 1);
Compute w/ for each x; (by Equation 2);
Find best distillation strategy {o(x;)) } via DP with us w; and Waisin (see Alg. 2):
Update Distillation Results Every Step
Update current distillation set Dassill Duisin U {x;} whose o(z,) 1;
Update current distillation cost Waisann in terms of updated Daisini
end
return distillation sentences set Daistnn'
Note that we initialize Wassin with the length title. At every iteration of the loop solve the DP problem with the limit set by current W Wasann which is a variable that estimates how many sentences are already the distillation set (no more than W ). The DP- algorithm used in Algorithm for finding the best distillation strategy is shown is Algorithm 2 below.
A.3 STATISTICS OUR ALL THREE DATASETS, AND EXPERIMENTAL SETUP
We present more statistical details of all three datasets in Table 7. All of our experiments were run on single RTX-2080 GPU, with batch size and 2/3/2 epochs for LaMer in the above three TST tasks. Following previous work, we choose style control accuracy (ACC), BLEU score as our main metrics. ACCs are judged by pre-trained RoBERTa (Liu et al., 2019) classifier with F1 = (0.97, 0.86, 0.93) using the training data of the three TST tasks. About the J-score we calculate in few-shot experiments, the models released by the author
A.4 ADDITIONAL DETAILS ABOUT TOKEN-LEVEL SCENE PRESERVATION
In this section, we provide additional details regarding how compute the token-level scene preservation dpsv. Figure 4 shows an example sentiment transfer task. For a given positive (source
style) sentence are and its corresponding negative (target style) parallels let we first extract their
The evaluation scripts, and pre-trained models be found here: http:/style.cs.umas.com
18

---

Published as a conference paper at ICLR 2022
Sentiment
Formality
Political Stance
Training Data (S-Emb, SAS)
positive
negative
formal
informal
liberal
conservative
# Sent Pairs
553,430
93.157
36,307
Scene Entities
132,136
777,839
145,731
133,779
194,032
272,071
Avg. Sent
7.32
7.58
11.52
11.05
19.39
24.75
Avg. Scene Ent. Sent
1.34
1.41
1.56
1.44
5.34
7.49
Avg. Scene Ent. serige (vs. RD)
0.95 (0.73)
0.49 (0.02)
0.96 (0.10)
Avg. SAS (vs. RD)
0.72 (0.55)
0.29 (0.01)
0.25 (0.01)
Human Reference
positive
negative
formal
informal
liberal
conservative
Sent Pairs
500
1020
525
Scene Entities
964
905
1,467
1,388
2,915
3,012
Avg. Sent
9.90
9,80
10.9
9.35
10.30
11.19
Avg. Scene Ent. / Sent
1.93
1.81
1.44
1.36
5.58
5.76
Avg. Scene Hint. serigt
1.53
0.55
1.78
Avg. SAS
0.73
0.29
0.30
Table 7: Data statistics for the three TST tasks in this work: Sentiment, Formality, and Political Stance Transfer. We compare the S-Emb. SAS aligned data (the upper part). with the auman-annotated reference file (the bottom part). Notice that after aligmment, the SAS of training data becomes much closer to that of human reference, which demonstrates we successfully align the non-parallel data into parallel form (comparable to the human reference). Among the three TST datasets, we find the political stance TST has the longest sentence length (see Avg. Sent Len), and more scene entities involved average (see Avg. # Scene Ent. / Sent). Human written references also reveal similar statistics (see the lower part of the table), which will potentially increase the difficulty for entity-matched transfer. We thus consider the political stance TST more challenging task than the other two.
Algorithm 2: DP-based Best Distillation Strategy Search
Input: stance saliency U. distill cost w;, and current cost limit Waisill- Record the number of stance saliency M = |,y]: Init DP-table 7[M + 1][ + 1] with all O;
for i 1.2, ........... M do
for j ..., ... W disnil do ir w:-1 then
r[OD] T[i - 1][5]:
Record distillation picking choice o(x;);
else
T m 1][s]
T - 1][; w;-1] D;-1);
Record distillation picking choice o(x;);
end end end
() backtracking via records;
return best distillation strategy (01x;))=1):
scene entities using the method by Wu al. (2019) (Figure 4 (a)), and based on these entities compute two distances donder and dgxist as sub-components of dpsv-
For donter. as shown in Figure 4 (b), we compare each the s entities with the s are entities, and measure how much uu needs to be edited to make the two sets of entities from opposite styles identical to each other. As mentioned in the paper, distance; whose set edits operation is: 1) insertion of a single token, 2) deletion of a single token, and 3) substitution of single token. Each of these operations has unit cost, so the Levenshtein
the Levenshtein distance as the edit
19

---

Published as a conference paper at ICLR 2022
Positive
dorder
dexitt
they an extensive menu and the boef
[menu, beef]
They. ment. beof]
was delicious.
(thyy, menu, beef]
11. 1.
dgsin
Negative
dosder
[0, 1. 1. 0
overpriced menu, and mediocre heef".
[nothing, menu]
[they, ment, nothing, beof
nothing great their menu.
(tey, menu, beef]
[I, 0,
Figure 4: (a) extraction of scene entities (underlined) from source sentence and parallel target sentences. Sentences with correspond gl and those with a correspond to s igl (i.e., expert and amateur demonstrations for imitation learning); (b) entity appearing order (donter- computed by
edit distance); and (c) entity existence (dExist computed by hamming distance).
distance is equal to the minimum number of operations required transform target style entities to source style entities.
As shown in Figure (c), we calculate another distance component, desias via hamming distance to emphasize the the difference between the sets of entities in source and target sentences. We first get the union of the entities from a pair of source and target sentences; based on the union set, we can obtain the corresponding binary vector representations of both style entities: means the entity exists certain style sentence, and 0 otherwise. Note the binary vectors for each of the sentences will be the same length (since it corresponds to the union of entities). Hamming distance measures the difference between source and target sentences on the shared entities set (this was shown to be an upper bound on the Levenshtein distance). Though there overlaps between this metric and donder we use this metric to emphasize the difference in scene entities.
A.5 MORE DETAILS ABOUT POLICY GRADIENT
We detail our policy gradient algorithm in Algorithm 3. Note that J is a safe-reward threshold. If the greedy policy can already reach this threshold the policy will not be updated. Empirically we set JjL to {0.8, 0.6, 0.4) for the three TST tasks (sentiment, formality, and political stance).
Algorithm 3: Imitating Learning of LaMer
Input: current policy ne. expert policy trajectory s 1gtt and amateur policy trajectory s 1st- Output: learned policy approaching to expert policy x²
> Get baseline via Greedy Decoding Bgready arg max LLLLM, (a 19::-1 ))E()
Compute jerody with ggreddy s 18t+ and 1st- by Eq. 3;
JR..
greedy
J then
No Optimization on Already Good Policies
Skip current step Imitation Learning;
Exploration by Sampling Decoding
while i max exploration steps do
(4)
Sample < sampling से LM(asleaa-DDeen..r
Compute J.
ssaple with Sample 1ett and s 1gl- by Eq. 3;
Optimize Policy by REINFORCE
Compute policy gradient as (0) - EE,, Vo log (J Sample -
Update policy with policy gradient by taking K steps SGD (via Adam):
return To
20

---

Published as a conference paper at ICLR 2022
We choose the REINFORCE algorithm (Williams, 1992) to optimize the current policy zg- One known issue the vanilla REINFORCE method is the instability of its optimization because of the high variance of the gradients. A common remedy is to add baseline function (i.c., parameterized value-network whose purpose is to estimate the unbiased future behavior of current policy (Ranzato al., 2016)). Rennie al. (2017) introduce better approach that estimates the baseline test time. They use the greedy decoding results as the baseline in their self-critic sequence training algorithm. We adopt their method to provide the baseline, and add an exploration step using sampling decoding. As shown in Algorithm& 3, by iteratively switching between these two modes, we calculate corresponding contrastive losses defined by Equation and take the difference of two modes losses as the advantage, feeding into the REINFORCE algorithm to optimize our policy тT. Finally, the output policy T can generate style transferred sequences resemble the expert demonstrations 1gt
A.6 ADDITIONAL. DETAILS ABOUT HUMAN EVALUATION
We provide more details about our human evaluation in this section. The results on sentiment TST shown in Table and those of formality TST are shown in Table 9.
Details About Participants and Compensation. Participants (N=212) were all from the United States and above 18 years old. Participants were required have HIT approval rate greater than 95%. The average age of the participants was 34.95 years-old (SD=9.90, Median=32). 141 of the participants (66.5%) were male, and 71 female (33.5%). Participants received 14.41 years of education on average (SD=4.46, Median=16). When asked to self-report their party affiliation, 52.4% participants self-reported as Democratic, 35.3% participants were self-reported Republicans, and 12.3% participants identified independent. Each participant was compensated USD per task. The average completion time per questionnaire was about 7.6 minutes as reported by the platform, which results in S7. 9/h/rr wage, above the regional and federal minimum wage.
Table 8: Human judgement results (on 7-point scale) of seven existing methods and LaMer on sentiment transfer task. We bold the highest average rating. SD: Standard Deviation.
Sentiment
Style
Content
Readability
Methods
Mean
SD
Mean
SD
Mean
SD
CAE (Shen al., 2017)
5.09
1.49
4.79
1.56
4.84
1.50
DelRetrGen (Li et al., 2018a)
4.94
1.62
4.73
1.63
5.05
1.39
Dual RL (Luo et al., 2019)
5,43
1.40
4.89
1.73
5.10
1.40
Style Transformer (Dai et al., 2019)
5.09
1.55
4.71
1.56
5.04
1.50
Deep Latent Seq (He al., 2020)
4.89
1.79
4.63
1.76
5.05
1.49
TSF-DelRetGen (Sudhakar et al., 2019)
4.66
1.35
4.73
5.30
1.13
STARP (Krishna et al., 2020)
5.50
1.20
4.65
1.40
5.33
1.20
Ours: LaMer
5.45
1.21
4.91
1.71
5.43
1.18
Table 9: Human judgement results (on 7-point scale) of seven existing methods and LaMer on formality transfer task. We bold the highest average rating. SD: Standard Deviation.
Formality
Style
Content
Readability
Methods
Mean
SD
Mean
SD
Mean
SD
CAE (Shen et al., 2017)
5.17
1.59
5.09
1.67
4.89
1.64
DelRetrGien (Li et al., 2018a)
5.19
1.57
4.91
1.65
4.86
1.62
Dual RT. (Luo al., 2019)
5.49
1.27
5.13
1.29
5.36
1.31
Style Transformer (Dai et 2019)
5.25
1.38
5.22
5.26
1.37
Deep Latent Seq (He et al., 2020)
5.22
1.52
5.30
1.46
5.17
1.47
TSF-DeIRetGen (Sudhakar ct al., 2019)
4.86
1.25
5.23
1.24
5.20
1.49
STARP (Krishna et al., 2020)
5.33
1.21
4.65
1.37
5.39
1.20
Ours: LaMer
5.50
1.23
5.45
1.29
5.39
1.21
Details About Human Evaluation Procedure. We evaluate the style transferred sentences gener- ated by baselines and our LaMer. The sentences picked from the test set generations for the three
21

---

Published as a conference paper at ICLR 2022
tasks. Each participant was asked to read two sets of sentences (one set containing source style sentence, corresponding human-writth written target style sentence as ground-trume and eight versions of target style sentences generated by seven existing methods and LaMer for comparison). After reading each set, they were asked to rate how well the generated sentences did in style control, readability, 7-point Likert scales. For example, for content preservation, they were asked "How much do you agree with the following statement (1- Strongly disagree to 7- Strongly machine-generated target style text preserves the context of the source style text well (i.e. they talk about the same thing).". Higher ratings correspond to better performance.
and content preservation
agree): The machine- generated
A.7 PERPLEXITY MEASURE OF LAMER AND OTHER BASELINES
In Table 10 we present the perplexity measure of LaMer and other baselines as a proxy for fluency. As be seen, certain methods, such CAE and Style Transformer, have very high perplexity in more challenging TST tasks (e.g.. Political Stance), potentially because the task requires longer generation and more entities involved in the sentences (see Table 7 for further discussions). However, it should also be noted that prior work have shown perplexity to be a poor measure of fluency (Krishna et al., 2020; Mir al., 2019; Lee et al., 2021).
Table 10: Perplexity measure of LaMer and other baselines. We use GPT2-large the judgement LM, and bold the lowest perplexity (the best), and underline the second best.
Sentiment
Formality
Political Stance
Existing Methods
PPL
PPL
PPL
Input Copy (ref.)
54.0
16.2
28.0
CAE
53.4
135.4
1869.0
DelRetrGen
18.5
30.3
86.0
Dual RL
17.0
31.9
141.2
Style Transformer
78.0
129.1
350.0
Deep Latent Seq
27.8
17.2
113.7
LM-based
TSF-DeRRerreen
52.1
24.5
102.3
IMaT
16.7
25.1
67.2
START
12.5
15.9
33.7
Ours: LaMer
w/. RD
13.9
13.6
32.3
w/. S-Emh.
11.4
13.5
47.6
w/. S-Emh. + SAS
13.3
11.7
33.5
22


Reviews:
[{"review_id": "bsCXyPscnvW", "reviewer": "Reviewer_8fDY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe paper tackles an important issue of text style transfer with novel ideas that aims to preserve content in an effective manner. \nThe paper is relatively well written with justifications and reasons for the choices. The authors could have given examples \nThe paper presents a simple idea to use reinforcement learning: imitation learning in particular  to better maintain the content between the two styles. \n\n**Weaknesses**\n\n1. The authors assume that  parallel sentences inherently exist between the source and the target style corpus. However, the existing unsupervised text style transfer methods do not make such assumptions. Since the entire basis of the paper is on such an assumption, providing examples of the inherent parallel sentences is crucial. The paper can be made stronger by  providing  examples (either manually picked or mined).\n\n2. **Scene graph alignment** - What are the scene entities? The authors mention that the use (Wu et al. 2019) paper on Unified Visual Semantic Embeddings. The reviewer skimmed through the other paper as well. However, which parser is used in the current paper?  Wu et al. use a syntactic parser to mine the objects in the sentence, adjectives, nouns, subjects of verbs etc and initialize an embedding for different entities and learn a joint embedding space. Here are my concerns with respect to this.\na) The authors need to be clear on the parser which is used to parse the sentence in this paper. Referring to Wu et al.’s paper without describing what the scene graph entities make it unclear on the reason to include this as a refinement step. \nb) If the above parser to extract objects is used, how is it important for style transfer? For example, the style transfer datasets used in the paper might not share any “visual” entity like a clock, plate etc. Instead they might share some abstract entities like the service provided, the cleanliness of the restaurant etc. \n\n     The reviewer understands that the sentences that talk about similar entities are important, the approach to obtain those entities is unconvincing. Seeing Figure S1 in the appendix, it is clear that the scene entities are words that are salient in the given style or words that carry content. Although the reviewer understands the idea, it would be a good idea to rewrite this to make it clear to the reader.\n\n\n3. **Evaluation Measures**: The used evaluation measures in the paper have known problems. For example, using BLEU does not measure semantic similarity with the input sentence and certain sentences with unnatural structure can still have low perplexity scores. Refer to Kalpesh Krishna et al 2020(https://arxiv.org/abs/2010.05700), which is included as a baseline method to compare against. The adoption of these measures would have made the paper stronger. Although the reviewer agrees that the majority of the text style transfer papers use the metrics used in this paper, the reviewer would appreciate if the authors could evaluate based on the paper from Krishna et al 2020 and add it in the Appendix or supplementary material\n\n4. **Evaluation**: The evaluation section should concentrate on the major contribution of the paper: maintaining the content between the two styles well. For example, with imitation learning we expect the BLEU metrics (and the i-PINC metric)  to do better because imitation learning aims to preserve the content in a better manner between the source and the target style which needs to be the highlight of the evaluation section. The other metrics might not be the focus of the paper and achieving high measures on them is an added bonus. The word “performance” is excessively used in the evaluation section. Instead, the specific measure that the others are talking about should be mentioned. \n\tAlso, Mean and Standard Deviation with multiple experiment runs are missing from the evaluation section.\n\n5. **Conclusion**: The conclusion is a summary of the main findings of the paper. It would be ideal if the authors can provide implications of the current work going forward. Can this be extended to other works that try to control the fine-grained syntax of the sentence etc would be a welcome addition to the conclusion \n\n6. **New Dataset**: More details on the new dataset introduced in the paper is needed. Why is the new dataset more challenging compared to the other two datasets that are mentioned in the paper?\n\n**Minor Typos, Grammatical Errors and Other Readability Issues**\n\n1. There is an extra space on Page 2 Para 2 Line 2 after Figure till (b)\n2. The authors introduce the term “scene graphs” in the introduction on Page 2 Para 2 and Line 4. Readers who are not aware of the computer vision literature might be left wondering on how scene graphs are relevant for Text Style Transfer \n3. The rows showing ablations in Table 1 are confusing “w/” is read as without or with. The authors could instead use “+” or “-” symbol to indicate with or without\n4. Table  1 highlight boxes need to be explained to the reader.  Please mention that you are highlighting the best performing method for that metric. Also some of highlight boxes are missing in the Sentiment dataset\n", "summary_of_the_paper": "The paper proposes to transform the popular unsupervised style transfer to a semi supervised task. First the paper proposes simple methods to mine parallel sentences from two unlabelled corpora. Further, they use the mined parallel sentences, in a MLE based training (sequence to sequence framework using BART) and refine it using Contrastive Learning based imitation learning. The paper achieves improvements on preserving the content between the transferred sentences. In addition, the authors also propose a new dataset for political style transfer in a semi-automated way which is a welcome addition to the paper.\n", "summary_of_the_review": "1. Overall, the paper is easy to read and the ideas to improve content preservation between the source and the target sentence is simple and effective. The ablation studies and parameter sensitivity studies are well performed.  \n2. The reviewer would require more details on the entities that are preserved between the sentences, examples on the inherent parallel sentence between the two corpora in the introduction.\n3.  Problematic evaluation measures used in the literature need to be revised \n4. The evaluation section needs to focus on the main contribution of the paper. Splitting Section 3.2 into more paragraphs might be a starter. \n5. The benchmark table needs to have multiple runs with mean and standard deviation reports.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TjxswEYQYj8", "reviewer": "Reviewer_itTT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\n1. I liked the ideas in the paper. As far as I know, scene graphs have not been used previously in style transfer and are a nice way to ensure semantic preservation. I believe parallel data mining has been done before in unsupervised machine translation [1] and pretraining [2] (papers I suggest the authors cite), but not for style transfer. The part of the approach using imitation learning to refine the pseudo parallel data is also new and effective as confirmed by the ablation studies.\n\n2. The authors collect a new dataset for political stance rewriting. This dataset is fairly large (56K pairs) and the task sounds harder than sentiment transfer. This is a valuable contribution and it could be a useful benchmark for future style transfer research. The authors have also carefully curated the dataset to remove hate speech, and used a filtering method (Appendix B) to ensure mined sentences are faithful to the attribute.\n\n3. The authors show improvements in three different text rewriting tasks over several baselines on both automatic metrics and human evaluations. Additionally, the authors provide ablation studies and analyze the choice of their data mining hyperparameters.\n\n*Weaknesses*\n\n1. While I liked the ideas in the paper, I thought the main technical contribution of mining psuedo parallel data from an unpaired corpus is fundamentally limited. First, there's no guarantee the unpaired corpus will have a similar content distribution across attributes (which is required to retrieve roughly parallel sentences). The formality / political stance transfer benchmarks tested were originally parallel datasets, which ensures a shared content distribution. Second, you might need a very large unpaired corpus to ensure this mining is successful, which may not be practical in lower resource settings (this is why recent work is moving towards few-shot setups for style transfer [4, 5]). I suggest the authors try style transfer tasks where the content distributions will be very different across attributes, such as Shakespeare <---> Tweets from [6].\n\n2. Not a very big concern (since your method outperforms baselines on all three metrics), but it will be nice to see an aggregated overall score for style transfer performance (especially on human evaluation). The three style transfer metrics (accuracy, similarity, fluency) often conflict with each other [7], so an overall score is generally more informative. You could do this for both automatic and human evaluations. For human evaluations, since you use 7-point Likert scales you could follow the approach in [8] who calculate the number of instances which got a Likert score of (let's say 5 or more) on all three metrics *simultaneously*. A more general alternative is presented in [4, 6] which can be used for both automatic and human evaluations.\n\n**Minor**\n\n1. The acronym LaMer is not specific to the contributions of this paper (pseudo parallel data, scene graphs, imitation learning). It is also a slang for \"dull\" in informal settings. I suggest changing this acronym to something else.\n\n2. In 2.4, you could alternatively use unlikelihood training [3] on the amateur policies.\n\n3. I suggest reducing focus on the details of the RL methodology and adding more details on the new benchmark dataset proposed in the main body of the paper.\n\n4. Question about human evaluation ---> did you randomly shuffle the order of generations from different systems? That way you would avoid annotator bias from their previous annotations (\"the last generation is always best\")\n\n5. Several links seem to be broken, especially those to the Appendix.\n\n6. I suggest the authors add more qualitative outputs from their system into the Appendix part of the PDF, rather than in an external attachment.\n\n[1] - https://aclanthology.org/P19-1178  \n[2] - https://arxiv.org/abs/2006.15020  \n[3] - https://arxiv.org/abs/1908.04319  \n[4] - https://arxiv.org/abs/2110.07385  \n[5] - https://aclanthology.org/2021.acl-long.293  \n[6] - https://arxiv.org/abs/2010.05700  \n[7] - https://arxiv.org/abs/1910.03747  \n[8] - https://aclanthology.org/N18-1169", "summary_of_the_paper": "Style transfer is a text generation task where sentences need to be rewritten in such a way that a particular target attribute is introduced (like formality), but at the same time the content is approximately preserved. Style transfer is usually researched in unsupervised settings --- researchers assume no access to parallel data of sentences differing only in the target attribute. However, most prior works assume access to a corpus of unpaired sentences in each of the target attributes.\n\nThe main technical idea in this paper is searching for (mining) roughly parallel pairs in the unpaired corpora, to create a \"pseudo-parallel dataset\". The authors use two methods to extract pairs - (1) semantic similarity based on RoBERTa embeddings; (2) similarity of a scene-graph built by using an off-the-shelf parser. These pseudo-parallel data is used to fine-tune a large pretrained language model (BART). To improve performance, the authors utilize imitation learning via REINFORCE. The key idea is checking whether the current greedy sample is closer by a margin to the closest mined parallel sentence (positive example), compared to other mined parallel sentences (negative examples). If not, sampling is used to explore the space and the sampled sequence is rewarded / penalized according to the increase / decrease in margin using REINFORCE.\n\nThe authors test their approach on sentiment transfer, formality transfer and politcal stance transfer (which is a new dataset introduced by them), and note improvements over several popular baselines on automatic metrics and human evaluations. The authors also analyze their choice of hyperparameters and conducted several ablation studies to justify their design decisions.", "summary_of_the_review": "Overall, I'm in favour of acceptance since the paper has several interesting new ideas, a large new dataset for political stance transfer, and automatic + human evaluation comparing the benefits of their approach against baselines. The weakness #1 prevents me from giving the next higher score (of 8).\n\n-----------\n\n**After author response**: I'm pleasantly surprised by the few-shot style transfer results, and commend the authors for their rigorous rebuttal to all reviewers. I think the paper will be stronger with results on a dataset with a different content distribution between the two styles (irrespective of size, such as Shakespeare <---> Tweets). My overall assessment is more like a 7 or 7.5, which means I'll increase my score to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "45KJB7LsgwN", "reviewer": "Reviewer_53GA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The scene graph similarity used to select pseudo parallel pairs is novel and makes sense to me.\n2. The reinforced imitation learning shows better performance compared with baseline of MLE loss for one-to-many mapping.\n3. The empirical results establish the effectiveness of this method.\n\nWeaknesses:\n1. This method is based on the assumption that there are parallel pairs in the original corpora of two styles so that sentences of the same content and different styles can be found out, which limits the use of the method to cases where abundant corpora are existing and there are parallel sentences there in the corpora. In those cases where only small unsupervised corpora are there and there are no sentences pairs sharing the same content, this method would not work.\n2. In the current used three datasets, I would like to know how good are those paired sentences selected out. This kind of quantitative human evaluation of constructed pseudo-parallel corpora is very important to establish the effectiveness of this work, which cannot be lacking.\n3. What is the BLEU score? Is it self-BLEU or ref-BLEU? If ref-BLEU, is it calculated on one human reference or four? I know at least for Sentiment transfer and formality transfer, the test sets contain four references. \n4. In Table 1, according to the Tabl1 of the original DualRL paper: https://arxiv.org/pdf/1905.10060.pdf, the BLEU scores can be 55.2 and 44.9 for Yelp and GYAFC datasets, while the scores reported in this work are significantly lower. I am not sure whether the references are different here. Please explain this gap.\n5. In Table 1, the performance of Ours w/. RD also looks very high, especially for the Political Stance dataset, the difference between w/. RD and w/. S-Emb are very small for BLEU and PPL of w/. RD is even lower, which contradicts to my intuition. Using a pseudo-parallel corpora constructed by random selection should not produce any good performance, should it? This kind of RD baseline even outperforms many strong SOTA baselines, for example of DUAL RL, which is ironic. \n6. In Table 1, it shows that the SAS score is only helpful for one datasets out of three, then why is it so? Why is it only useful in the formality case?\n7. This work is inspired by this work: https://aclanthology.org/D19-1306/, however, it has never been compared with it. ", "summary_of_the_paper": "This work has proposed a new method to textual style transfer, which is based on the assumption that there exist some pseudo-parallal sentences pairs between two styles. It first construct synthetic parallel corpora by using two similarity measures: semantic similarity based on larger LMs and scene graph similarity. Then it trains the generation model via reinforced imitation learning. ", "summary_of_the_review": "This work has merits, but its weaknesses are also salient as mentioned above. I am looking forward to author responses for addressing my concerns, or I vote for rejecting this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-u3H36OX1TC", "reviewer": "Reviewer_7agN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- A step-wise way of training text style transfer models on non-parallel datasets\n- A new dataset for style transfer, collected from the political domain. \n\nWeakness\n\n- First, I am not sure about the novelty of the proposed method. Given the ideas of MLE and RL/IL (specifically, the INFORCE algorithm used in this work) has been used in text generation for a while, I am not sure how much novelty here by simply combining them together. \n- Second, the writing of this work needs to be improved.\n  - In section 2.2, the Scene Alignment Score seems to be an important component of the proposed first step, however, I don't think this work ever (1) explain what is a scene graph and (2) justify the validity of using the Scene Alignment Score for alignment. The numbers in Table 1 may provide some empirical evidence for the second question, but it is not sufficient. \n  - I was confused by the terms used in section 2.4, including \"Reinforced Imitation Learning\", \"Reinforced Policy Gradient\". To be specific, based on the description, I didn't understand the difference between reinforced imitation learning and imitation learning. In addition, I am also not sure what reinforced policy gradient is. It looks like the description in section 2.4 mixes many terms together, without an appropriate explanation. \n  - In Algorithm 1, what is $J_{IL}^{safe}$? \n  - About table 1, I am not sure how the highlights were selected in this table. Apparently, not all the highlights are the best results. ", "summary_of_the_paper": "This paper proposes an approach to addressing text style transfer with non-parallel data. The basic idea is to follow three steps: (1) for a given source sentence, mine some nearly parallel sentences from the target domain; (2) perform an MLE learning; and (3) augmented with an imitation learning. \n\nThe proposed method was evaluated on three text style transfer tasks: (1) sentiment transfer; (2) formality transfer; and (3) a new task called political stance transfer. ", "summary_of_the_review": "My major concern of this work is the technical novelty. In addition, the writings of this paper also needs to be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-_gBOl2XPd0", "reviewer": "Reviewer_Hwf9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper uses a novel combination of existing approaches to achieving competitive results.\n\n- The code provided with the paper is clean, structured, and can be useful for developing future models for the same task.\n\n- The paper itself is well written and structured.\n\nWeaknesses and questions:\n\n- The authors claim \"existing methods learn a mapping without considering the self-parallelism, ... they tend to learn the mapping between source and target style by randomly mapping sentence pairs.\" However, no proofs or references for such behavior were given. On the contrary, most of the works mentioned as related have no such problematic behavior by design.\n\n- The first step of the proposed approach (matching sentences from the source and target corpora) implies that both corpora have a similar distribution. However, it is not the case for some definitions of \"styles.\" This assumption possibly limits the method's applicability, but it wasn't addressed in the motivation or discussion sections.\n\n- Moreover, the token-level scene preservation scheme implies that scene entities couldn't be style markers themselves. This assumption also limits possible applicability until the opposite is proven.\n\n- Again, the part of the loss is based on the difference in the orders of scene entities, but in practice, it can be an important part of the style. Consider the example from the 'political stance' dataset presented with the paper:\nStyle1: Mulvaney tapped to lead Trump s budget office.\nStyle2: Trump picks debt warrior Mulvaney to lead White House budget office.\n\n- There is a known inevitable trade-off between content-preservation and style-transfer metrics. Thus, to show the improvement of the new model, one needs to demonstrate Pareto-improvement both of them (in this case, this means the improvement both in terms of ACC and BLEU metrics). However, the reported results show the TSF-DelRetGenLM model is better in terms of one of these two metrics for both sentiment and formality datasets. Thus, the evaluation shows the comparability of the model with the baselines on two standard datasets (and the superiority only on the novel proposed dataset). At the same time, the authors claim \"SOTA results according to many metrics on the three TST tasks,\" which doesn't sound fair. \n\n\n\n\n", "summary_of_the_paper": "The paper addresses a text-style transfer task based on non-parallel datasets.\n\nThe authors propose a three-step approach:\n- First, to match each source sentence with several sentences from the target style dataset using sentence embedding similarity and scene graph matching.\n- Next, to fine-tune a seq2seq network (BART) in an NMT-like setup with several references.\n- Finally, to use imitation learning to enforce the loss contrast between the best target candidate and all others.\n\nThe authors propose a new dataset for the TST task, called \"political stance transfer,\" and use it for evaluation among the previously known datasets. The results of such a combined approach are competitive with previous works.", "summary_of_the_review": "I like the proposed approach and the clarity of the paper and code.\nI'm slightly concerned with the paper's novelty since it presents a combination of previously known methods, but still, I think it can be useful for the community.\nThe main concern is somehow ambiguous motivation and definition of the scope of applicability of the proposed approach (check the questions above). \nI believe this should be clarified.\n\nMy score for the paper is \"marginally above the acceptance threshold\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for unsupervised text style transfer by assuming there exist some pseudo-parallal sentences pairs in the data. The method thus first mines and constructs a synthetic parallel corpus with certain similarity metrics, and then trains the model via imitation learning. Reviewers have found the method is sound and the empiricial results are decent. The assumption on pseudo-parallal pairs would limited the application of the methods in other settings where the source/target text distributions are very different. The authors have added discussion on this limitation during rebuttal.
Decision: Accept (Poster)