SlowGuess/NAACL_2021_OCR huggingface dataset:
STYLEPTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Yiwei Lyu\*, Paul Pu Liang\*, Hai Pham\*, Eduard Hovy\*, Barnabás Póczos\*, Ruslan Salakhutdinov\*, Louis-Philippe Morency\* Machine Learning Department, Carnegie Mellon University \*Language Technologies Institute, Carnegie Mellon University {ylyul, pliang, htpham} @cs.cmu.edu
https://github.com/lvyiweil/StylePTB/
Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, STYLEPTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on STYLEPTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, STYLEPTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation.
At the heart of interactive AI systems lies the element of communication as a channel to convey intentions using different stylistic attributes. Research in human-AI interaction has focused on building dialog systems (Celikyilmaz et al., 2018), virtual assistants (Cooper et al., 2004), and intelligent agents (Kim et al., 2013; Liang et al., 2020a; Pittermann et al., 2010) that can communicate their intentions with specific styles for different situations, target audiences, and environments (Lample
Figure 1: STYLEPTB provides a large-scale resource to study fine-grained compositional style transfer. The styles provided in STYLEPTB (in green) span lexical, syntax, semantic, and thematic aspects (DiMarco and Hirst, 1993) which can be composed to form high-level style transfers as commonly studied in existing benchmarks (e.g. Yelp for sentiment (Shen et al., 2017) and GYAFC for formality (Rao and Tetreault, 2018)).
et al., 2019; Li et al., 2018). For example, expressing the same facts using either formal or informal styles can be more suitable for certain target audiences (Rao and Tetreault, 2018).
What is a style in natural languages? Existing style transfer benchmarks primarily focus on individual high-level stylistic changes across sentiment (Shen et al., 2017), formality (Rao and Tetreault, 2018), politeness (Madaan et al., 2020), and writing styles (Jhamtani et al., 2017). Figure 1 provides some motivating examples to show that the high-level style transfers as commonly studied in existing benchmarks (e.g. Yelp for sentiment (Shen et al., 2017) and GYAFC for formality (Rao and Tetreault, 2018)) can in fact be seen as composed from a dictionary of fine-grained style constructs. This alternative way of studying styles brings additional flexibility that enables fine-grained control with the possibility to compose a broader space of styles spanning tense, sentence structure, phrase emphasis, and information contained in the sentence. However, the missing link is a benchmark dataset that offers this type of fine
\* authors contributed equally
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2116-2138 June 6-11, 2021. ©2021 Association for Computational Linguistics
grained style constructs, with the controllability to compose these stylistic transfers.
To fill this gap, we leverage research in linguistics to study formulations of styles across 4 representational categories: lexical, syntax, semantics, and theematics, that span the fundamental atomic transfers that text can undergo (McDonald and Pustejovsky, 1985; DiMarco and Hirst, 1993). Using these insights, we introduce a large-scale benchmark with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning the most atomic lexical, syntactic, semantic, and thematic style constructs, as well as (2) compositions of multiple transfers which model how fine-grained style constructs compose to form more complex, high-level transfers. Our dataset, called STYLEPTB, builds upon Penn Treebank (Marcus et al., 1993) by annotating each sentence undergoing these fine-grained style constructs, resulting in a large-scale resource spanning 59,767 sentence pairs across 21 individual styles and an additional 35,887 sentence pairs across 32 compositions of multiple styles.
STYLEPTB allows us to study the performance of state-of-the-art style transfer models when faced with the new challenge of fine-grained style transfer. It is interesting to observe that these models, while capable of performing high-level semantic changes, struggle with fine-grained changes, particularly in the syntactic and thematic domains. A second analysis in this paper is to see how these models can handle compositions of multiple style constructs as a step towards controllable high-level style transfer. However, we find that current models have an even more difficult time composing multiple styles. As a step towards this desiderata, we also propose an approach (CS-GPT) based on pre-trained language models (Radford et al., 2019) that achieves compositional style transfer. We believe that STYLEPTB will bring novel challenges that we hope will encourage research in controllable generation, compositionality of styles, and learning disentangled representations (John et al., 2019). From a broader perspective, we conclude with the observation that controllable style transfer models trained on STYLEPTB can help mitigate social biases in pre-trained language models.
Several lines of research have aimed to formalize styles in natural languages through computational and linguistic perspectives (DiMarco and Hirst, 1993). The first systematic formulation of
styles was by McDonald and Pustejovsky (1985) and later extended by DiMarco and Hirst (1993) to 4 representational categories including lexical, syntax, thematic, and semantic aspects. Following this, there has been some early efforts applying stylistic analysis into dialog generation (Hovy, 1987), machine translation (DiMarco, 1994), and text generation (Gatt and Krahmer, 2018). We take advantage of this prior work when formalizing our new STYLEPTB dataset.
Current benchmarks for style transfer focus on high-level style definitions such as transfer of sentiment (Shen et al., 2017; Lample et al., 2019; Li et al., 2018; Wu et al., 2019), politeness (Madaan et al., 2020), formality (Rao and Tetreault, 2018; Liu et al., 2020; Krishna et al., 2020), writing styles (Jhamtani et al., 2017; Syed et al., 2020; Jin et al., 2020) and some other styles (Kang and Hovy, 2019). However, these only focus on only high-level styles, unlike STYLEPTB.
Computational models for style transfer span statistical NLP methods (Hovy, 1987; Xu et al., 2012), neural generative models (Prabhumoye et al., 2018; Lample et al., 2019; He et al., 2020), and Retrieve-and-Edit approaches (Li et al., 2018; Hashimoto et al., 2018; Guu et al., 2018; Sudhakar et al., 2019; Madaan et al., 2020). These approaches work for a predefined set of styles but are unable to generalize to compositions of styles.
Evaluating style transfer is difficult due to the diversity of plausible transferred sentences. In addition to automatic scores such as BLEU, perplexity, or binary classification accuracy of style transfer (Hu et al., 2017; Lample et al., 2019; He et al., 2020), other automatic metrics (Fu et al., 2018; Mir et al., 2019) and human evaluation are also commonly used (Li et al., 2018; Shen et al., 2017).
3 Fine-Grained Style Constructs
As a step towards enabling fine-grained control with the possibility to compose a broader space of styles, we first define style constructs at fine-grained levels spanning lexical, syntactic, semantic, and thematic aspects. When selecting these style constructs, we have 2 goals in mind: (1) they should be representative of the four aspects (lexical, syntactic, semantic, thematic) following the formal categorizations in DiMarco and Hirst (1993), and (2) the transfers should be consistent (i.e. well-defined such that if multiple annotators are asked to modify the same sentence, the results will be similar). With these goals in mind, we summarize the
Table 1: Examples of each of the 21 defined style constructs across lexical, syntactic, semantic, and thematic aspects found in STYLEPTB. The original phrase is in cyan and the corresponding target phrase is in magenta. Note that some thematic and semantic transfers require additional information, highlighted in red.
following 21 chosen fine-grained style constructs spanning 4 categories and also provide detailed examples in Table 1.
Lexical transfers are those at fine-grained lexicon levels (i.e. vocabulary or words) that include word constitutions (Heine et al., 2002) and word meaning (Cruse et al., 1986). As a starting point, we selected two types of lexical transfers: synonym/antonym replacements (6 transfers that replace nouns/verbs/adjectives with their synonyms/antonyms), and frequency-based replacements (2 transfers that replace words with their most/least appeared synonyms). The synonym/antonym resources are taken from Wordnet (Fellbaum, 2012).
Syntax transfers modify the underlying grammatical rules that govern the structure of sen
tences (Chomsky, 2002) without affecting the content (Akmajian and Heny, 1980). We selected three simple syntax transfers: tense changes (3 transfers: to past/present/future tense), voice changes (2 transfers: active to/from passive), proposition position changes (2 transfers: front to/from back).
Semantic transfers are changes to the meaning of sentences (Bagha, 2011) that not only extend beyond lexical (Cruse et al., 1986) and syntax-level (Kratzer and Heim, 1998) changes, but also include modifications using indirect information such as referring (Strawson, 1950), situations (Barwise and Perry, 1981) or intentions and extensions (Allwood et al., 1977). As a starting point, we defined two simple types of semantic transfers: (1) Info removal: 3 transfers on different deletions: word-level (removing adjectives and adverbs), phrase
(a) Distribution of sentence lengths.
(b) Distribution of parts of speech.
(c) Top 30 most frequent tokens, excluding stop-words.
Figure 2: Statistics: (a) the distribution of sentence lengths, (b) count of word tokens by part-of-speech, and (c) the top 30 most frequent tokens. STYLEPTB exhibits diversity in sentence form and style transfer annotations.
level (removing propositions), and substatement level (removing entire substatements) that represent referring and situations, as well as (2) Info addition: 1 transformation that adds a given piece of information regarding a particular phrase in the current sentence representing extension.
Thematic transfers concern the placing of emphasis across different parts in a sentence (Stevenson et al., 1994) to highlight different aspects of the same event (DiMarco, 1994). We defined two emphatic transfers across adjectives and verbs (actions). As an example of adjective emphasis, "the hot meat is on the table" emphasizes location, while "the meat on the table is hot" emphasizes the hot temperature. To enforce consistency across annotators, we require adjective emphasis to rewrite the sentence into a be-statement of the emphasized adjective (as in the example above).
Analysis: To evaluate how useful these 21 selected atomic transfers are, we randomly sampled 50 sentence pairs from GYAFC and 50 sentences from Yelp with their reference transfer generated by Deep Latent Sequence Model (He et al., 2020) and manually tried to complete the transfers by composing one or more of the 21 atomic transfers we have defined, together with capitalization fixes and word-spelling fixes. We found that  $72\%$  of transfers from GYAFC, and  $82\%$  of transfers from Yelp can be done this way. Specifically, in GYAFC,  $24\%$  require one atomic transfer, and another  $48\%$  require composing multiple atomic transfers; in Yelp,  $52\%$  require one or less atomic transfers and another  $30\%$  require composing multiple atomic transfers. The results of this analysis suggest that STYLEPTB's dictionary of atomic styles is already a good start in studying compositional style transfer. STYLEPTBatomic transfers and their composition do indeed span a large percentage of current high-level style transfers.
4 The STYLEPTB Dataset
Using these selected 21 style constructs, we now illustrate the steps towards collecting and annotating parallel sentences across style transfers.
4.1 Dataset Preprocessing
We use Penn Treebank (PTB) (Marcus et al., 1993) as our source of sentences. Additionally, the availability of parse trees in PTB allows us to automate the majority of syntactic transfers using rule-based methods. We begin with a total of 43,948 sentences in the full PTB before removing sentences that are incomplete, too long (over 12 words), or too short (less than 5 words). This leaves 7,719 sentences (see Figure 2 for statistics and Appendix A.1 for full details).
4.2 Generating transferred sentences
We give a brief overview of the data annotation process (see Appendix A.3 for full details).
Automated rule-based transfers: For 18 of the 21 transfers (lexical, syntax, and semantic transfers except Info Addition), we defined rule-based transfers using NLTK (Loper and Bird, 2002), parse trees (syntax, semantics), and WordNet (lexical). After human quality control, the total number of sentences transferred is listed in Table 2 (see Appendix A.2 for more details on automated generation and Appendix A.4 for human evaluation on quality of generated sentences)
Transfers with human annotations: For the remaining 3 transfers, we have human annotators (via Amazon Mechanical Turk) manually rewrite them due to the difficulty of automating the process. See Appendix A.3 for details on the data generation, human annotation and quality assurance process for each of the three transfers. After annotations and quality control, we obtained 696 rewritten sentences for adjective emphasis, 1201 rewritten sentences for verb emphasis, and 2114
Category	Transfer	Number
Lexical	Noun synonym replacement	5948
Noun antonym replacement	2227
Verb synonym replacement	2574
Verb antonym replacement	1284
ADJ synonym replacement	434
ADJ antonym replacement	1146
Most frequent synonym replacement	4722
Least frequent synonym replacement	7112
Syntax	To future tense	7272
To present tense	4365
To past tense	4422
Active ↔ passive	2808
PP front ↔ back	467
Semantics (information deletion)	ADJ or ADV removal	4863
PP removal	4767
Substatement removal	1345
Information Addition	2114
Thematic	Verb/action emphasis	1201
Adj emphasis	696
Table 2: STYLEPTB is a large-scale resource spanning 59,767 sentence pairs across 21 individual styles.
Difficulty	Transfer	Hamming ↓
Easy	ADJ or ADV removal	1.531
To Present tense	2.318
To Past tense	2.447
To Future tense	3.341
Medium	Information addition	3.729
PP removal	4.079
PP back to front	5.429
Substatement removal	5.625
PP front to back	6.235
Hard	Active to passive	8.147
Passive to active	8.817
Adjective emphasis	8.846
Verb/Action emphasis	11.614
valid sentence-information pairs with their transferred sentence with information added.
4.3 Relative Difficulty of Transfers
Lexical transfers can be done by replacing individual words and is simple to evaluate. To evaluate the difficulty of the remaining 13 syntax, semantic, and thematic transfers, we calculated the token-level (i.e. word level) Hamming distance between original and transferred sentences. Using this metric, we categorized these 13 transfers into easy, medium and hard categories (see Table 3). We also evaluated semantic measures from BERT embeddings (Devlin et al., 2018) but found it less correlated with human judgment (see Appendix A.5).
Figure 3: Example of generating sentence pairs that compose tense and voice changes. Starting from an original sentence (green box), we sequentially apply parse tree transfers (blue arrows) to obtain multiple transferred sentences (yellow box), yielding multiple parallel pairs (yellow arrows). We use transfer tokens  $(\Delta_{1}, \Delta_{2})$  to track changes (see Section 5 for details).
Table 3: Average token-level Hamming distance between original and transferred sentences for all syntax, semantics and thematic transfers.
No Voice Change (0)	Active To Passive (1)	Passive To Active (2)
No Tense Change (0)	2808	2808	2808
To Future Tense (1)	5294	2647	2647
To Past Tense (2)	2077	656	1421
To Present Tense (3)	3304	1536	1768
Table 4: Number of sentence pairs for each composition of tense change and voice change in the generated compositional dataset.
4.4 Compositional Transfers
To allow for compositionality, we also generated compositional data that includes parallel pairs of sentences linked by multiple sequential transfers. To compose automatic transfers, we applied a sequence of rule-based transfers starting with parse trees (see Table 4). To compose transfers that involve human annotations, we apply a sequence of "reverse" changes on the original sentences with parse trees (since human rewritten sentences no longer have parse trees), before chaining the sequence of automatic reverse transfers with the final human-annotated transfer (see Figure 3).
5 A Model for Compositional Transfer
We extend the pre-trained GPT2 language model (Radford et al., 2019) for parallel style transfer by giving it designated style transfer tokens as input in addition to the source sentence. For example, for each individual binary style  $s_i$ , we define a style transfer token  $\Delta_i \in \{0,1,2\}$  where  $\Delta_i = 0$  represents keeping  $s_i$  unchanged,  $\Delta_i = 1$  represents a change from  $s_i = 0$  to  $s_i = 1$ , and vice versa for  $\Delta_i = 2$ . We likewise extend the definition of  $\Delta_i$  for styles taking more than 2 values.
Given a parallel (source, target) pair  $(s,t)$ , we define the appropriate transfer token  $\Delta \in \{0,1,2\}$
Easy Transfers	Baseline Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
To Future Tense	GPT2	0.895	0.852	0.813	0.778	0.540	0.899	7.709
SEQ2SEQ	0.527	0.368	0.261	0.188	0.173	0.531	1.525
RETRIEVEEDIT	0.899	0.854	0.815	0.778	0.531	0.901	7.731
HUMAN	0.954	0.915	0.884	0.855	0.636	0.964	9.174
ADJ or ADV Removal	GPT2	0.647	0.508	0.394	0.308	0.313	0.652	3.259
SEQ2SEQ	0.450	0.274	0.172	0.112	0.140	0.469	1.171
RETRIEVEEDIT	0.897	0.841	0.786	0.731	0.511	0.919	7.461
HUMAN	0.933	0.894	0.870	0.847	0.591	0.965	8.924
Medium Transfers	Baseline Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
Substatement Removal	GPT2	0.430	0.332	0.247	0.176	0.250	0.588	3.090
SEQ2SEQ	0.317	0.192	0.110	0.001	0.100	0.368	1.041
RETRIEVEEDIT	0.706	0.678	0.647	0.607	0.405	0.767	6.183
HUMAN	0.731	0.720	0.705	0.685	0.607	0.788	7.691
Information Addition	GPT2	0.479	0.305	0.189	0.121	0.207	0.475	1.359
SEQ2SEQ	0.345	0.180	0.094	0.053	0.098	0.335	0.632
RETRIEVEEDIT	0.493	0.396	0.328	0.275	0.284	0.603	3.401
HUMAN	0.846	0.762	0.690	0.624	0.521	0.892	6.863
Hard Transfers	Baseline Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
Active To Passive	GPT2	0.476	0.329	0.238	0.189	0.216	0.464	1.820
SEQ2SEQ	0.373	0.220	0.141	0.103	0.131	0.345	0.845
RETRIEVEEDIT	0.681	0.598	0.503	0.427	0.383	0.663	4.535
HUMAN	0.931	0.881	0.835	0.795	0.587	0.905	8.603
Adjective Emphasis	GPT2	0.263	0.079	0.028	0.000	0.112	0.188	0.386
SEQ2SEQ	0.187	0.058	0.018	0.000	0.059	0.179	0.141
RETRIEVEEDIT	0.387	0.276	0.211	0.164	0.193	0.369	1.679
HUMAN	0.834	0.753	0.679	0.611	0.522	0.811	6.796
Verb/Action Emphasis	GPT2	0.309	0.170	0.095	0.041	0.140	0.292	0.593
SEQ2SEQ	0.289	0.127	0.066	0.038	0.098	0.275	0.300
RETRIEVEEDIT	0.416	0.284	0.209	0.148	0.223	0.423	1.778
HUMAN	0.649	0.569	0.493	0.421	0.433	0.693	5.668
Table 5: Evaluation results on easy (top), medium (middle), and hard (bottom) transfers. Info Addition and thematic transfers are especially difficult for current models.
Figure 4: CS-GPT uses multiple transfer tokens  $\Delta_{i} \in \{0,1,2\}$  to enable compositional style transfer across multiple styles in our model STYLEPTB.
and train using maximum likelihood estimation to predict every word  $t_j$ , for  $j = 1,2,\ldots,T$ , in the target sentence given the source and  $\Delta$ :
$$
\theta^ {*} = \underset {\theta} {\arg \max } \mathbb {E} _ {(s, t) \sim \mathcal {D}} \left[ \sum_ {j = 1} ^ {T} \log p _ {\theta} \left(t _ {j}; s, \Delta\right) \right], \tag {1}
$$
where  $\theta$  denotes the pre-trained GPT2 parameters
and  $\theta^{*}$  denotes the parameters after fine-tuning on STYLEPTB. Note that we also train the model to reconstruct the same source sentence again when setting  $\Delta = 0$  (no style change), which we found to help bridge the domain shift between data used to pre-train GPT2 and sentences in STYLEPTB.
As a step towards compositionality, we also train with (source, target) pairs that undergo multiple atomic style transfers as provided in STYLEPTB, resulting in multiple style transfer tokens  $\Delta_{i}$  being activated at the same time. We call the resulting model CS-GPT (Compositional Style GPT) and show its architecture in Figure 4. Learning separate representations for each  $\Delta_{i}$  results in disentangled style variables that can then be composed as desired. Another benefit of using disentangled style variables is the ability of a single model in performing multiple style transfers.
We test the performance of current style transfer models on STYLEPTB. Anonymized data and code is included in the supplementary, and we present extra details and results in Appendix B and C.
6.1 Datasets and Metrics
We use STYLEPTB and evaluate on the 13 non-lexical transfers (since lexical changes works best with fixed word substitutions). Please refer to Appendix B.1 for dataset preprocessing details. Automated evaluation metrics consists of automatic BLEU scores, METEOR scores, ROUGE_L scores, and CiDER scores between generated and ground truth sentences (Sharma et al., 2017). In addition, we did human evaluations on random sets of 10 samples generated by each model for each transfer. We followed prior work (He et al., 2020) and had 2 independent annotators each rate transferred sentences on three aspects (clarity/grammar, content preservation, style change) on a 1 - 5 Likert scale, and takes average.
6.2 Baseline Models
We evaluate the following baselines commonly used in style transfer. Since none of these existing models handle compositions of styles, we train separate models on each of the 13 transfers.
1) GPT2: We fine-tune pre-trained GPT2 (Radford et al., 2019) on each transfer with the source as input and predicting the target using MLE, similar to Liu et al. (2020); Syed et al. (2020).
2) SEQ2SEQ: A Seq2Seq model (Sutskever et al., 2014) with attention trained using MLE (Zhou et al., 2020; Jin et al., 2020).
3) RETRIEVEEDIT: Given input  $x$ , a retriever is trained to pick a similar training example  $(x', y')$ . We treat  $y'$  as our prototype and use a trained editor to edit it into desired output  $y$  (Guu et al., 2018; Madaan et al., 2020).
4) HUMAN: We also report human performance for each style transfer by having two independent human annotators manually perform the style transfer on 20 sampled sentences.
6.3 Results and Observations
We evaluate these 3 baseline models on the style transfers in STYLEPTB and show results in Table 5. We make the following observations: