SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhengyuan Liu, Nancy F. Chen
Institute for Infocomm Research, A*STAR, Singapore
{liu_zhengyuan, nfychen}@i2r.a-star.edu.sg
Text style transfer is an important task in controllable language generation. Supervised approaches have pushed performance improvement on style-oriented rewriting such as formality conversion. However, challenges remain due to the scarcity of large-scale parallel data in many domains. While unsupervised approaches do not rely on annotated sentence pairs for each style, they are often plagued with instability issues such as mode collapse or quality degradation. To take advantage of both supervised and unsupervised paradigms and tackle the challenges, in this work, we propose a semi-supervised framework for text style transfer. First, the learning process is bootstrapped with supervision guided by automatically constructed pseudo-parallel pairs using lexical and semantic-based methods. Then the model learns from unlabeled data via reinforcement rewards. Specifically, we propose to improve the sequence-to-sequence policy gradient via stepwise reward optimization, providing fine-grained learning signals and stabilizing the reinforced learning process. Experimental results show that the proposed approach achieves state-of-the-art performance on multiple datasets, and produces effective generation with as minimal as  $10\%$  of training data.
Text style transfer is a task in natural language generation, which aims to automatically control certain attributes during sentence paraphrasing, such as formality, sentiment, and humor (Rao and Tetreault, 2018; Li et al., 2018). Style transfer has many practical applications, such as altering emotions of spoken utterances, removing biases in transcripts, and conveying politeness in messages (Hovy, 1987). The key for a successful rewrite is to preserve the semantic content of the source sentence, while transforming it to a particular target style without sacrificing fluency and grammatical accuracy.
Therefore, the performance of style transfer models is commonly assessed on both style accuracy and content preservation. When large-scale annotated sentence pairs are available, training neural sequence-to-sequence models via supervised learning shows impressive generation quality (Rao and Tetreault, 2018; Lai et al., 2021). However, in many use cases, it is unfeasible to adopt supervised approaches because parallel samples are unavailable. To address data insufficiency bottlenecks, various unsupervised approaches have been proposed for text style transfer, including learning disentangled representations of style and content (Shen et al., 2017) and adopting pairwise backtranslation (Prabhumoye et al., 2018). Recently, reinforcement learning (RL) is introduced to develop unsupervised models such that rewards of content preservation and style conversion are used to optimize sequence generation (Luo et al., 2019; Gong et al., 2019). However, RL-based methods are often challenging to train in practice. For instance, the rewards have high variance during early stages when learning from scratch, which affects the training stability; and they cannot provide fine-grained learning signals as traditional token-level maximum likelihood estimation, since they are often calculated on the entire generated sequence (de Masson d'Autume et al., 2019). As a result, models are prone to mode collapse and often fail to produce acceptable generations in reality.
Herein, we propose a semi-supervised framework for text style transfer, and optimize it on training stability and signal fineness. Our semi-supervised model uses a small amount of parallel data for supervised learning, and gets further improvement by learning from a large amount of unlabeled data. In contrast to prior work that often relies on human-annotated parallel pairs like (Chawla and Yang, 2020), the approach we propose bootstraps the training process with automatically constructed pseudo parallel data. Two pseudo pair
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2633 - 2648 July 10-15, 2022 ©2022 Association for Computational Linguistics
matching methods are investigated: a lexical-based strategy, which is straightforward by calculating the token-level overlap; and a semantic-based strategy, which uses semantic similarity as criteria and would have better general potential.
Furthermore, to obtain fine-grained signals for the RL-based sequence-to-sequence training process, we propose a stepwise reward re-weighting strategy. This is inspired by the observation that the style transfer weights are not uniform across tokens/spans in the source sentence: some tokens weigh more during attribute-guided text style transfer (Li et al., 2018). Therefore, instead of using the reward (e.g., style strength scores) calculated from the entire generated sentence (Luo et al., 2019; Lai et al., 2021), we use the token-level reward. Specifically, we extract attribute-related attentive scores from a pre-trained style discriminator, obtain a stepwise reward by re-weighting the sequence-level score, and utilize it as a fine-grained signal for policy gradient back-propagation.
We evaluate the proposed framework that incorporates both supervision and reward-based learning on three style transfer corpora (Section 4). Experiments show that our model achieves state-of-the-art performance. Particularly, the proposed model can produce reasonable generations with only  $10\%$  training data on the Yelp and Amazon corpora, and it also outperforms the supervised baselines when applying on the well-annotated GYAFC dataset.
Neural Text Style Transfer The aim of text style transfer is to automatically convert text to a certain style while preserving the content (McDonald and Pustejovsky, 1985; Hovy, 1987). It has many applications, like persona-based dialogue generation (Niu and Bansal, 2018). Recently, neural sequence-to-sequence architectures becomes popular for this task. When parallel data are available, supervised training with cross-entropy loss is typically applied (Rao and Tetreault, 2018). However, annotated data are hard to obtain in many use cases, thus learning from non-parallel corpora has become an active research area. There are two approaches: (1) Disentangling style and content by learning a distinct representation for each element. For example, variational autoencoders are first used to transform a sentence into a low-dimension hidden state. Then the attribute-related latent representation is extracted to guide the decoder for target style generation (Shen
et al., 2017; Fu et al., 2018; John et al., 2019); (2) Back translation, which uses cyclic reconstruction to improve content preservation (Zhang et al., 2018; Prabhumoye et al., 2018; Lample et al., 2019; Luo et al., 2019). For model optimization, some studies focus on applying reinforcement learning (RL), which defines a reward from a style classifier or a reward from back-translation to enhance style strength and content preservation (Gong et al., 2019; Luo et al., 2019; Wu et al., 2019; Sancheti et al., 2020). Recently, large-scale pre-trained language models are introduced to improve generation quality (Radford et al., 2019), and have been incorporated in both semi-supervised (Chawla and Yang, 2020) and supervised approaches (Lai et al., 2021). In this work, we use the BART (Lewis et al., 2020) as our language model backbone.
Pseudo Data Augmentation To tackle the data scarcity challenge in text style transfer, one solution is to build pseudo pairs from massive non-parallel data. Zhang et al. (2020b) proposed several augmentation methods for pre-training a Transformer-based model and fine-tuning on human annotations. Wang et al. (2019) proposed using harness-rule-based pre-processing, and joint training of bidirectional transfer and auto-encoder with two auxiliary losses (Wang et al., 2020). Jin et al. (2019) and Nikolov and Hahnloser (2019) constructed the pseudo corpora by iteratively matching via cosine similarity of sentence embeddings and hierarchical alignment. In this work, we use pseudo data as weak-supervision to bootstrap the training process, and further combine it with RL-based learning.
Attribute Salience Assessment In template-based and prototype editing methods for text style transfer, attribute marker detection is used to label the salient words and spans (Li et al., 2018). Aside from n-gram statistical features, neural attention-based methods train attribute-related classifiers, and consider words with attention weights higher than average as markers (Bahdanau et al., 2015; Xu et al., 2018; Sudhakar et al., 2019). Zhou et al. (2020) use the attribute salient scores as one of the model prediction output. To the best of our knowledge, we are the first to employ token-level attribute salience scores for reward re-weighting on policy gradient for sequence generation, and prior work only focuses on using attribute markers for text manipulation such as token replacement and template construction (Niu and Bansal, 2018).
Figure 1: Overview of the proposed framework. Text samples in two different styles are in yellow and in blue. The sequence-to-sequence model is shared by style transfer and cyclic generation. The MLE loss, reconstruction reward, and style reward flows are in blue, yellow, and green arrow lines, respectively. See Algorithm 1 for training process.
Define  $S$  as the source style and  $T$  as the target style (e.g.,  $S = \text{negative}$ ,  $T = \text{positive}$ ). Let  $\mathcal{D}_S$  and  $\mathcal{D}_T$  be the two datasets which are comprised of sentences in each style respectively. The style transfer system, denoted as a text encoding-decoding model  $G$ , is to generate sentences in the target style. The goal is formulated to maximize  $P(\boldsymbol{y}|\boldsymbol{x}; \theta_G)$  where  $\theta_G$  are the model parameters. In our setting, we make the rewriting bidirectional, i.e. it can be used to transfer source style to target style and verse versa. In this case, an additional input  $\boldsymbol{c} \in \{S, T\}$  is fed to  $G$  specifying the style to which the sentence is to be converted. Hence, the objective is to maximize  $P(\boldsymbol{y}|\boldsymbol{x}, \boldsymbol{c}; \theta_G)$ .
The overview of our proposed semi-supervised framework is shown in Figure 1. Given the non-parallel datasets  $\mathcal{D}_S$  and  $\mathcal{D}_T$ , we use lexical or semantic features for pseudo parallel pair matching. The training process consists of two stages: (1) the generator model  $G$  is trained on the pseudo parallel samples, where cross-entropy loss over the target sentence tokens is used to optimize generated output probabilities, i.e. the bootstrapping step; (2) we incorporate reconstruction and style rewards to enhance attribute rewriting and content preservation, where reinforcement learning is used to optimize the generation, i.e. the reward-based learning. Moreover, the second stage can use pseudo parallel pairs as well as the non-parallel samples.
3.2 Pseudo Parallel Data Construction
To build the pseudo parallel data for bootstrapping, we investigate lexical similarity and semantic similarity for sentence matching.
Lexical Similarity In text style transfer, rewriting is often accomplished by changing a few words or phrases that are indicative of a particular attribute in the source sentence, namely attribute markers, while leaving the rest largely unaltered (Li et al., 2018). For example, "Moving past the shape, they were dry and truly tasteless," a sentence with a negative sentiment style, can be transferred to a positive style by changing or replacing sentiment-specific words "dry" and "tasteless", while keeping other words intact. This intuition has inspired the template-based and editing-based rewriting approaches (Li et al., 2018). Here we employ it for the lexical feature extraction. First, from unaligned corpora of two styled subsets (e.g., positive, negative), we identify attribute markers by sorting phrases that occur with far higher frequency within one attribute than the other (e.g., "worst" and "very disappointed" are negative markers). Second, for each sentence in the two subsets, we remove those markers, and regard the remaining words as its content-preserved spans. Then we match the content-preserved spans of style  $S$  to those of style  $T$  with the smallest Levenshtein editing distance (see examples shown in Table 1).
Semantic Similarity While the lexical features are straightforward and computationally-efficient, it may not generalize well in some tasks like formality conversion due to the ubiquitous span paraphrasing. Therefore, in this paper, we introduce semantic features for the pseudo data construction. While samples in different styles stand in different polarities, they are expected to be similar in the content-level semantic space. More specifically, for a sample  $i$  in style  $S$ , we match it to the closest sentence in style  $T$  in a semantic space. We use an unsupervised sentence representation
Source Sentence: if there were a way to put no stars, i would!  
Lexical Match: i'd give it more stars if i could.
Semantic Match: love love love, if i could give you num stars i would.
Source Sentence: the manager sat us at our table, and she seemed very angry.
Lexical Match: the manager and employees are very nice. Semantic Match: the manager alice herself came by our table and greeted us as well.
Source Sentence: furthermore, i would rather drive _num_ minutes more to concord to race there.
Lexical Match: furthermore, they have a nice bar that goes both indoor and outdoor.
Semantic Match: i drive_num Minutes to get here and it is definitely worth it!
Table 1: Pseudo parallel sentence pairs extracted from Yelp sentiment transfer dataset. Source sentences are from the negative polarity set, and are matched to sentences from the positive set.
model with contrastive learning (Gao et al., 2021), which achieves comparable performance to the supervised sentence embedding models, and calculate cosine similarity to measure the distance. As shown in Table 1, the pseudo parallel data are similar at the semantic level, and they can be used as weak-supervision samples.
3.3 Learning with Supervision
With the pseudo parallel data, we can conduct supervised learning with token-level maximum likelihood estimation (MLE). In our framework, we use a sequence-to-sequence neural network. Since the large-scale pre-trained language models boost the performance of various downstream tasks, we use BART (Lewis et al., 2020) as the language backbone, which is a denoising autoencoder with strong language generation capability. Given a source sentence  $\mathbf{x}$  and a reference sentence  $\mathbf{y}$ , the cross-entropy loss is calculated between the decoder's output and the reference sentence:
$$
L _ {\mathrm {M L E}} = - \Sigma_ {i} \log \left(p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {y} _ {1: i - 1}, \boldsymbol {x}, \boldsymbol {c}; \theta_ {G}\right)\right) \tag {1}
$$
Moreover, to avoid the generation becoming overfitting to the pseudo parallel data, we add the label smoothing on the cross-entropy loss (Muller et al., 2019), with the smoothing weight  $\lambda = 0.15$ .
3.4 Learning with Rewards
Upon the supervised learning from the pseudo parallel data, the model can be further improved by
unsupervised learning from the massive unlabeled data. For the unsupervised stage, we adopt reinforcement learning, and use two rewards to enhance style rewriting and content preservation.
Reconstruction Reward Back translation has proved effective to improve content preservation, we feed the transferred sentence to model  $G$  for the backward rewriting, and calculate reconstruction reward on the cyclic generation. Here we measure the reward based on BLEU (Papineni et al., 2002) score as in (Sancheti et al., 2020) to foster content preservation, and adopt policy gradient (Sutton et al., 1999) with Self-Critical Sequence Training to reduce the variance (Rennie et al., 2017):
$$
R _ {c y c l i c} = \operatorname {s c o r e} \left(G \left(\boldsymbol {y} ^ {\prime}\right), \boldsymbol {x}\right) - \operatorname {s c o r e} \left(G (\hat {\boldsymbol {y}}), \boldsymbol {x}\right) (2)
$$
where  $\pmb{x}$  is the backward target,  $G(\hat{\pmb{y}})$  is the back-translated output from greedy decoding generation  $\hat{\pmb{y}}$ , and  $G(\pmb{y}')$  is the back-translated from sampling-based generation  $\pmb{y}'$  over a multi-nominal distribution. Noted that the score function can also be ROUGE and language model perplexity. The former is more suitable for summarization tasks; the latter needs additional computation.
Style Classification Reward Aside from content preservation, we use a style strength reward to optimize the model. We train a Transformer model for the binary style classification, and use it to evaluate how well the transferred sentence  $\mathbf{y}^{\prime}$  matches the target style. The style reward is  $R_{\text{style}}$  defined as the classification score:
$$
p \left(s _ {\text {s t y l e}} \mid \boldsymbol {y} ^ {\prime}\right) = \operatorname {s o f t m a x} \left(\operatorname {s t y l e C L S} \left(\boldsymbol {y} ^ {\prime}, \phi\right)\right) \tag {3}
$$
where styleCLS denotes the style classifier,  $\phi$  are the parameters of the classifier, which are fixed during the training of the generation framework.  $y^\prime$  is the generated sentence by sampling from the multi-nominal distribution at each step. Then, the reward-based learning is conducted via Policy Gradient (Sutton et al., 1999) back-propagation:
$$
R = \lambda_ {\text {c y c l i c}} R _ {\text {c y c l i c}} + \lambda_ {\text {s t y l e}} \left(R _ {\text {s t y l e}} - \gamma\right) \tag {4}
$$
$$
\nabla_ {\theta_ {G}} J = E \left[ R \cdot \nabla_ {\theta_ {G}} \log \left(P \left(\boldsymbol {y} ^ {\prime} \mid \boldsymbol {x}, \boldsymbol {c}; \theta_ {G}\right)\right) \right] \tag {5}
$$
where  $R$  is the sum of cyclic and style reward,  $\pmb{y}^{\prime}$  is the generated sentence by sampling from the multinominal distribution at each step,  $\theta_{G}$  are trainable parameters of the generator, the weight ratio  $\lambda$  are added on cyclic and style reward separately, and  $\gamma$  is a style reward penalty (see Table 9). The overall objectives for  $\theta_{G}$  are the loss of the base model (Eq. 1) and the policy gradient of RL rewards (Eq. 5).
$^{1}$ Additionally, we observed that in some corpora like Amazon (Li et al., 2018), there are a number of samples labeled with incorrect style due to data noise, and the semantic approach is sensitive on this issue. Therefore, we use a style classifier to filter out the incorrectly clustered samples.
Figure 2: Our proposed stepwise reward re-weighting.
Stepwise Reward Re-weighting When applying reinforcement learning algorithms on sequence-to-sequence training, it is difficult for models to conduct end-to-end back-propagation due to the discrete nature of text. One of the common solutions is adopting policy gradient optimization (Sutton et al., 1999), where the rewards are generally calculated on the whole output sequence. Since all generated tokens obtain the same reward value, this coarse-grained signal is suboptimal for learning performance and stability (de Masson d'Autume et al., 2019). For instance, when positive sentiment is targeted, the output sentence "I dislike this movie!" will obtain a negative reward of style strength if its gold reference is "I love this movie!". In this context, the word "dislike" should be punished more than the others in the sentence, but with sequence-level reward all words receive the same penalty. To address this drawback, we propose a solution by granulating the sequence-level reward with token-level salience scores, namely, stepwise re-weighting.
To re-weight the coarse-grained reward, we use the normalized attentive scores from the style classification model as the token-level attribute-salient scores. For the Transformer architecture, it is shown that heavily attended tokens correlate strongly with tokens that are indicative of the target style (Hewitt and Manning, 2019; Vig and Belinkov, 2019). Since the softmax linear layer is used over the attention stack of the first token  $\langle \mathrm{s}\rangle$  in a 'RoBERTa-base' model, the attention weights of other input tokens that correspond to  $\langle \mathrm{s}\rangle$  are of special interest in identifying significant sentence tokens. We inspect the attentions computed by the Transformer with 12 multi-head layers, and empirically observed that the attention weights of top layers correlate strongly with salient tokens (see the visualization in Appendix Figure 4). Given the attention matrix  $A_{i}$  in the  $i$ -th multi-head layer,  $a_{i}^{j}$  represents the attention vector of the first token (e.g.,  $\langle \mathrm{s}\rangle$ , "[CLS]") from the  $j$ -th attention head, which is normalized across all tokens. We
Corpus	Train	Valid	Test
Yelp (Sentiment-Positive)	270K	2,000	500
Yelp (Sentiment-Negative)	180K	2,000	500
Amazon (Sentiment-Positive)	277K	985	500
Amazon (Sentiment-Negative)	278K	1,015	500
GYAFC E&M (Formality-Paired)	52.6K	2,877	1,416
GYAFC F&R (Formality-Paired)	51.9K	2,788	1,432
Table 2: Statistics of the style transfer datasets. The GYAFC Entertainment&Music (E&M) and Family&Relationships (F&R) are comprised of paired samples. For Yelp and Amazon, only their test sets include human-written parallel references.
max-pool  $A_{i}$  over all attention heads to form  $a_{i}$ , which represents the maximum extent to which each token was attended to by any head, and further max-pool the weights across the top-2 layers as the final stepwise attribute-salient scores (see layer selection in Section 5.2), which are in the range of  $(0,1)$ . Then sequence-level rewards are expanded to the token length  $n$ , and re-weighted by the stepwise scores (see Figure 2 and Algorithm 1), and the policy gradient is formulated as following:
$$
\nabla_ {\theta_ {G}} J = E \left[ \frac {1}{n} \sum_ {t = 1} ^ {n} R _ {t} ^ {\prime} \cdot \nabla_ {\theta_ {G}} \log \left(P \left(\boldsymbol {y} _ {t} ^ {\prime} \mid \boldsymbol {y} _ {1: t - 1} ^ {\prime}, \boldsymbol {x}, \boldsymbol {c}; \theta_ {G}\right)\right) \right] \tag {6}
$$
For extensive experiments, in this paper, we select three representative text style transfer corpora: Yelp (business reviews), Amazon (product reviews), and Grammarly's Yahoo Answers Formality Corpus (GYAFC) (Li et al., 2018; Rao and Tetreault, 2018). The training, validation, and test split are the same as previous work (Luo et al., 2019; Chawla and Yang, 2020), and their task types and statistics are shown in Table 2. In the non-annotated corpora Yelp and Amazon, human-written references are only available for the test set. Therefore, to build the pseudo parallel data described in the previous section, we filter out the sentence pairs with lexical or semantic similarity lower than a threshold, and remove sentences that are shorter than 5 words. The pseudo parallel set is used for the bootstrapping training (Section 3.3), and the rest samples are used for the unsupervised stage (Section 3.4).
The framework is implemented with Pytorch and Hugging Face Transformers<sup>2</sup>. The 'BART-base'