SlowGuess/NAACL_2021_OCR huggingface dataset:
Yixin Liu, Graham Neubig, John Wieting
{yixinl2,gneubig,jwieting}@cs.cmu.edu
In most cases, the lack of parallel corpora makes it impossible to directly train supervised models for the text style transfer task. In this paper, we explore training algorithms that instead optimize reward functions that explicitly consider different aspects of the styletransferred outputs. In particular, we leverage semantic similarity metrics originally used for fine-tuning neural machine translation models to explicitly assess the preservation of content between system outputs and input texts. We also investigate the potential weaknesses of the existing automatic metrics and propose efficient strategies of using these metrics for training. The experimental results show that our model provides significant gains in both automatic and human evaluation over strong baselines, indicating the effectiveness of our proposed methods and training strategies.
Text style transfer aims to convert an input text into another generated text with a different style but the same basic semantics as the input. One major challenge in this setting is that many style transfer tasks lack parallel corpora, since the absence of human references makes it impossible to train the text style transfer models using maximum likelihood estimation (MLE), which aims to maximize the predicted likelihood of the references. As a result, some of the earliest work (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018) on unsupervised text style transfer proposed training algorithms that are still based on MLE by formulating the style transfer models as auto-encoders optimized with reconstruction loss. Specifically, during training the model is tasked to generate a style-agnostic encoding and reconstruct the input text based on this encoding with style-specific embeddings or decoders. During inference, the model aims to transfer the source
text style using the target style information. While these methods have seen empirical success, they face the inherent difficulty of coming up with a style-agnostic but content-preserving encoding – this is a non-trivial task and failure at this first step will diminish style transfer accuracy and content preservation of the final output.
Another line of work (Xu et al., 2018; Pang and Gimpel, 2019; Luo et al., 2019) proposes training algorithms based on rewards related to the automatic evaluation metrics, which can assess the model performance more directly during training. This approach is conceptually similar to training algorithms that optimize models using rewards related to the corresponding evaluation metrics for other NLP tasks, such as machine translation (Shen et al., 2016; Wieting et al., 2019a) or text summarization (Paulus et al., 2018; Li et al., 2019). As for unsupervised style transfer, the widely used automatic metrics mainly attend to three desiderata: (1) style transfer accuracy – the generated sentence must be in the target style, commonly measured by the accuracy of a style classifier applied to the transferred text, (2) fluency – the generated text must be grammatically correct and natural, commonly measured by the perplexity of a language model and (3) content preservation – the semantics need to be preserved between the source and target, commonly measured by the BLEU score between the system outputs and source texts. Since these automatic metrics only require the system outputs and source texts, they can be used as rewards for training. Moreover, the two lines of approaches can be used together, and previous work (Yang et al., 2018; John et al., 2019; Madaan et al., 2020) proposed methods which use the auto-encoders as the backbone augmented with task-specific rewards. In particular, the style transfer accuracy reward is used by most of the recent work.
However, reward-based training algorithms still have their limitations, and in this paper we aim
1Code and data are available at: https://github. com/yixinL7/Direct-Style-Transfer
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4262-4273 June 6-11, 2021. ©2021 Association for Computational Linguistics
to identify and address the bottlenecks of these methods. Specifically, we focus on two problems: (1) the difficulty of designing an efficient reward for content preservation, (2) the lack of robustness of the existing automatic evaluation metrics.
Content preservation is more difficult to measure compared to style transfer accuracy and fluency because it needs to consider the overlap in the semantics between the source text and system outputs. While using BLEU score between the source text and system output would be a direct solution (Xu et al., 2018), this approach has an inherent limitation in that  $n$ -gram based metrics such as BLEU are sensitive to lexical differences and will penalize modifications that are necessary for transferring text style. In fact, previous work has proposed various different proxy rewards for content preservation. One of the most popular methods is the cycle-consistency loss (Luo et al., 2019; Dai et al., 2019; Pang and Gimpel, 2019), which introduces a round-trip generation process, where the model generates an output in the target style, and the ability of a reconstruction model to re-generate the original text is used as a proxy for content preservation. While this method is more tolerant to lexical differences, the correlation between the reconstruction loss and content preservation can be weak.
Therefore, we aim to design a reward for content preservation which can directly assess the semantic similarity between the system outputs and input texts. Specifically, we note that models of semantic similarity are widely studied (Wieting et al., 2016; Sharma et al., 2017; Pagliardini et al., 2018; Zhang* et al., 2020), and we can leverage these methods to directly calculate the similarity between the system outputs and input texts. This renders our method applicable for even unsupervised settings where no human references are available.
Another key challenge for reward-based training algorithms is that the existing automatic evaluation metrics are not well-correlated with human evaluation (Li et al., 2018). It poses general risks to the work in this field with respect to model training and evaluation since these metrics are widely used. An important observation we made from our experiments is that style transfer models can exploit the weaknesses of the automatic metrics. They do this by making minimal changes to the input texts which are enough to trick the classifier used for style transfer accuracy while achieving high content preservation and fluency scores due to the high
lexical similarity with the input texts. Upon identifying this risk, we re-visit and propose several strategies that serve as auxiliary regularization on the style transfer models, effectively mitigating the problem discussed above.
We empirically show that our proposed reward functions can provide significant gains in both automatic and human evaluation over strong baselines from the literature. In addition, the problems we identify with existing automatic evaluation metrics suggest that the automatic metrics need to be used with caution either for model training or evaluation in order to make it truthfully reflect human evaluation.
Data for unsupervised text style transfer can be defined as
$$
D = \{(x ^ {(1)}, s ^ {(1)}), \dots , (x ^ {(i)}, s ^ {(i)}), \dots , (x ^ {(n)}, s ^ {(n)}) \},
$$
where  $x^{(i)}$  denotes the text and  $s^{(i)}$  denotes the corresponding style label. The objective of the task is to generate (via a generator  $g$ ) the output with the target style conditioned on  $s$  while preserving most of the semantics of the source  $x$ . In other words,  $\hat{x} = g(x,s)$  should have style  $s$  and the semantics of  $x$ . We define the style as a binary attribute such that  $s \in \{0,1\}$ , however, it can be easily extended to a multi-class setting.
For our generator, we fine-tune a large-scale language model GPT-2 (Radford et al., 2019). GPT-2 is pre-trained on large corpora and can be finetuned to generate fluent and coherent outputs for a variety of language generation tasks (Wolf et al., 2019). Since GPT-2 is a unidirectional language model, we reformulate the conditional generation task as a sequence completion task. Namely, as input to the generator, we concatenate the original sentence with a special token which indicates the target style. The sequence following the style token is our output.
2.3 Reward Functions
We use four reward functions to control the quality of the system outputs. The quality of the outputs is assessed in three ways: style transfer accuracy, content preservation, and fluency. We attend to each of these factors with their respective rewards.
Figure 1: SIM Loss v.s. Cycle-Consistency Loss
Here we denote the input text  $x$  having style  $s$  by  $x_{s}$ , and denote the output by  $\tilde{x}_s$ , i.e.,  $\tilde{x}_s = g(x_s,1 - s)$ .
Rewards for Style Transfer Accuracy We use a style classifier to provide the supervision signal to the generator with respect to the style transfer accuracy. The min-max game between the generator  $g$  and the classifier  $f_{cls}$  is:
$$
\begin{array}{l} \min _ {\theta_ {g}} \max _ {\theta_ {f _ {c l s}}} \mathbb {E} _ {x _ {s}} [ \log (1 - f _ {c l s} (g (x _ {s}, 1 - s), 1 - s)) ] \\ + \mathbb {E} _ {x _ {s}} [ \log f _ {\text {c l s}} (x _ {s}, s) + \log (1 - f _ {\text {c l s}} (x _ {s}, 1 - s)) ]. \tag {1} \\ \end{array}
$$
The style transfer accuracy reward for the generator is the log-likelihood of the output being labeled as the target style:
$$
r _ {c l s} (\tilde {x} _ {s}) = \log (f _ {c l s} (\tilde {x} _ {s}, 1 - s)). \qquad (2)
$$
Following prior work, we use the CNN-based classifier (Kim, 2014)  $f_{cls}$ , which takes both the sentence and the style label as input and its objective is to predict the likelihood of the sentence being coherent to the given style.
Rewards for Content Preservation To ensure that the system outputs still preserve the basic semantics of the source sentences, we use the pretrained SIM model introduced in Wieting et al. (2019b,a) to measure the semantic similarity between the source sentences and system outputs. The SIM score for a sentence pair is the cosine similarity of its sentence representations. These representations are constructed by averaging sub-word embeddings. Compared to the cycle-consistency loss (Luo et al., 2019; Dai et al., 2019; Pang and
Gimpel, 2019), our method is more direct since it doesn't require a second-pass generation. It also has advantages over  $n$ -gram based metrics like BLEU (Papineni et al., 2002) since it is more robust to lexical changes and can provide smoother rewards.
In Wieting et al. (2019a), SIM is augmented with a length penalty to help control the length of the generated text. We use their entire model, SIMILE, as the content preservation reward,
$$
r _ {s i m} \left(\tilde {x} _ {s}\right) = \mathrm {L P} \left(x _ {s}, \tilde {x} _ {s}\right) ^ {\alpha} \mathrm {S I M} \left(x _ {s}, \tilde {x} _ {s}\right), \tag {3}
$$
$$
\mathrm {L P} (r, h) = e ^ {1 - \frac {\operatorname* {m i n} (| r | , | h |)}{\operatorname* {m a x} (| r | , | h |)}}, \tag {4}
$$
and  $\alpha$  is an exponential term to control the weight of the length penalty, which is set to 0.25.
We also use the cycle-consistency loss  $L_{cyc}$  to bootstrap the training:
$$
L _ {c y c} \left(\theta_ {g}\right) = \mathbb {E} _ {x _ {s}} \left[ - \log \left(p _ {g} \left(x _ {s} \mid g \left(x _ {s}, 1 - s\right), s\right)\right) \right]. \tag {5}
$$
Here,  $p_g$  is the likelihood assigned by the generator  $g$ . This introduces two generation passes, i.e.,  $\tilde{x}_s = g(x, 1 - s)$  and  $\bar{x}_s = g(\tilde{x}_s, s)$  while SIM reward only requires one generation pass, as illustrated in Fig. 1.
Rewards for Fluency Style transfer accuracy rewards and content preservation rewards do not have a significant effect on the fluency of the outputs. Therefore, we again use the pre-trained GPT-2 model, but as a reward this time. To encourage the outputs to be as fluent as the source sentences, we define the fluency reward as the difference of the perplexity between the system outputs and source sentences:
$$
r _ {l a n g} \left(\tilde {x} _ {s}\right) = p p l \left(x _ {s}\right) - p p l \left(\tilde {x} _ {s}\right). \tag {6}
$$
Here,  $ppl$  denotes the length-normalized perplexity assigned by the language model fine-tuned on the training set.
As will be further discussed in Section 3.3, we found that using the rewards mentioned above can still result in unnatural outputs. Therefore, we additionally use a LSTM-based (Hochreiter and Schmidhuber, 1997) discriminator  $f_{adv}$  to provide a naturalness reward, whose job is to discriminate the system outputs and the real sentences, i.e., an adversarial discriminator. It constructs a min-max game with the generator:
$$
\begin{array}{l} \min  _ {\theta_ {g}} \max  _ {\theta_ {f _ {a d v}}} \mathbb {E} _ {x _ {s}} [ \log (1 - f _ {a d v} (g (x _ {s}, 1 - s))) ] \tag {7} \\ + \mathbb {E} _ {x _ {s}} [ \log (f _ {a d v} (x _ {s})) ]. \\ \end{array}
$$
The naturalness reward is the log-likelihood of the outputs being classified as real sentences:
$$
r _ {a d v} (\tilde {x} _ {s}) = \log (f _ {a d v} (\tilde {x} _ {s})). \tag {8}
$$
2.4 Learning
The final corresponding loss term is:
$$
L _ {*} \left(\theta_ {g}\right) = - \frac {1}{N} \sum_ {i = 1} ^ {N} r _ {*} \left(\tilde {x} _ {s} ^ {(i)}\right). \tag {9}
$$
Here,  $N$  is the number of samples in the dataset. To train the model, we use the weighted average of the losses defined in the previous section:
$$
\begin{array}{l} L (\theta_ {g}) = \lambda_ {c l s} L _ {c l s} (\theta_ {g}) + \lambda_ {a d v} L _ {a d v} (\theta_ {g}) \\ + \lambda_ {s i m} L _ {s i m} \left(\theta_ {g}\right) + \lambda_ {l a n g} L _ {l a n g} \left(\theta_ {g}\right) \tag {10} \\ + \lambda_ {r e c} L _ {r e c} (\theta_ {g}). \\ \end{array}
$$
where  $\lambda$  denotes the weight of the corresponding term. The setting of  $\lambda$  is chosen to make the training stable and have balanced style transfer accuracy and content preservation performance on the development set.  $L_{rec}$  is the reconstruction loss, i.e.,
$$
L _ {r e c} \left(\theta_ {g}\right) = \mathbb {E} _ {x _ {s}} \left[ - \log \left(p _ {g} \left(x _ {s} \mid x _ {s}, s\right)\right) \right]. \tag {11}
$$
We follow a two-stage training procedure. We first use the cycle-consistency loss  $L_{cyc}$  to bootstrap the training and then fine-tune the model with the rewards we introduced above to improve the output quality.
In the bootstrap stage, the objective function is
$$
\begin{array}{l} L _ {\text {b o o t}} \left(\theta_ {g}\right) = \lambda_ {\text {c y c}} L _ {\text {c y c}} \left(\theta_ {g}\right) + \lambda_ {\text {c l s}} L _ {\text {c l s}} \left(\theta_ {g}\right) \tag {12} \\ + \lambda_ {r e c} L _ {r e c} \left(\theta_ {g}\right) \\ \end{array}
$$
We select the checkpoint with the highest mean of the style transfer accuracy and BLEU on the development set as the starting point for the second training stage.
In the second stage, the generator is optimized with Eq. 10. The classifier  $f_{cls}$  for  $L_{cls}$  is pretrained and the language model for  $L_{lang}$  is finetuned on the training set. During training, the discriminator  $f_{adv}$  for  $L_{adv}$  is trained against the generator.  $f_{cls}$  is fixed when trained on some datasets, while it is trained against the generator on others. We select the checkpoint that has the style transfer accuracy and BLEU score similar to that from the first stage and the lowest perplexity on the development set.
Lastly, since gradients can not be propagated through the discrete samples, we use two approaches to circumvent this problem. For the content preservation reward (Eq. 3) and fluency reward (Eq. 6), we use the REINFORCE (Williams, 1992) algorithm to optimize the model,
$$
\begin{array}{l} \nabla_ {\theta_ {g}} \mathbb {E} _ {\tilde {x} _ {s} \sim p _ {g} (\tilde {x} _ {s})} [ r (\tilde {x} _ {s}) ] \tag {13} \\ = \mathbb {E} _ {\tilde {x} _ {s} \sim p _ {g} (\tilde {x} _ {s})} [ \nabla_ {\theta_ {g}} \log p _ {g} (\tilde {x} _ {s}) r (\tilde {x} _ {s}) ] \\ \end{array}
$$
We approximate the expectation by greedy decoding and the log-likelihood is normalized by sequence length, i.e.,  $\frac{1}{L}\sum_{i = 1}^{L}\log p_g(\tilde{w}_i)$ , where  $\tilde{w}_i$  denotes the  $i$ -th token of  $\tilde{x}_s$  and  $L$  is sequence length. For the style transfer accuracy reward (Eq. 2) and naturalness reward (Eq. 8), we use a different approach to generate a continuous approximation of the discrete tokens, which allows gradients to be back-propagated to the generator. Namely, taking the style classifier  $f_{cls}$  as an example, we use the distribution  $p_i$  of each token produced by the generator as the input of the classifier. This distribution is then multiplied by the classifier's word embedding matrix  $W^{\text{embed}}$  to obtain a weighted average of word embeddings:
$$
\hat {w} _ {i} = p _ {i} W ^ {\text {e m b e d}} \tag {14}
$$
Then, the classifier takes the sequence of  $\hat{w}_i$  as its input. We chose this method because it provides a token-level supervision signal to the generator, while the REINFORCE algorithm provides sentence-level signals.
We evaluate our approach on three datasets for sentiment transfer with positive and negative reviews: Yelp review dataset, Amazon review dataset provided by Li et al. (2018),² and the IMDb movie review dataset provided by Dai et al. (2019).³
We also evaluate our methods on a formality style transfer dataset, Grammarly's Yahoo Answers Formality Corpus (GYAFC),<sup>4</sup> introduced in Rao and Tetreault (2018). Although it is a parallel corpus, we treat it as an unaligned corpus in our experiments. In order to compare to previous work,
$^{2}$ https://github.com/lijuncen/Sentiment-and-Style-Transfer
3https://github.com/fastnlp/nlp-dataset
$^{4}$ https://github.com/raosudha89/GYAFC-corpus
Dataset	Style	Train	Dev	Test
Yelp	Positive	266K	2000	500
Negative	177K	2000	500
Amazon	Positive	277K	985	500
Negative	279K	1015	500
IMDb	Positive	178K	2000	1000
Negative	187K	2000	1000
GYAFC	Formal	52K	2247	500
Informal	52K	2788	500
Table 1: Number of samples in the Train, Dev, and Test splits for each dataset in our experiments.
Dataset	Eq.	λcls	λadv	λsim	λlang	λrec	λcyc
Yelp	(10)	2	0.5	20	2	0.1	-
(12)	1	-	-	-	1	1.5
Amazon	(10)	2	0.5	20	2	1	-
(12)	5	-	-	-	1	0.5
IMDb	(10)	1	0.5	20	2	1	-
(12)	1	-	-	-	1	1
GYAFC	(10)	2	0.5	20	2	1	-
(12)	1	-	-	-	1	1
we chose the Family & Relationships category for our experiments. Datasets statistics are shown in Table 1.
3.2 Experimental Details
Following previous work, we measure the style transfer accuracy using a FastText $^5$  (Joulin et al., 2017) style classifier trained on the respective training set of each dataset. To measure content preservation, we use SIM and BLEU as metrics where self-SIM and self-BLEU are computed between the source sentences and system outputs, while ref-SIM and ref-BLEU are computed between the system outputs and human references when available. To measure the fluency we use a pre-trained GPT-2 model to compute the perplexity. $^6$  Our generator, GPT-2, has 1.5 billion parameters, and we train on a GTX 1080 Ti GPU for about 12 hours.
The weights of the loss terms in Eq. 10 and Eq. 12 are detailed in Table 2. While during our experiments we found that there are other possible configurations which give higher scores with respect to the automatic evaluation metrics, as will be discussed in Section 3.3, we also found that
Table 2: Hyperparameter setting of Eq. 10 and Eq. 12 on each dataset.
Dataset	Model	Acc	PPL	BLEU
Yelp	DIRR-CYCLE	91.7	392	18.7
DIRR-YELP-ADV	95.2	353	20.7
Amazon	DIRR	62.2	205	30.1
DIRR-AMAZON-ADV	83.2	228	29.0
Table 3: Adversarial Results. DIRR-YELP-ADV and DIRR-AMAZON-ADV denote the models which generate adversarial examples. Acc denotes the style transfer accuracy, PPL denotes the perplexity, BLEU is computed between the human references and system outputs.
better performance in automatic evaluation doesn't always entail better performance in human evaluation. Therefore, we also manually checked the quality of the transferred texts on development set when we chose the value of the hyperparameters.
We compare our model with several state-of-the-art methods: DeleteAndRetrieve (D&R) (Li et al., 2018), B-GST (Sudhakar et al., 2019), CycleMulti (Dai et al., 2019), Deep-Latent (He et al., 2020), Tag&Gen (Madaan et al., 2020), and DualRL (Luo et al., 2019). We also compare our final model, DIRR(Direct-Reward), with the model only trained with the first stage (DIRR-CYCLE) as mentioned in Section 2.4.
3.3 Adversarial Examples
Yelp and Amazon are arguably the most frequently used datasets for the sentiment transfer task. In our experiments, we found that the automatic evaluation metrics can be tricked on these datasets. Table 3 shows the performance of the models which generate adversarial examples. Upon identifying these risks, we propose several design options that can effectively mitigate these problems.
Yelp Dataset For the Yelp dataset, when trained without the adversarial discriminator  $f_{adv}$  and the fluency reward, our model (DIRR-YELP-ADV) is able to discover a trivial solution which receives high automatic evaluation scores: injecting a word that carries strong sentiment at the beginning of the output, and making minimum changes (if any) to the source sentences, as illustrated in Table 8. This obviously does not meet the objective of content-preserving sentiment transfer and is easily detectable for humans. In fact, after we manually removed the first word from each of the output sentences, the transfer accuracy dropped from 95.2 to 58.4. To address this problem, we introduced an
<sup>5</sup>https://fasttext.cc/
$^{6}$ Note that we didn't fine-tune it on the training set
Model	"game"	"phone"
Pos.	Neg.	Pos.	Neg.
Train	58	7548	8947	2742
Test	0	10	20	6
Human	1	10	18	6
B-GST	55	0	13	44
Tag&Gen	69	0	14	5
DIRR	26	0	19	45
DIRR-AMAZON-ADV	291	0	190	4
auxiliary discriminator  $f_{adv}$  as we discussed above to penalize the trivial outputs since they can be easily captured by the discriminator. On the other hand, the output perplexity is not sensitive enough to this local feature so using the fluency reward alone is not sufficient. Our final model has much more stable performance when the first word of its output sentences is removed, experiencing only a small drop of the style transfer accuracy from 94.2 to 88.2.
Amazon Dataset For the Amazon dataset, we found that the style classifier  $f_{cls}$  needs to be updated during the training to prevent the model exploiting the data imbalance problem of the dataset. Namely, in the Amazon dataset some categories of products appear mostly in negative or positive reviews. In Table 4, we show the word frequency of game and phone in both negative and positive reviews. In the original dataset, game mostly appears in negative reviews while phone mostly appears in positive reviews. Therefore, without any prior knowledge, it is very likely that these words will be used as informative features by the sentiment classifier, which makes its predictions unreliable.[7]
When our second-stage model is trained with the fixed style classifier, it (DIRR-AMAZON-ADV) learns to exploit this dataset bias by changing the nouns in the original sentences to game or phone, which achieves better transfer accuracy. We list some examples in Table 5. DIRR-AMAZON-ADV generated 291 game in 500 positive reviews, which obviously changes the semantics of the source sentences. In order to show that this phenomenon is independent to the classifier architec