SlowGuess/NAACL_2022_OCR huggingface dataset:
Text style transfer (TST) is a well-known task whose goal is to convert the style of the text (e.g., from formal to informal) while preserving its content. Recently, it has been shown that both syntactic and semantic similarities between the source and the converted text are important for TST. However, the interaction between these two concepts has not been modeled. In this work, we propose a novel method based on Optimal Transport for TST to simultaneously incorporate syntactic and semantic information into similarity computation between the source and the converted text. We evaluate the proposed method in both supervised and unsupervised settings. Our analysis reveals the superiority of the proposed model in both settings.
Text style transfer (TST) is an important task in NLP that aims to change the style of a given text from source style to target style (e.g., formal to informal) while preserving its content. For instance, the formal sentence "However, I do believe it to be punk" is converted to the informal equivalent sentence "I'd say it is punk though". This task could be helpful for downstream applications such as text simplification, information extraction, and question answering.
Due to the importance of TST, this task has been approached with different techniques ranging from feature-based models (Xu et al., 2012) to recent advanced deep learning solutions (Chen et al., 2018; Lee et al., 2021a; Huang et al., 2021). The recent work can be categorized as supervised (i.e., parallel corpus with sentences in source and target style) (Lai et al., 2021), unsupervised (i.e., sentences in source and target style are available but they are not aligned) (Krishna et al., 2020), or semi-supervised (combination of parallel and non-aligned corpora) (Chawla and Yang, 2020) methods. The three crit-
ical objectives of any TST system are to (1) generate a text in the target style, (2) keep the content of the source text, and (3) generate fluent sentences (Krishna et al., 2020). It has been shown that fine-tuning transformer-based language models on each of these objectives (i.e., using Reinforcement Learning) can achieve promising results (Lai et al., 2021; Liu et al., 2021). However, one of the limitations of the existing works is that the content preservation (i.e., the second objective) is fulfilled at either the surface-form level (i.e., by encouraging the same words to appear in both texts) (Lai et al., 2021) or at the semantics level (i.e., by encouraging high mutual information between the two texts) (Chawla and Yang, 2020); ignoring the role of syntactic information. Syntactic information (e.g., dependency tree) can be used to explicitly encode the connections between the words of the sentence, thereby playing an important role in the equivalency of two sentences. For instance, consider the source sentence "a crap touch bar with a nice screen!!!" and the converted sentence "The screen is great but the touch bar is terrible". The corresponding dependency between "touch bar → crap" in the source sentence and "terrible → touch bar" in the target sentence and also "screen → nice" in the source sentence and "great → screen" in the target sentence are helpful to assess the equivalency of the two sentences. Although the pre-trained language models such as BERT have been shown to be able to encode the syntactic information, it is not yet verified that these models can take into account the syntactic dependencies when computing the similarity between two sentences, especially for the TST task. To the best of our knowledge, there is one prior work that shows the importance of the syntactic information for transformer-based TST models (Ma et al., 2019). Specifically, Ma et al. (2019) shows that reconstructing both the words of the source text and their POS tags could boost the performance of TST. However, there are two limi
Human Language Technologies, pages 2532 - 2541
tations in this work: (1) the syntactic structure (i.e., dependencies between words) is ignored; (2) the interaction between semantics and the syntax of the sentences is neglected. More specifically, to obtain the most value of the syntactic information, it is crucial to consider the relations between the words and also their semantics as shown in the example above. As such, in this work, we propose a novel method to simultaneously incorporate the interaction between syntax and semantics of the sentences into the content preservation objective of TST training. More specifically, for the first time in text style transfer, we propose to use Optimal Transport (OT) as an efficient method to consider both the syntax and the semantics of the two sentences when computing their content similarity. OT has been shown to be an effective method for image style transferring (Kolkin et al., 2019; Risser, 2020) and our work exhibits its application for the domain of the text. We evaluate the proposed model on three benchmark datasets and two settings, i.e., supervised and unsupervised. Our extensive analysis reveals the effectiveness of the proposed model by establishing new state-of-the-art results.
Problem Definition: The task of text style transfer is formally defined as follows: Given the input sentence  $D = [w_{1}, w_{2}, \ldots, w_{n}]$  with style  $s$ , the goal is to generate a new sentence  $D' = [w_{1}', w_{2}', \ldots, w_{m}']$  in the target style  $t$  while preserving the content of  $D$  in  $D'$ . We study both supervised and unsupervised settings. Specifically, in the supervised setting, for every training sample  $(D, s)$  there is an aligned sentence  $\bar{D}$  in target style  $t$ , i.e.,  $(\bar{D}, t)$ , whose content is the same as  $D$ . Whereas for the unsupervised setting, there is no equivalent pair for  $(D, s)$ . Note that in the unsupervised setting, there are sentences for both styles.
In this work, we employ a transformer-based generative language model, i.e., GPT-2 (Radford et al., 2019), for TST and we train the model using REINFORCE algorithm. Specifically, the source sentence  $D$  is prompted to the GPT-2 model to generate the target sentence  $D^{\prime}$ . Following the prior work, (Lai et al., 2021), the GPT-2 model is encouraged to generate the sentence  $D^{\prime}$  in the target style  $t$  and with the same content as  $D$ . Also, in the supervised setting, we use the gold target sentence  $\bar{D}$  as an additional supervision signal to train the
model. Since  $\bar{D}$  is not available in the unsupervised setting, we follow the prior work (Lee et al., 2021a) to use reconstruction loss as an additional training signal. The rest of this section provides details for generating sentences, rewards for generation, and training procedures.
2.1 Generating Target Sentence
Following the prior work (Lai et al., 2021), we employ the input sentence  $D$  as a prompt to GPT-2 model to generate the target sentence  $D^{\prime}$ . More specifically, the prompt to GPT-2 consists of the sequence  $P = [BOS, w_{1}, w_{2}, \ldots, w_{n}, SEP]$ , where  $BOS$  and  $SEP$  are special token indicating the beginning and the end of the input sentence  $D$ . In addition to the input document  $D$ , during training of the supervised model, the gold target sentence  $\bar{D} = [\bar{w}_{1}, \bar{w}_{2}, \ldots, \bar{w}_{n^{\prime}}]$  is concatenated to the prompt to create the training sequence  $S = [BOS, w_{1}, w_{2}, \ldots, w_{n}, SEP, \bar{w}_{1}, \bar{w}_{2}, \ldots, \bar{w}_{n^{\prime}}]$  and the model is trained in an auto-regressive manner:
$$
\mathcal {L} _ {L M} = \sum_ {i} ^ {n + n ^ {\prime} + 2} - \log (Q \left(S _ {i} \mid S _ {<   i}, \theta\right)) \tag {1}
$$
where  $\theta$  is GPT-2 parameters and  $Q(\cdot | S_{<i}, \theta)$  is the distribution over vocabulary obtained from the last hidden states of GPT-2 model. During inference, only the prompt  $P$  is provided to the GPT-2 model and the words of  $D'$  are sampled from the distribution predicted by GPT-2 model until EOS is sampled.
Unlike the supervised setting in which the GPT-2 model is trained for uni-directional style conversion, i.e., from the source style to the target style, in the unsupervised setting, the model is trained for both directions, i.e., from the source to the target and vice versa. Specifically, given a sentence and a style, the GPT-2 model is trained to generate another sentence with the same content in the given style. Formally, the prompt  $P$  is concatenated with the style  $st$  where  $st \in \{s,t\}$ , i.e.,  $S = [BOS,w_1,w_2,\dots ,w_n,SEP,st]$ . To train the model, following the prior work (Lee et al., 2021a), two types of reconstruction loss are employed:
Self-Reconstruction: The GPT-2 model is encouraged to reconstruct the original input sentence  $D$  when  $st$  is  $s$ , i.e., the given style to the model is the
same as the input sentence style. Concretely, the loss function  $\mathcal{L}_{LM}$  is defined as follows:
$$
\mathcal {L} _ {L M} = \sum_ {i} ^ {n} - \log (Q (D _ {i} | D, s, \theta)) \tag {2}
$$
Cycle Reconstruction: If  $st$  is  $t$ , i.e., the given style to the model is different from the style of the input sentence, then the GPT-2 model is first employed to generate the sentence  $\bar{D}'$  in the style  $t$ . Next, the model is encouraged to reconstruct the original input sentence  $D$  using the input  $S = [BOS, \bar{w}_1, \bar{w}_2, \dots, \bar{w}_{n'}, SEP, s]$ , where  $\bar{w}_i$  is the  $i$ -th word in the generated sentence  $\bar{D}'$ . Concretely, the loss function  $\mathcal{L}_{LM}$  is defined as follows:
$$
\mathcal {L} _ {L M} = \sum_ {i} ^ {n} - \log (Q (D _ {i} | \bar {D} ^ {\prime}, s, \theta)) \tag {3}
$$
2.2 Rewarding GPT-2 Model
Prior work shows that rewarding generative models to observe the requirements for TST could improve the performance (Lai et al., 2021; Liu et al., 2021). Hence, we follow this optimization step to update the GPT-2 model based on two different rewards, i.e., Style Conversion and Content Preservation.
Style Conversion: One of the critical objectives of TST is to change the style of the given text. To encourage the model for this objective, prior works commonly use a pre-trained discriminator to predict the style of the generated text. Here, we follow the same approach by pre-training a BERT model (Devlin et al., 2019) on the combination of the training sentences  $D$  in both styles to identify the style of the given text (i.e., a binary text classification task). Next, during the training stage of the GPT-2 model, we send the generated sentence  $D'$  to the pre-trained BERT model. The probability of the target style is employed as the style conversion reward:  $R_{SC}(D') = Q_{BERT}(t|D',\phi)$ , where  $\phi$  is the BERT parameters.
Content Preservation Content preservation is an important requirement of TST and prior works use either surface form of  $D$  and  $D^{\prime}$  (Lai et al., 2021; Huang et al., 2021), their semantics (Chawla and Yang, 2020), or only shallow syntax (Ma et al., 2019) to compute the content overlap between the source and the generated sentence. None of these works consider the syntactic structure of two sentences and more importantly its interaction with the semantics of the sentence. As the main novelty of the proposed work, inspired by the success of
Optimal Transport in image style transfer (Kolkin et al., 2019; Risser, 2020) and other related NLP tasks (Xu et al., 2021), we show that OT is an effective tool for addressing the shortcoming of syntax-semantics interaction for content preservation in prior TST literature.
To represent the semantics of the source and target sentence  $D$  and  $D'$ , we employ the hidden states of the final GPT-2 layer for each word  $w_i$  and  $w_j'$ , i.e.,  $H = [h_1, h_2, \ldots, h_n]$  and  $H' = [h_1', h_2', \ldots, h_n']$ . Moreover, the syntactic structures of the two sentences are obtained from an off-the-shelf dependency tree parser $^1$ , represented by  $T$  and  $T'$ . The criterion we use to compute the content preservation between two sentences  $D$  and  $D'$  is that the semantically related words in both sentences should have the same syntactic importance too. In particular, we expect that similar words appear at the same level in the dependency tree of  $T$  and  $T'$ . However, since the structure of the sentence might change during style conversion and also the number of words might alter, similar words might appear in other levels too. As such, the optimal mapping between similar words in the dependency trees  $T$  and  $T'$  is not trivial. Fortunately, optimal transport (OT) can be helpful to solve this issue. OT is a mathematical method to compute the cheapest plan for converting one data distribution to another one. We first formally describe OT and then we elaborate on how it is employed for our purpose.
OT is an established method to find the optimal plan to convert (i.e., transport) one distribution to another one. Formally, given the probability distributions  $p(x)$  and  $q(y)$  over the domains  $\mathcal{X}$  and  $\mathcal{Y}$ , and the cost/distance function  $C(x,y): \mathcal{X} \times \mathcal{Y} \to \mathbb{R}_+$  for mapping  $\mathcal{X}$  to  $\mathcal{Y}$ , OT finds the optimal joint alignment/distribution  $\pi^*(x,y)$  (over  $\mathcal{X} \times \mathcal{Y}$ ) with marginals  $p(x)$  and  $q(y)$ , i.e., the cheapest transportation from  $p(x)$  to  $q(y)$ , by solving the following problem:
$$
\begin{array}{l} \pi^ {*} (x, y) = \min  _ {\pi \in \Pi (x, y)} \int_ {\mathcal {Y}} \int_ {\mathcal {X}} \pi (x, y) C (x, y) d x d y \tag {4} \\ \begin{array}{l} \text {s . t .} x \sim p (x) \text {a n d} y \sim q (y), \end{array} \\ \end{array}
$$
where  $\Pi (x,y)$  is the set of all joint distributions with marginals  $p(x)$  and  $q(y)$ . Note that if the distributions  $p(x)$  and  $q(y)$  are discrete, the integrals in Equation 4 are replaced with a sum and the joint distribution  $\pi^{*}(x,y)$  is represented by a
<sup>1</sup>We employ Stanford dependency parser
0 → 1	1 → 0
Domain	Train	Valid	Test	Valid	Test
F&R	51,967	2,788	1,332	2,247	1,019
E&M	52,595	2,877	1,416	2,356	1,082
Table 1: Statistics of GYAFC dataset employed for supervised setting (number of samples).  $0=$  informal,  $1=$  formal
Dataset	Style	Train	Dev	Test
Yelp	Positive	266,041	2,000	500
Negative	177,218	2,000	500
IMDB	Positive	178,869	2,000	1,000
Negative	187,597	2,000	1,000
Table 2: Statistics of the IMDB and Yelp datasets employed for unsupervised setting (number of samples).
matrix whose entry  $(x,y)$ $(x\in \mathcal{X},y\in \mathcal{Y})$  represents the probability of transforming the data point  $x$  to  $y$  to convert the distribution  $p(x)$  to  $q(y)$ . Finally, the cost of optimal conversion (i.e., Wasserstein distance  $Dist_W$ ) is computed by:  $Dist_{W} = \Sigma_{x\in \mathcal{X}}\Sigma_{y\in \mathcal{Y}}\pi^{*}(x,y)C(x,y)$ .
In our method, we use the words  $w_{i} \in D$  as the domain  $\mathcal{X}$  and the words  $w_{j}^{\prime} \in D^{\prime}$  as the domain  $\mathcal{Y}$ . In order to define their distance, we use the Euclidean distance between their semantic vector representation  $C(w_{i},w_{j}^{\prime}) = \left\| h_{i} - h_{j}^{\prime}\right\|$ . Finally, to define the distributions  $p(x)$  and  $q(y)$ , we use the level of words  $w_{i}$  and  $w_{j}^{\prime}$  in the dependency tree  $T$  and  $T^{\prime}$ , respectively. Concretely,  $p(w_{i}) = \text{softmax}(M - L_{i})$ , where  $M$  is the maximum depth of  $T$ ,  $L_{i}$  is the depth of  $w_{i}$  in  $T$  and softmax is computed over all words  $w_{i} \in D$ . Similarly,  $q(w_{j}^{\prime})$  is defined by  $q(w_{j}^{\prime}) = \text{softmax}(M^{\prime} - L_{j}^{\prime})$ . By solving the equation  $4^{2}$ , the cheapest conversion of the two sentence  $D$  and  $D^{\prime}$  is obtained and its cost is equal to Wasserstein distance  $Dist_{W}$ . We use this distance as the content preservation penalty, i.e.,  $R_{CP}(D^{\prime}) = -Dist_{W}$ .
3 Training
To train the model, we combine the content preservation reward  $R_{CP}(D')$ , with style conversion and the language model loss. We use REINFORCE algorithm (Williams, 1992) to train the model. In particular, the GPT-2 model is trained on the combination of the language model loss,
i.e.,  $\mathcal{L}_{LM}$  and the rewards of style conversion and content preservation. The REINFORCE algorithm is employed to incorporate rewards into fine-tuning of GPT-2. First, the overall reward is computed by  $R(D^{\prime}) = R_{SC}(D^{\prime}) + \alpha R_{CP}(D^{\prime})$  where  $\alpha$  is a trade-off hyper-parameter. Next, we seek to minimize the negative expected reward  $R(D^{\prime})$  over the possible choices of  $D^{\prime}$ :  $\mathcal{L}_R = -\mathbb{E}_{\hat{D}^{\prime}\sim P(\hat{D}^{\prime}|D)}[R(\hat{D}^{\prime})]$ . The policy gradient is then estimated by:  $\nabla \mathcal{L}_R = -\mathbb{E}_{\hat{D}^\prime \sim P(\hat{D}^\prime |D)}[(R(\hat{D}^\prime) - b)\nabla \log P(\hat{D}^\prime |D)]$ . Using one roll-out sample, we further estimate  $\nabla \mathcal{L}_R$  via the generated sentence  $D^{\prime}$ :  $\nabla \mathcal{L}_R = -(R(D^{\prime}) - b)\nabla \log P(D^{\prime}|D)$  where  $b$  is the baseline to reduce variance. In this work, we obtain the baseline  $b$  via:  $b = \frac{1}{|B|}\sum_{i = 1}^{|B|}R(D_i')$  where  $|B|$  is the mini-batch size and  $D_{i}^{\prime}$  is the generated sentence for the  $i$ -th sample in the mini-batch.
Datasets: We evaluate the proposed model, i.e., Optimal Transport-based Text sStyle Transfer (OT4), in two different settings, i.e., supervised and unsupervised. In the supervised setting we employ the Grammarly's Yahoo Answers Formality Corpus (GYAFC) dataset (Rao and Tetreault, 2018). GYAFC is a parallel dataset in two domains Entertainment & Music (E&M) and Family & Relationships (F&R). Table 1 shows the statistics of this dataset.
For the unsupervised setting, we employ two commonly used datasets: Yelp (Li et al., 2018) and IMDB (Dai et al., 2019) review. Both datasets contain sentiment-annotated reviews. The text style transfer on these datasets is defined as sentiment polarity conversion. In particular, given a sentence with a specific sentiment polarity (e.g., positive), the goal is to generate a new sentence with the opposite sentiment polarity (e.g., negative). Note that no parallel data is available for the sentences in these datasets. The statistics of both datasets are provided in Table 2
**Evaluations:** We validate the model performance using both automatic and human evaluation. For the automatic evaluation in the supervised setting, following the prior work (Lai et al., 2021), we assess the performance of the models based on: (1) Style Strength (ACC): The binary style classifier TexCNN (Kim, 2014) (with  $87.0\%$  and  $89.3\%$  accuracy on E&M and F&R domains, respectively) is employed to predict the strength of the style conversion; (2) Content Preservation (BLEU): The BLEU
2Note that as solving the OT problem in Equation 4 is intractable, we employ the entropy-based approximation of OT and solve it with the Sinkhorn algorithm (Peyre and Cuturi, 2019).
score computed using four reference sentences; (3) HM: The harmonic mean of ACC and BLEU; and (4) BLEURT: A new metric for content preservation proposed by Sellam et al. (2020). For the automatic evaluation in the unsupervised setting, following prior work (Lee et al., 2021b), we use: (1) Style Transfer Accuracy (S-ACC): Following (Lee et al., 2021b), a Bi-GRU layer with attention mechanism, trained for style classification on IMDB and Yelp dataset, is employed to assess the style transfer; (2) Content Preservation (self-BLEU, ref-BLEU, BERT-P, BERT-R, and BERT-F1): To validate the content preservation in the generated sentence, its BLEU score with input sentence, i.e., self-BLEU, and with the human-generated sentence, i.e., ref-BLEU, are used. Moreover, to incorporate contextual semantics, the BERT score proposed by (Zhang et al., 2020) is employed to assess the similarity between the generated sentence and the human reference. Following prior work (Lee et al., 2021b), we report precision, recall, and F1 score for this metric; (3) Fluency (PPL): The fluency of the generated sentences is evaluated based on the perplexity of the sentences using the 5-gram KenLM (Heafield, 2011) model trained on both datasets;
For human evaluation, following prior work (Lee et al., 2021b), we randomly select 150 documents for each test set and we hire 4 annotators to rate model predictions from 1 (Very Bad) to 5 (Very Good) on content preservation, style conversion, and fluency. For each annotator, we provide them with the source text, source style, target style, and model-generated text.
Baselines: We compare our model with the prior state-of-the-art models in each setting. Specifically, for the supervised setting on GYAFC, we compare our model with GPT-2 + SC & BLEU (Lai et al., 2021): Similar to our model, this baseline employs GPT-2 to generate the target sentence. The generative model is trained using rewards for style conversion (SC) and content preservation (BLEU); BART + SC & BLEU (Lai et al., 2021): The same as the previous baseline with the difference of using BART instead of GPT-2; NMT-Combined (Rao and Tetreault, 2018): This baseline casts TST as a machine translation problem and employs attention based BiLSTM encoder-decoder architecture; Bi-directional FT (Niu et al., 2018): This model employs BiLSTM encoder to jointly learn text formality style transfer in both direction (from formal
to informal and vice versa); CPLS (Shang et al., 2019): This baselines employs an encoder-decoder architecture to obtain latent space representation of the styles, then a projection model converts the styles in the latent space; GPT-CAT (Wang et al., 2019): This baseline combines rule-based methods with neural-based TST systems. GPT-2 is employed as the neural component; TS  $\rightarrow$  CP (Sancheti et al., 2020): This model exploits reinforcement learning to explicitly encourage content preservation and transfer strength. It exerts BLEU score between generated and ground-truth sentence to compute content preservation reward; and Chawla's (Chawla and Yang, 2020): This baseline uses a language model discriminator to guide the text formality style transfer. For content preservation, it employs mutual information between source and target sentence.
For the unsupervised setting on IMDB and Yelp, we compare with Cross-Alignment (Shen et al., 2017): This baseline is trained to generate a sentence in the target style that could match the example sentences in the source style. To this end, a cross-aligned auto-encoder is utilized; ControlledGen (Hu et al., 2017): This model employs variational author encoder (VAE) with attribute discriminators to impost semantic structure, including text style; Style Transformer (Dai et al., 2019): In this baseline, a transformer model is employed to directly takes the input sentence and target style to generate the target sentence; Deep Latent (He et al., 2020): This baseline models the unsupervised text style transfer as the task of inferring latent variables, i.e., target sentences, on the partially observed data of each style. A recurrent language model is employed to fulfill the objective. RACoLN (Lee et al., 2021b): In this baseline, the reverse attention technique is employed to remove style information from the representations of the tokens in the source sentence.
Supervised: Table 3 shows the results of the evaluations on the test set. Following prior work, we compare the performance of the proposed OT4 model in the following settings: (1) Informal  $\leftrightarrow$  Formal: In this setting the performance of the baselines for converting a formal to informal text or vice versa is evaluated. From this table, we observe that GPT-2 model has a better capability of style conversion. However, the baseline model using GPT-2