SlowGuess/NAACL_2021_OCR huggingface dataset:
Navita Goyal, Balaji Vasan Srinivasan, Anandhavelu N, Abhilasha Sancheti
Adobe Research, India
{navgoyal, balsrini, anandvn, sancheti}  $@$  adobe.com
Style transfer has been widely explored in natural language generation with non-parallel corpus by directly or indirectly extracting a notion of style from source and target domain corpus. A common shortcoming of existing approaches is the prerequisite of joint annotations across all the stylistic dimensions under consideration. Availability of such dataset across a combination of styles limits the extension of these setups to multiple style dimensions. While cascading single-dimensional models across multiple styles is a possibility, it suffers from content loss, especially when the style dimensions are not completely independent of each other. In our work, we relax this requirement of jointly annotated data across multiple styles by using independently acquired data across different style dimensions without any additional annotations. We initialize an encoder-decoder setup with transformer-based language model pre-trained on a generic corpus and enhance its re-writing capability to multiple target style dimensions by employing multiple style-aware language models as discriminators. Through quantitative and qualitative evaluation, we show the ability of our model to control styles across multiple style dimensions while preserving content of the input text. We compare it against baselines involving cascaded state-of-the-art uni-dimensional style transfer models.
Style transfer is a popular task in natural language processing and has been studied on attributes like age or gender (Subramanian et al., 2018), styles emanating from social construct like formality (Rao and Tetreault, 2018) and politeness (Madaan et al., 2020), linguistic styles based on author writing style (Syed et al., 2020), or psycho-linguistic styles based on personality types (Mairesse and Walker, 2011). While early style transfer frameworks were modeled as a supervised learning task
on a parallel corpus, state-of-the-art models are semi-supervised/unsupervised and operate on non-parallel corpus. These models achieve style transfer by aligning source and target distribution of sentences from non-parallel corpus (Shen et al., 2017), disentangling content space from style space in latent representation (Hu et al., 2017) or employing self-reconstruction (Dai et al., 2019) and back translation (Lample et al., 2018) objectives to achieve pseudo-supervision with non-parallel corpus. Recent works have also modeled this in a self-supervised manner where rewriting (transfer) is achieved by utilizing corpus from the target style alone (Syed et al., 2020). These wide studies have also led to the curation and benchmarking of non-parallel dataset for various style dimensions, such as sentiment (Li et al., 2018), formality (Rao and Tetreault, 2018), politeness (Danescu-Niculescu-Mizil et al., 2013), excitement (Sancheti et al., 2020), etc. But availability of data with joint tagging across multiple styles is limited and has restricted the ability of existing approaches to scale from single-dimensional transfer to multiple style dimensions. In this paper, we propose a multidimensional style transfer approach that can work off partially labelled data for style transfer across multiple dimensions simultaneously.
The work by Subramanian et al. (2018) attempts style transfer with multiple attributes such as age, gender, and sentiment simultaneously. However, their approach avails corpus tagged with each of these three style dimensions. In contrast to this and other similar explorations in multi-style transfer, our approach does not require jointly labelled data across all the stylistic dimensions in source and/or target corpus. We focus on the problem where independent corpus is available across different stylistic dimensions (say sentiment and formality) and we achieve style transfer spanning different stylistic dimensions (say make a sentence more positive and formal). While state-of-the-art approaches can be
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3500-3510 June 6-11, 2021. ©2021 Association for Computational Linguistics
extended to achieve this by sequentially transferring one style after another, it is limited as different style dimensions are not necessarily independent of each other. In aspects that are not independent, changing one style aspect of the text might affect another aspect considered, making a sequential brute-force approach non-ideal. As we show in our experiments later, the cascaded setup also lacks common grounding between the content from different styles leading to erratic changes in content. We circumvent this by grounding our framework on the linguistic understanding of a large language model. Our model builds understanding of interplay between the different styles by incorporating multiple discriminative language models (LM) with language model-based encoder-decoder setup. The key contributions of this paper are:
1) An encoder-decoder setup with multiple language models as discriminator, with each entity harnessing the language understanding from a large pre-trained transformer model.
2) Relaxing the requirement of jointly labelled data for multi-style transfer, by leveraging independently acquired disjoint corpus for different styles.
3) Achieving better style control with better content preservation in multi-dimensional style transfer than a cascaded setup of state-of-the-art unidimensional style transfer models.
One line of work in style transfer attempts to learn disentangled latent representation for style and content, and transfer style by manipulating latent representation of style (Shen et al., 2017). Although these approaches perform well with one style at a time, they do not trivially scale to multidimensional style transfer. Several other works develop unsupervised approach for style transfer by employing Denoising Autoencoding (DAE) (Fu et al., 2017) and back-translation (BT) (Lample et al., 2018) loss to develop interaction and hence transfer between the source and target domain. Subramanian et al. (2018) extend this approach to multiple styles by conditioning on average of embedding of each target attribute and using combination of DAE and back-translation techniques. DAE takes as input a sentence  $x$  from style  $s$  and tries to reconstruct sentence  $x$  from its corrupted version  $\tilde{x}$ . This relies on the assumption that the input sentence  $x$  is from a certain style combination  $s = \{s_1, s_2, \ldots, s_k\}$ . Similarly back translation
(BT) objective with input sentence  $x$  from style  $s$ , first estimates  $x' = f(x, s')$ , where  $s \neq s'$  and then reconstruct  $x$  from  $\tilde{x} = f(x', s)$ . Thus, these approaches are inherently dependent on knowledge of annotation of each sentence with all the style combinations. Dai et al. (2019) achieve state-of-the-art style transfer in single style dimensions by employing transformer-based model in conjunction with classifier-based discriminator. In addition to discriminator losses, their proposed technique uses self-reconstruction and cycle reconstruction losses, which similar to DAE and BT losses are also reliant on availability of jointly annotated data to be extendable to multiple style setup.
Language modeling is integral to several natural language generation (NLG) tasks like text summarization, spelling correction, image captioning, etc. The model architecture for these tasks has evolved from n-gram based methods to Recurrent Neural Networks to transformer architectures. The introduction of Transformer-based architecture accompanied with generative pre-training (Radford, 2018) capabilities have led to strong improvements in many downstream generation and GLUE (Wang et al., 2018) tasks. Generative pre-training aims to adapt a large Transformer language model to large unsupervised corpus. This capability of generative pre-training is exploited in many large language models like BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2018), ERNIE 2.0 (Sun et al., 2020) which have the ability to perform tasks like reading comprehension (Xu et al., 2019), summarization (Liu and Lapata, 2019), question-answering (Rajpurkar et al., 2016) and translation (Clinchant et al., 2019) in zero-shot and few-shot settings.
Recently these pre-trained generative language models have been explored in translation (Conneau and Lample, 2019) and style transfer tasks (Syed et al., 2020). Conneau and Lample (2019) develop cross-lingual models for unsupervised machine translation by initializing encoder and decoder with a pre-trained language model trained on Masked Language Modeling (MLM) (Devlin et al., 2019) objective and fine-tuning the encoder-decoder framework with adversarial training. Syed et al. (2020) extend this to stylized re-writing task by employing DAE during fine-tuning. The joint encoder-decoder framework learns to reconstruct sentences in target-domain from its noisy version using DAE objective. As previously discussed, the DAE objective is reliant on the corpus being tagged
for the target domain style (or combination of style) and restricts the generalization of this setup to multiple attributes. We overcome this by employing discriminative language models to assist the decoder with feedback for various target styles.
Shen et al. (2017) show that even with non-parallel data, the content distribution across source and target style is shared. Based on this, a language model trained on target style will have high perplexity on transferred text if it does not match target style and low perplexity otherwise. Yang et al. (2018) exploit this ability of language models to replace standard binary classifier-based discriminators with an implicitly trained language model as discriminator. They show that using the language model as structured discriminator allows for more stable training by eliminating the adversarial step. We extend this idea to a multi-discriminator approach. Training a LM on combination of target styles is not possible in absence of jointly labelled dataset. Due to this, we attempt to use multiple discriminators for each of the target styles. Since with multiple styles, the underlying corpus is independently acquired, the variation in content distribution across different styles is more noticeable. Consequently, an independently trained LM on one of the target styles might have high perplexity even if the transferred sentence fits in the corresponding target style, due to the content space of source sentence. To equip discriminative LM with more generalized notion of content, we use large transformer-based LM pre-trained on large unsupervised corpus to establish generic content distribution before style-oriented fine-tuning.
Our proposed approach has two key elements — a Transformer-based encoder-decoder model initialized with a pre-trained Transformer Language Model and fine-tuned on DAE loss to achieve style transfer (Section 3.1) and the multiple language models as discriminators stacked together to enable multi-style transfer (Section 3.2).
3.1 Pre-trained LM as Encoder-Decoder
Similar to Syed et al. (2020), we first pre-train a Transformer-based language model with Masked Language Modeling (MLM) objective on English Wikipedia data extracted using WikiExtractor. This equips LM with the ability to predict masked
words over a large corpus. Masked Language Modeling leverages bidirectional context of the input, thus enabling better language understanding. Following Masked Language Modeling objective from Devlin et al. (2019), we randomly sample  $15\%$  of the tokens from the text stream and replace them with the [MASK] token  $80\%$  of the time, by a random token  $10\%$  of the time and keep them unchanged  $10\%$  of the time, with the objective of predicting the original identity of the masked word based on its bidirectional context. To enable style transfer from a given sentence to target style, we use independently trained language models (LMs) to initialize the encoder and decoder and connect these with randomly initialized attention layers to arrive at a encoder-decoder setup. As discussed by Syed et al. (2020), the Transformer architecture (Vaswani et al., 2017) allows such independent initialization by implicitly aligning encoder-decoder layers via attention mechanism.
Pre-training an encoder only transformer on generative task and then leveraging it to initialize as both encoder and decoder as opposed to pretraining a joint encoder-decoder model has several advantages. Transformer-based models with encoder-only (Devlin et al., 2019) or decoder-only (Radford et al., 2018) blocks have been shown to perform well in generative pre-training task. Clearly, pre-training a single transformer block on generative task and then utilizing it as both encoder and decoder blocks has lower computational cost than training the entire encoder-decoder block jointly. Moreover, this also enables us to use the same pre-trained model to initialize both style transfer module and the discriminator models, explained in the following section. This is not only computationally more efficient but it also closely ties the underlying language distribution of the two modules. This is expected to make the discriminative feedback more effective while fine tuning the transfer model for multiple styles.
In Syed et al. (2020)'s setup, both encoder and decoder in the style transfer module are initialized with the pre-trained language model (trained on MLM objective). Instead, we initialize the decoder with the language model fine-tuned with the target style using Causal Language Modeling (CLM) objective, before training the joint encoder-decoder model, as detailed in Section 3.2. The encoder is initialized with the pre-trained model directly. Aligning the decoder to the distribution of the tar
$^{1}$ https://github.com/attardi/wikiextractor
Figure 1: Model Architecture - Left: Generative pre-training using MLM objective, and Fine-tuning encoder-decoder LM with multiple discriminative losses and Right: Discriminator fine-tuning with language modeling (next token prediction) objective. Color for model blocks represents the pre-trained model used for initialization prior to fine-tuning.
get style helps speed up the fine-tuning process as decoder is more adept at generating stylized outputs. This does not add to computational overhead as these fine-tuned models are repurposed as discriminators for stylistic feedback (Section 3.2).
To instill style-awareness to the encoder-decoder setup initialized with pre-trained Transformer models, we fine-tune it with Denoising Autoencoder (DAE) loss using the target-domain corpus. In case of multiple styles, we use a randomized mixture of target-domain corpus from each of the target styles. Under the DAE objective, the encoder takes a noisy masked version  $\tilde{x}$  of the text  $x$  as input and attempts to fill in the mask token as per the MLM objective that it was pre-trained on. In turn, the decoder re-creates stylistic version of original sentence from this noisy output from the encoder. The overall training objective is
$$
\mathcal {L} _ {D A E} \left(\theta_ {G}\right) = \mathbf {E} _ {x \sim T} \left[ - \log P _ {\theta_ {G}} (x | \tilde {x}) \right], \tag {1}
$$
where  $\theta_{G}$  are the trainable parameters of the encoder-decoder model. The noisy version of sentence  $x$  from the target corpus  $T$  is obtained after dropping tokens from  $x$  with probability  $p_{drop}$  and masking with a probability of  $p_{mask}$ . In conjunction, the encoder and decoder enable style transfer to the target style. The noteworthy aspect here is that the model has no sense of source style and is trained to generate sentences to match the style of the target-domain corpus with which it is trained.
3.2 Fine-tuned LM as discriminators
To extend the single-dimensional style transfer setup above to multi-dimensional setting, we use language models as discriminators to provide the feedback to the model for partially annotated nature of input data. As opposed to a classifier-based discriminator, the language model as discriminator takes into account the wider language distribution of the target style. Additionally, such a setup allows us to use only the target style corpus for training the transfer model, whereas the classifier would require both source and target style corpus to distinguish between a sentence as being from one style or another. Inspired by Yang et al. (2018), we fine-tune a language model on the target style  $s_i$ , so that the language model is equipped with language distribution of target domain data. This entails generating the probability of next token, given the previous tokens — also known as Causal Language Modeling objective (Conneau and Lample, 2019). The training loss for the LM for target style  $s_i$  with corresponding corpus  $T_i$  is
$$
\mathbf {E} _ {x \sim T _ {i}} \left[ \sum_ {t = 1} ^ {n} \left[ - \log P _ {L M} \left(x _ {t} \mid x _ {1}, \dots , x _ {t - 1}\right) \right] \right] \tag {2}
$$
We show in our experiments that such a finetuning step transforms language distribution of this language model to style  $s_i$  and hence serve as soft-discriminator for our framework. We exploit this capability of language models to imbibe style of
fine-tuning corpus by employing language models as style discriminators for transferred sentences. This is based on the idea that if the transferred sentence does not fit well in the target style, then the perplexity of language model fine-tuned on that style will be high (Section 4.1).
For  $k$ -dimensional style transfer with target styles  $s = \{s_1, s_2, \ldots, s_k\}$ , we independently fine-tune  $k$  language models on each of the target styles. As discussed in Yang et al. (2018), we are able to forgo the adversarial training for the discriminator, since the fine-tuned discriminative language model is implicitly capable of assigning high perplexity to negative samples (out-of-style samples), as shown in Section 4.1. For the transferred sentence  $x'$ , the training objective for each target style  $s_i$  is,
$$
\begin{array}{l} \underset {\theta_ {G}} {\operatorname {a r g m i n}} \mathcal {L} ^ {s _ {i}} = \mathbf {E} _ {x \sim T, x ^ {\prime} \sim P _ {\theta_ {G}} (x)} \\ \left[ \sum_ {t = 1} ^ {n} - \log P _ {L M _ {i}} \left(x _ {t} ^ {\prime} \mid x _ {1} ^ {\prime},.., x _ {t - 1} ^ {\prime}\right) \right] ^ {(3)} \\ \end{array}
$$
This dictates that transferred sentence  $x'$  has low perplexity on the language model fine-tuned on style  $s_i$ , for each target style  $s_i$ . However, we cannot directly find the  $\mathrm{argmin}_{\theta_G}$  using gradient descent because of discrete sampling of  $x' \sim P_{\theta_G}(x)$ . To account for this, we use a policy gradient reinforcement learning approach using REINFORCE algorithm (Sutton et al., 1999). The reward for an input sequence  $x$  to the style discriminator  $LM_i$  is calculated as,
$$
r (x) = \sum_ {t = 1} ^ {n} \log P _ {L M _ {i}} \left(x _ {t} \mid x _ {1},.., x _ {t - 1}\right) \tag {4}
$$
Using these rewards, the RL objective is to minimize the loss  $\mathcal{L}^{s_i}$  given by,
$$
\begin{array}{l} \mathcal {L} ^ {s _ {i}} = \mathbf {E} _ {x \sim T, x ^ {\prime} \sim P _ {\theta_ {G}} (x)} \left(r \left(x ^ {\prime}\right) - r (x)\right) \tag {5} \\ [ - \log P _ {\theta_ {G}} (x ^ {\prime} | \tilde {x}) ] \\ \end{array}
$$
for style  $s_i$ , where  $P_{\theta_G}(x|\tilde{x})$  is as in Equation 1 and  $r(x')$  is the reward in the Equation 4 for the transferred sentence  $x'$ . The rewards  $r(x)$  represents the baseline reward of greedily sampling the input sequence  $x$  by the style discriminator  $LM_i$ .
For the style combination  $s = \{s_1, s_2, \ldots, s_k\}$ , the joint encoder-decoder model is trained on randomized mixture of data from each of the target-domain corpus. The mixture is thus agnostic of individual style of each of the sentence and the
discriminative LM for each style guides the generation towards that specific style by rewarding style adherence in the transferred sentence. Randomized mixture of training corpus across styles allows for unified and cohesive understanding of multiple styles by diversifying rewards from different discriminators across samples. The overall training loss for the joint encoder-decoder model is
$$
\mathcal {L} = \lambda_ {D A E} \mathbf {E} _ {x \sim T} [ - \log P _ {\theta} (x | \tilde {x}) ] + \sum_ {i = 1} ^ {k} \lambda_ {i} \mathcal {L} ^ {s i}, \tag {6}
$$
where  $\mathcal{L}^{s_i}$  is as defined in Equation 5, and  $\lambda_{DAE}$  and  $\{\lambda_i\}_{i = 1}^k$  are hyper-parameters.
The overall training process is summarized in Figure 1. First, we pre-train a transformer model with Masked language modeling objective as shown in Figure 1(Left). We then initialize discriminator model with this pre-trained language model and fine-tune it with Causal language modeling objective, shown in Figure 1(Right), for each target style. Finally, we initialize the encoder and decoder of the style transfer module with the pretrained and style-specific fine-tuned language models, respectively. In case of multiple styles, the decoder can be initialized with the language model which is fine-tuned with CLM loss on the mixture of data from target styles, i.e., CLM loss in Equation 2 with  $x \sim T$ . The joint encoder-decoder model (Figure 1(Centre)) is then trained with a combination of DAE objective and rewards from fine-tuned discriminators of respective target styles.
We experiment with a combination of sentiment and formality styles. For sentiment, we use a mixture of IMDB (Maas et al., 2011) and Yelp dataset (Li et al., 2018) with  $300k$  examples in the positive and negative sentiment each. For formality, we use GYAFC corpus (Rao and Tetreault, 2018) which has  $104k$  examples in each formal and informal class. The test set has 3000 and 4849 examples for sentiment and formality respectively, following the data split available in Dai et al. (2019); Rao and Tetreault (2018). For both datasets, the training corpus is non-parallel and the test corpus has human written references available, which we use for content evaluation (Section 4.2).
For pre-training, we use 12-layer Transformer model with 512 hidden units, 16 heads, a dropout rate of 0.1 and learned positional embedding. We train our models with the Adam optimizer, and
Style/Dimension	Sentiment %	Formality %
Positive	71.41	67.09
Negative	76.17	75.59
Table 1: Accuracy of sentences generated by model fine-tuned on style  $s_i$  as % of generated sentences labelled as class  $s_i$  by the classifier trained on the corresponding style dimension.
Fine-tuning
corpus	Test Corpus
Same ↓	Opposite ↑
Positive	6.9275	9.6850
Negative	7.7131	9.9637
Table 2: Perplexity of test corpus on models fine-tuned positive and negative corpus (rows). The column Same represents that test corpus is same as fine-tuning corpus, leading to lower perplexities and Opposite represent test corpus from opposite polarity as fine-tuning corpus leading to higher perplexity.
a learning rate of  $10^{-4}$ . To handle large vocabulary sizes, we use Byte Pair Encoding (BPE) (Sennrich et al., 2016) learned on the Wikipedia dataset. The  $\lambda$ s in Equation 6 are determined using hyperparameter tuning on validation set, with style transfer accuracy (Section 4.2) as search criteria.
4.1 Style-awareness of Language Models