SlowGuess/ACL_2020_OCR huggingface dataset:
Chulun Zhou $^{1}$ , Liangyu Chen $^{2}$ , Jiachen Liu $^{2}$ , Xinyan Xiao $^{2}$ , Jinsong Su $^{1*}$ , Sheng Guo $^{2}$ , Hua Wu $^{2}$
$^{1}$ Xiamen University, Xiamen, China  $^{2}$ Baidu Inc., Beijing, China
clzhou@stu.xmu.edu.cn jssu@xmu.edu.cn
{chenliangyu, liujiachen, xiaoxinyan, guosheng, wu_hua}@baidu.com
Unsupervised style transfer aims to change the style of an input sentence while preserving its original content without using parallel training data. In current dominant approaches, owing to the lack of fine-grained control on the influence from the target style, they are unable to yield desirable output sentences. In this paper, we propose a novel attentional sequence-to-sequence (Seq2seq) model that dynamically exploits the relevance of each output word to the target style for unsupervised style transfer. Specifically, we first pretrain a style classifier, where the relevance of each input word to the original style can be quantified via layer-wise relevance propagation. In a denoising auto-encoding manner, we train an attentional Seq2seq model to reconstruct input sentences and repredict word-level previously-quantified style relevance simultaneously. In this way, this model is endowed with the ability to automatically predict the style relevance of each output word. Then, we equip the decoder of this model with a neural style component to exploit the predicted word-level style relevance for better style transfer. Particularly, we fine-tune this model using a carefully-designed objective function involving style transfer, style relevance consistency, content preservation and fluency modeling loss terms. Experimental results show that our proposed model achieves state-of-the-art performance in terms of both transfer accuracy and content preservation.
Text style transfer is a task that changes the style of input sentences while preserving their style-independent content. Due to its wide applications, such as sentiment transfer (Hu et al., 2017; Shen
et al., 2017) and text formalization (Jain et al., 2019), it has become a research hotspot in natural language generation in recent years. However, due to the lack of parallel training data, researchers mainly focus on unsupervised style transfer.
In this aspect, many approaches (Hu et al., 2017; Shen et al., 2017; Fu et al., 2018) resort to an autoencoding framework, where the encoder is used to disentangle the content and style, and the decoder generates the output sentence with the target style. Another line of research (Xu et al., 2018; Li et al., 2018) focuses on removing the style marker of the input sentence to obtain a style-independent sentence representation. When generating the output sentence, both lines directly feed the target style into the decoder as a whole. From another perspective, some researchers treat the style transfer as a translation process and adapt unsupervised machine translation to this task (Logeswaran et al., 2018; Zhang et al., 2018; Lample et al., 2019), where the style switch is implicitly achieved. Besides, many recent models further explore this task in different ways, such as gradient-based optimization method (Liu et al., 2019), dual reinforcement learning (Luo et al., 2019), hierarchical reinforcement learning (Wu et al., 2019) and transformer-based model (Dai et al., 2019). Overall, in these models, the quality of output sentences mainly depends on the content representation of input sentence and the exploitation of the target style.
However, one main drawback of the aforementioned models is that they lack the fine-grained control of the influence from the target style on the generation process, limiting the potential of further style transfer improvement. Intuitively, the frequencies of words occurring the sentences with different styles are distinct, and thus they are related to various styles in different degrees. In view of the above, we believe that during the ideal style transfer, impacts of the target style should be distin
$^{\dagger}$ Equal contribution. This work is done when Chulun Zhou was interning at Baidu Inc., Beijing, China.  
*Corresponding author
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7135-7144 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
guished depending on different words. If we equip the current style transfer model with a neural network component, which can automatically quantify the style relevance of the output sentence at word level, the performance of the model is expected to be further improved.
In this paper, we propose a novel attentional sequence-to-sequence model (Seq2seq) that dynamically predicts and exploits the relevance of each output word to the target style for unsupervised style transfer. Specifically, we first pre-train a style classifier, where the relevance of each input word to the original style can be quantified through layer-wise relevance propagation (LRP) (Bach et al., 2015). After that, in a denoising auto-encoding manner, we train a basic attentional Seq2seq model to reconstruct the input sentence and repredict its word-level previously-quantified style relevance simultaneously. In this way, this model is endowed with the ability to automatically predict the style relevance of each output word. Then, we equip the decoder of this model with a neural style component to exploit the predicted word-level style relevance for better style transfer. Particularly, we fine-tune this model using a carefully-designed ojective function involving style transfer, style relevance consistency, content preservation and fluency modeling loss terms.
Compared with previous approaches, our proposed model avoids the complex disentanglement procedure, of which the quality can not be guaranteed. Also, our model is able to solve the issue of the source-side information loss caused by unsatisfactory disentanglement or explicitly removing style markers. More importantly, our model is capable of achieving fine-grained control over the impacts of target style on different output words, leading to better style transfer. To sum up, our contributions can be summarized as follows:
- We explore a training approach based on LRP and denoising auto-encoding for the Seq2seq style transfer model, which enables the model to automatically predict the word-level style relevance of output sentences;
- We propose a novel Seq2seq model, which exploits the predicted word-level style relevance of output sentences for better style transfer. To the best of our knowledge, the text style transfer with fine-grained style control has not been explored before;
- Experimental results and in-depth analysis on
two benchmark datasets strongly demonstrate the effectiveness of our model. We release our code at https://github.com/PaddlePaddle/ Research/tree/master/NLP/ACL2020-WST
In recent years, unsupervised text style transfer has attracted increasing attention. Most of previous work (Hu et al., 2017; Shen et al., 2017; Fu et al., 2018; Prabhumoye et al., 2018; Xu et al., 2018; Li et al., 2018) aimed at producing a style-independent content representation from the input sentence and generate the output with target style. For example, Hu et al. (2017) employed a variational auto-encoder with an attribute classifier as discriminator, forcing the disentanglement of specific attributes and content in latent representation. Shen et al. (2017) exploited an auto-encoder framework with an adversarial style discriminator to obtain a shared latent space cross-aligning the content of text from different styles. Based on multi-task learning and adversarial training of deep neural networks, Fu et al. (2018) explored two models to learn style transfer from non-parallel data. Prabhumoye et al. (2018) learned a latent representation of the input sentence to better preserve the meaning of the sentence while reducing stylistic properties. Then adversarial training and multi-task learning techniques were exploited to make the output match the desired style. Although these work has shown effectiveness to some extent, however, as analyzed by some recent work (Li et al., 2017; Lample et al., 2019), their style discriminators are prone to be fooled.
Meanwhile, some studies (Luo et al., 2019; Li et al., 2018) explicitly removed style-related words identified by a pre-trained classifier to get a style-independent content representation, and then added target style to generate output sentences. Nevertheless, such approaches tend to cause the information loss of the input sentence, since its style-related words often contain meaningful content. Besides, there have been several researches (Logeswaran et al., 2018; Zhang et al., 2018; Lample et al., 2019) adopting back-translation to build style transfer models without parallel data. Logeswaran et al. (2018) introduced a reconstruction loss interpolating auto-encoding and back-translation loss components, where attribute compatibility is encouraged by a discriminator. Along this line, Zhang et al. (2018) and Lample et al. (2019) directly adapted un
7136
supervised machine translation approaches to this task, where the style transfer is implicitly achieved via iterative back-translations between texts in different styles.
Very recently, some attempts (Liu et al., 2019; Luo et al., 2019; Wu et al., 2019; Dai et al., 2019) have been made to perform style transfer from other different perspectives. For example, Liu et al. (2019) mapped a discrete sentence into a continuous space and then used the gradient-based optimization with a pre-trained attribute predictor to find the latent representation satisfying desired properties (e.g. sentence length, sentiment). Luo et al. (2019) performed a one-step mapping to directly transfer the style of the original sentences via dual reinforcement learning. Wu et al. (2019) adopted a hierarchical reinforced sequence operation method to iteratively revise the words of original sentences. Dai et al. (2019) proposed a Transformer-based (Vaswani et al., 2017) style transfer model without disentangling latent representation. Finally, note that exploring word-level style relevance has also been studied in other NLP tasks, such as machine translation (Zeng et al., 2018; Su et al., 2019).
Given a set of labelled training instances  $D = \{(X_{i},s_{i})\}_{i = 1}^{|D|}$ , where  $s_i\in \{0,1\}$  is the style label of the sentence  $X_{i}$ , we aim to train a style transfer model that can automatically convert an input sentence  $X = x_{1},x_{2},\ldots ,x_{|X|}$  with the original style  $s$  into a content-invariant output sentence  $Y = y_{1},y_{2},\dots,y_{|Y|}$  with the target style  $s'$ .
To achieve this goal, we extend the standard attentional Seq2seq model (Sutskever et al., 2014; Bahdanau et al., 2015) by equipping its decoder with a neural style component to achieve fine-grained control over the impacts of target style on different output words. As shown in Figure 1, the training of our model consists of two stages, which will be described below.
3.1 Stage 1: Train a Basic Attentional Seq2seq Model with Repredicting Word-level Style Relevance
At this stage, we first introduce a pre-trained style classifier to quantify the word-level relevance of training sentences to the original style via LRP (Bach et al., 2015). Then, we train a basic attentional Seq2seq model in a denoising auto-encoding manner, where this model is required to reconstruct
the input sentence and repredict its word-level style relevance simultaneously. By doing so, our model acquires the preliminary ability to predict the style relevance of output words and reconstruct input sentences, which makes the training in the subsequent stage easier. Next, we briefly describe our basic model, and then introduce its objective function in detail.
3.1.1 Attentional Seq2seq Model
It mainly consists of an encoder and a decoder with an attention mechanism.
The encoder is a forward GRU network. Taking the sentence  $X = \{x_{1}, x_{2}, \ldots, x_{|X|}\}$  as input, this network maps the input words into a hidden state sequence as  $\mathbf{h}_i^e = \mathrm{GRU}(\mathbf{e}(x_i), \mathbf{h}_{i-1}^e)$ , where  $\mathbf{e}(x_i)$  and  $\mathbf{h}_i^e$  denotes the embedding vector and the hidden state of the word  $x_i$ , respectively. Specially, the last hidden state  $\mathbf{h}_{|X|}^e$  is used to initialize the decoder hidden state  $\mathbf{h}_0^d$ .
The decoder is also a forward GRU network. Its hidden state is updated by  $\mathbf{h}_j^d = \mathrm{GRU}(\mathbf{e}(y_{j - 1}),\mathbf{h}_{j - 1}^d,\mathbf{c}_j)$ , where  $\mathbf{h}_j^d$  is the decoder hidden state at the  $j$ -th timestep,  $\mathbf{e}(y_{j - 1})$  is the embedding vector of the previously generated word  $y_{j - 1}$ , and  $\mathbf{c}_j$  denotes the corresponding attention-based context vector. Formally,  $\mathbf{c}_j$  is defined as the weighted sum of all hidden states of input words:
$$
\mathbf {c} _ {j} = \sum_ {i} ^ {| X |} \frac {\exp \left(e _ {j i}\right)}{\sum_ {i ^ {\prime} = 1} ^ {| X |} \exp \left(e _ {j i ^ {\prime}}\right)} \mathbf {h} _ {i} ^ {e}, \tag {1}
$$
$$
e _ {j i} = \mathbf {v} ^ {\top} \tanh (\mathbf {W _ {e}} \mathbf {h} _ {i} ^ {e} + \mathbf {W _ {d}} \mathbf {h} _ {j} ^ {d}), \qquad (2)
$$
where  $\mathbf{v}$ ,  $\mathbf{W}_{\mathrm{e}}$ ,  $\mathbf{W}_{\mathrm{d}}$  are trainable parameters. Finally, the output prediction probability over vocabulary is calculated as  $P(y_{j}|y_{< j},X) = \text{Softmax}(\mathbf{W}_{o}\mathbf{h}_{j}^{d})$ , where  $\mathbf{W}_{o}$  is a learnable matrix. Please note that all bias terms in above equations are also trainable parameters, which are omitted for the simplicity of notation.
3.1.2 The Objective Function
To effectively train the above basic model, we define the following objective function  $L_{1}$ :
$$
L _ {1} = L _ {s r} + L _ {x \lambda}, \tag {3}
$$
where  $L_{sr}$  and  $L_{x\lambda}$  denote the sentence reconstruction loss and the style relevance restoration loss, respectively.
1. Sentence reconstruction loss  $L_{sr}$ : Using this loss, we expect our model to capture informative
Stage 1: Train a basic attentional Seq2seq model with Repredicting Word-level Style Relevance
Figure 1: Two training stages of our model: Stage 1: Train a basic attentional Seq2seq model that is required to reconstruct the input sentence and repredict its word-level style relevance; Stage 2: Extend the basic model by equipping its decoder with a neural style component, and then fine-tune it using a novel objective function.  $X$  and  $Y$  are the input sentence and the output sentence, respectively, and  $\widetilde{X}$  is the partially corrupted version of  $X$ . LRP indicates layer-wise relevance propagation (Bach et al., 2015).  $\lambda_{*}$  indicates the word-level style relevance.  $L_{*}$  represents the loss terms used in objective functions.
Stage 2: Fine-tune the extended model
features for reconstructing sentence. Formally, we define  $L_{sr}$  as follows:
$$
L _ {s r} \left(\theta_ {s 2 s}\right) = - \sum_ {i = 1} ^ {| \widetilde {X} |} \log P \left(x _ {i} \mid x _ {<   i}, \widetilde {X}\right). \tag {4}
$$
Here,  $\theta_{s2s}$  denotes parameters of this Seq2seq model, and  $\widetilde{X}$  is the partially corrupted version of  $X$ , where a certain proportion of input words are randomly replaced to prevent our model from simply copying  $X$ .
2. Style relevance restoration loss  $L_{x\lambda}$ : It is used to measure how well the word-level style relevance of an input sentence can be repredicted during the denoising auto-encoding. Formally, it is defined as
$$
L _ {x \lambda} (\theta_ {s 2 s}, \theta_ {\lambda}) = \frac {1}{| X |} \sum_ {i = 1} ^ {| X |} (\lambda_ {x i} - \hat {\lambda} _ {x i}) ^ {2}, \qquad (5)
$$
where  $\lambda_{xi}\in (0,1)$  and  $\hat{\lambda}_{xi}\in (0,1)$  denote the style relevance of the  $i$ -th input and output word, respectively, and  $\theta_{\lambda}$  denotes the set of other parameters used to calculate  $\hat{\lambda}_{xi}$  (see Equation 6). It is notable that  $\hat{\lambda}_{xi}$  and  $\hat{\lambda}_{xi}$  are not involved into the sentence reconstruction. Apparently, two key issues arise, namely, how to calculate  $\lambda_{xi}$  and  $\hat{\lambda}_{xi}$ ?
As for  $\hat{\lambda}_{xi}$ , we calculate it based on the previous decoder hidden state  $\mathbf{h}_{i-1}^{d}$ :
$$
\hat {\lambda} _ {x i} = \mathbf {v} _ {\lambda} ^ {\top} \mathrm {t a n h} (\mathbf {W} _ {\lambda} \mathbf {h} _ {i - 1} ^ {d}), \qquad (6)
$$
where  $\mathbf{W}_{\lambda}$ ,  $\mathbf{v}_{\lambda}$  form the previously-mentioned parameter set  $\theta_{\lambda}$  (see Equation 5).
To obtain  $\lambda_{xi}$ , we use the instances in the training set  $D$  to pre-train a TextCNN (Kim, 2014) style classifier. During this process, we apply LRP (Bach et al., 2015), which has been widely used to measure the contributions of neurons to the final prediction of a classifier, to quantify the word-level style relevance of sentences. Concretely, we calculate the relevance score  $r_k^{(l)}$  of the  $k$ -th neuron  $n_k^{(l)}$  at the  $l$ -th layer in a manner similar to back-propagation:
$$
r _ {k} ^ {(l)} = \sum_ {k ^ {\prime} = 1} ^ {K} \frac {z _ {k k ^ {\prime}}}{\sum_ {k ^ {\prime \prime}} z _ {k k ^ {\prime \prime}}} r _ {k ^ {\prime}} ^ {(l + 1)}, \tag {7}
$$
where  $z_{kk'} = v_k^{(l)}w_{kk'}^{(l,l + 1)}$ . Here,  $w_{kk'}^{(l,l + 1)}$  denotes the weight of the edge between adjacent neurons,  $v_{k}^{(l)}$  is the value of neuron  $n_k^{(l)}$  that can be computed in advance during the forward-propagation, and  $K$  denotes the neuron number of each layer. Through this way, we can obtain the neuron-wise contribution scores  $\{r_k^{(0)}(x_i)\}_{k = 1}^K$  of the embedding vector of input word  $x_{i}$  to the final style prediction. Furthermore, we define the style relevance score  $r(x_{i})$  of  $x_{i}$  as the sum of  $\{r_k^{(0)}(x_i)\}_{k = 1}^K$ , and finally map this score into the range [0, 1) via a  $\tanh (*)$  function:  $\lambda_{xi} = \tanh (\eta |r(x_i)|)$ , where the hyper-parameter  $\eta$  serves as a scaling factor. In practice, since too low style relevance may be noise, we directly treat the words with style relevance lower than  $\epsilon$  as style independent and set their style relevance as  $0$ .
*Experiments show that such treatment enhances the stability of our system.
Figure 2: The architecture of our proposed neural style component.  $\Delta \mathbf{h}_j^d$  is the revision to the hidden state  $\mathbf{h}_j^d$ ,  $\lambda_{yj}$  is a gate controlling to what extent the  $\mathbf{h}_j^d$  will be revised by  $\Delta \mathbf{h}_j^d$  and  $\tilde{\mathbf{h}}_j^d$  is the revised hidden state used to generate  $y$ .
3.2 Stage 2: Fine-tune the Extended Model
At this stage, we extend the above basic model by equipping its decoder with a neural style component, which predicts and then exploits the style relevance of the next output word to refine its generation. Likewise, we first give a detailed description to our extended model, and then depict how to fine-tune it using a novel objective function involving multiple loss terms.
3.2.1 The Extended Model
Here, we omit the descriptions of our basic encoder and decoder, which are identical to those of the previously-described basic model, and only depict the newly introduced neural style component. Figure 2 shows the architecture of this component.
To incorporate word-level style relevance into our decoder, at the  $j$ -th timestep, we predict the style relevance  $\lambda_{yj}$  following Equation 6, and then use  $\lambda_{yj}$  to revise the decoder hidden state  $\mathbf{h}_j^d$  as follows:
$$
\tilde {\mathbf {h}} _ {j} ^ {d} = \mathbf {h} _ {j} ^ {d} + \lambda_ {y j} \Delta \mathbf {h} _ {j} ^ {d}, \tag {8}
$$
where  $\Delta \mathbf{h}_j^d$  is the revision to the hidden state  $\mathbf{h}_j^d$ ,  $\lambda_{yj}$  is a gate controlling to what extent the  $\mathbf{h}_j^d$  will be revised by  $\Delta \mathbf{h}_j^d$  and  $\tilde{\mathbf{h}}_j^d$  is the revised hidden state used to generate  $y_j$  (see Subsection 2.1.1). Note that  $\lambda_{yj}$  reflects how much information of target style is incorporated.
More specifically,  $\mathbf{h}_j^d$  and  $\Delta \mathbf{h}_j^d$  are updated as below:
$$
\mathbf {h} _ {j} ^ {d} = \operatorname {G R U} \left(\mathbf {e} \left(y _ {j - 1}\right), \tilde {\mathbf {h}} _ {j - 1} ^ {d}, \mathbf {c} _ {j}\right), \tag {9}
$$
$$
\Delta \mathbf {h} _ {j} ^ {d} = f (\mathbf {e} (y _ {j - 1}), \tilde {\mathbf {h}} _ {j - 1} ^ {d}, s ^ {\prime}; \theta_ {\Delta h}), \tag {10}
$$
where  $\mathbf{e}(y_{j - 1})$  is the embedding vector of  $y_{j - 1}$ $\mathbf{c}_j$  is the context vector calculated as Equation 1,  $s^\prime$  is the target style, and  $f(*)$  is an MLP function with the parameter set  $\theta_{\Delta h}$  used to produce  $\Delta \mathbf{h}_j^d$
By doing so, we can impose fine-grained control over the influence of the target style on the generation of the next word. When our model predicts that the next word is strongly style-relevant, it encourages the decoder to produce the proper stylistic content by leveraging more target style information. Conversely, the target style will exert less influence on the decoder hidden state, avoiding being disturbed to lose original style independent content.
3.2.2 The Objective Function
After initializing our extended model with parameters of the basic model, we then fine-tune it using the following objective function:
$$
L _ {2} = L _ {s t} + \alpha L _ {y \lambda} + \beta L _ {c p} + \gamma L _ {l m}, \tag {11}
$$
where  $L_{st}, L_{y\lambda}, L_{cp}$  and  $L_{lm}$  denote the style transfer loss, the style relevance consistency loss, the content preservation loss, and the frequency modeling loss, with  $\alpha, \beta, \gamma$  as their balancing parameters.
1. **Style transfer loss  $L_{st}$ :** It is used to ensure the output sentence contains the target style well. To this end, we apply the above-mentioned pre-trained style classifier (See **Stage 1**) to classify the style of the output sentence, where related parameters of our model are updated to encourage the target style can be predicted from the output sentence:
$$
L _ {s t} \left(\theta_ {s 2 s}, \theta_ {\lambda}, \theta_ {\Delta h}\right) = - \mathbb {E} _ {(X, s) \sim D} \left[ \log P \left(s ^ {\prime} \mid G (Y)\right) \right]. \tag {12}
$$
$G(Y)$  denotes a "soft" generated sentence based on gumbel-softmax distribution (Jang et al., 2017), where the representation of each word is defined as the weighted sum of word embeddings with the prediction probability at the current timestep.
2. Style relevance consistency loss  $L_{y\lambda}$ : It ensures the predicted style relevances of output words are consistent with its stylistic outcomes evaluated by the classifier. Specifically, during the above style classification, we apply LRP to obtain the style relevance  $\hat{\lambda}_{yj}$  of each "soft" word, and then try to minimize the following loss:
$$
L _ {y \lambda} \left(\theta_ {s 2 s}, \theta_ {\lambda}\right) = \frac {1}{| Y |} \sum_ {j = 1} ^ {| Y |} \left(\lambda_ {y j} - \hat {\lambda} _ {y j}\right) ^ {2}. \tag {13}
$$
3. Content preservation loss  $L_{cp}$ : However, only using the above two style-related loss terms will lead to model collapse, producing an extremely short output sentence that matches the target style but totally loses the original meaning. To address
7139
this issue, we introduce a content preservation loss  $L_{cp}$  to prevent the model from collapsing.
Specifically, we define the content representations of input sentence  $X$  and output sentence  $Y$  as the weighted sums of their individual word embeddings according to the corresponding weights  $1 - |\lambda_{xi}|$  and  $1 - |\lambda_{yj}|$ , respectively. Then, we minimize the following loss term to force these two representations to be close:
$$
\begin{array}{l} L _ {c p} \left(\theta_ {s 2 s}, \theta_ {\lambda}, \theta_ {\Delta h}\right) \tag {14} \\ = [ \sum_ {i} ^ {| X |} (1 - | \lambda_ {x i} |) \mathbf {e} (x _ {i}) - \sum_ {j} ^ {| Y |} (1 - | \lambda_ {y j} |) \mathbf {e} (y _ {j}) ] ^ {2}. \\ \end{array}
$$
In this way, the less relevant a word is to the corresponding style, the more its embedding should be considered.