SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiangyang Li $^{1,*}$ , Xiang Long $^{2,*}$ , Yu Xia $^{1}$ , Sujian Li $^{1,\dagger}$
{xiangyangli，yuxia，lusujian}@pku.edu.cn
xianglong@bupt.edu.cn
Text style transfer (TST) without parallel data has achieved some practical success. However, most of the existing unsupervised text style transfer methods suffer from (i) requiring massive amounts of non-parallel data to guide transferring different text styles. (ii) colossal performance degradation when fine-tuning the model in new domains. In this work, we propose DAML-ATM (Domain Adaptive Meta-Learning with Adversarial Transfer Model), which consists of two parts: DAML and ATM. DAML is a domain adaptive meta-learning approach to learn general knowledge in multiple heterogeneous source domains, capable of adapting to new unseen domains with a small amount of data. Moreover, we propose a new unsupervised TST approach Adversarial Transfer Model (ATM), composed of a sequence-to-sequence pre-trained language model and uses adversarial style training for better content preservation and style transfer. Results on multi-domain datasets demonstrate that our approach generalizes well on unseen low-resource domains, achieving state-of-the-art results against ten strong baselines.
Text style transfer (TST) aims to change the style of the input text and keep its content unchanged, which has been applied successfully to text formalization (Jain et al., 2019), text rewriting (Nikolov and Hahnloser, 2018), personalized dialogue generation (Niu and Bansal, 2018) and other stylized text generation tasks (Gao et al., 2019; Cao et al., 2020; Syed et al., 2020).
Text style transfer has been explored as a sequence-to-sequence learning task using parallel datasets (Jhamtani et al., 2017; Wang et al., 2020b; Pryzant et al., 2020). However, parallel datasets are difficult to obtain due to expensive manual annotation. The recent surge of deep generative methods
(Hu et al., 2017a; Zhao et al., 2017; Li et al., 2018) has spurred progress in text style transfer without parallel data. However, these methods typically require large amounts of non-parallel data and do not perform well in low-resource domain scenarios.
One typical method is to resort to massive data from different domains, which has been studied as an effective solution to address the above data insufficiency issue (Glorot et al., 2011; Wang et al., 2017; Li et al., 2021b). However, directly leveraging large amounts of data from other domains for the TST task is problematic due to the differences in data distribution over different domains, as different domains usually use their domain-specific lexica (Li et al., 2019a). For instance, fine-tuning a TST model trained on a high-resource movie-related domain to a low-resource restaurant-related domain can get us unreasonable sentences like "the food is dramatic." The sentiment word "dramatic" is suitable for commenting a movie but weird to comment on the food.
In this work, we tackle the problem of domain adaptation in the scenarios where the target domain data is scarce and misaligned with the distribution in the source domain. Recently, model-agnostic meta-learning (MAML) has received resurgence in the context of few-shot learning scenario (Lin et al., 2019; Gu et al., 2018; Nooralahzadeh et al., 2020). Inspired by the essence of MAML (Finn et al., 2017), we propose a new meta-learning training strategy named domain adaptive meta-learning (DAML). Unlike MAML, DAML adopts a domain adaptive approach to construct meta tasks that would be more suitable to learn a robust and generalized initialization for low-resource TST domain adaption.
To well preserve content and transfer style, one typical strategy of a TST model is to decouple style information from the semantics of a text, and it tends to produce content loss during style transfer (Hu et al., 2017b; Dai et al., 2019; Carlson et al.,
Human Language Technologies, pages 3014 - 3026
2018). Here, we do not try to decouple content and style, and propose a new Adversarial style Transfer model ATM, which is composed of a sequence-to-sequence pre-trained language model combined with adversarial style training for style transfer. In this way, our model can better preserve the content information without disentangling content and style in the latent space.
Combining DAML and ATM, in this paper, we propose the method named DAML-ATM, which extends traditional meta-learning to a domain adaptive method combined with a sequence-to-sequence style transfer model. DAML contains two alternating phases. During the meta-training phase, a series of meta-tasks are constructed from a large pool of source domains for balanced absorption of general knowledge, resulting in a domain-specific temporary model. In the meta validation stage, the temporary model is evaluated on the meta validation set to minimize domain differences and realize meta knowledge transfer across different domains. In ATM, a pre-training language model based TST model is used to improve text content retention. Moreover, we propose a two-stage training algorithm to combine the DAML training method and ATM model better.
In summary, the main contributions in this paper are three-fold: (i) We propose a new unsupervised TST model, which achieves SOTA performance without disentangling content and style latent representations compared to other models. (ii) We extend the traditional meta-learning strategy to the domain adaptive meta transfer method, effectively alleviating the domain adaption problem in TST. (iii) We propose a two-stage training algorithm to train DAML-ATM, achieving state-of-the-art performance against multiple strong baselines.
Text style transfer based on deep learning has been extensively studied in recent years. A typical pattern is first to separate the latent space as content and style features, then adjust the style-related features and generate stylistic sentences through the decoder. (Hu et al., 2017a; Fu et al., 2017; Li et al., 2019a) assume that appropriate style regularization can achieve the separation. Style regularization may be implemented as an adversarial discriminator or style classifier in an automatic encoding process. However, these style transfer paradigms
use large amounts of annotation data to train models for specific tasks. If we already have a model for a similar task, it is unreasonable to need many data still to train the model from scratch.
On the other hand, some of the previous work learned to do TST without manipulating the style of the generated sentence based on this learned latent space. (Dai et al., 2019)use the transformer architecture language model to introduce attention mechanism, but they do not make full use of the prior knowledge of sequence to sequence pre-trained language model, such as Bart (Lewis et al., 2019) and T5 (Raffel et al., 2019), which have made significant progress in text generation tasks. In this paper, we proposed the DAML training method to solve the domain shift problem in TST and proposed a new TST model architecture named ATM, which makes no assumption about the latent representation of source sentence and takes the proven sequence-to-sequence pre-trained language model.
2.2 Domain adaptation
Domain adaptation has been studied in various natural language processing tasks (Glorot et al., 2011; Qian and Yu, 2019; Wang et al., 2017; Li et al., 2021a). However, there is no recent work about domain adaptation for a TST, except DAST (Li et al., 2019a). DAST is a semi-supervised learning method that adapts domain vectors to adapt models learned from multiple source domains to a new target domain via domain discriminator. Different From DAST, we propose to combine meta-learning and adversarial networks to achieve similar domain adaption ability, and our model exceeds the performance of DAST without domain discriminator. Although there are some methods perform well in few shot data transfer (Riley et al., 2021; Krishna et al., 2021), these methods discuss completely new text style transfer, while we focus on the domain adaptation issue.
2.3 Model-Agnostic Meta-Learning
Model-agnostic meta-learning (MAML) (Finn et al., 2017) provides a general method to adapt to parameters in different domains. MAML solves few-shot learning problems by learning a good parameter initialization. During testing, such initialization can be fine-tuned through a few gradient steps, using a limited number of training examples in the target domain. Although there have been some researches (Qian and Yu, 2019; Li et al., 2020; Wu et al., 2020) on MAML in natural language
processing, it is still scarce compared to computer vision. Unlike the above research on classification under few-shot learning, our research focuses on text style transfer based on text generation. In this paper, we seek a new meta-learning strategy combined with adversarial networks, which is more suitable for encouraging robust domain representation. As far as we know, we are the first to adopt meta-learning in the domain adaptation problem of text style transfer tasks.
In this section, we first define the problem of domain adaptive learning for TST. Then we describe our approach, DAML-ATM, in detail.
Let  $D_{S} = \{D_{1},\dots,D_{N}\}$  be  $N$  source domains in the training phase, where  $D_{n}(1\leq n\leq N)$  is the  $n$ -th source domain containing style-labelled non-parallel data  $D_{n} = \{(X_{i},l_{i})\}_{i = 1}^{L_{n}}$ , where  $L_{n}$  is the total number of sentences,  $X_{i}$  denotes the  $i^{th}$  source sentence, and  $l_{i}$  denotes the corresponding style label, which belongs to a source style label set:  $l_{i}\in L_{S}$  (e.g., positive/negative). Likewise, there are  $K$  target domains  $D_{T} = \{D_{1},\dots,D_{K}\}$  which are unseen in  $D_{S}$ . Our task is to transfer a sentence  $X_{i}$  with style  $l_{i}$  in the target domain to another sentence  $Y_{i}^{\prime}$  sharing the same content while having a different style  $\tilde{l}_i$  from  $l_{i}$  and domain-specific characteristics of the target domain.
We propose a two-stage algorithm for domain adaptation in TST: pre-training learning strategy and domain adaptive meta-learning strategy. In pre-training learning, our objective is to make the model more able to preserve content information and distinguish between different text styles. In domain adaptive meta-learning, our objective is to learn a meta-knowledge learner for the sequence-to-sequence model by leveraging sufficient source data  $D_{s}$ . Given a new unseen domain from  $D_{new}$ , the new learning task of TST can be solved by finetuning the learned sequence-to-sequence model (domain-invariant parameters) with only a small number of training samples.
3.2 DAML-ATM Approach
3.2.1 Overview of DAML
Model-agnostic meta-learning can utilize a few training samples to train a model with good generalization ability. However, since it is based on
Figure 1: Comparison of meta-learning and domain adaptive meta transfer learning (DAML). In DAML, each meta task contains  $n$  sentences from the same domain. In MAML, the data in each meta task come from different domains.
the assumption that the meta tasks are from the same distribution (Figure 1, left), simply feeding all the sources data into it might get sub-optimal results (Chen and Zhu, 2020). Therefore, we propose a modified way to construct meta tasks (Figure 1, right).
Different from MAML, for DAML, in one batch, the data in each meta task comes from the same source domain, and each meta task comes from a different domain. In this way, we can guarantee that DAML can learn generic representations from different domains in a balanced way. During each iteration, we randomly split all source domains into a meta-training set  $D_{tr}$  and a meta-validation set  $D_{val}$ , where  $D_{S} = D_{tr} \cup D_{val}$  and  $D_{tr} \cap D_{val} = \emptyset$ . A meta-training task  $T_{i}$  is sampled from  $D_{tr}$  and is composed of  $n$  instances from a specific domain. Likewise, a meta-validation task  $T_{j}$  is sampled from  $D_{val}$ . The validation errors on  $D_{val}$  should be considered to improve the robustness of the model. In short, with DAML, the parameters learned by the model in the parameter space are not biased towards any one particular domain with as little data as possible during model updating as shown in Figure 1(right).
In the final evaluation phase, the knowledge learned by the sequence-to-sequence model can be applied to new domains. Given a new unseen domain  $D_{new} = (T_{tr}, T_{te})$ , the learned sequence-to-sequence model and the discriminator are fine-tuned on  $T_{tr}$  and finally tested on  $T_{te}$ .
3.2.2 ATM Model
In this section, we give a brief introduction to our proposed model: ATM, which combines sequence-to-sequence pre-trained model with adversarial training. (1) For the content preservation, we train
Figure 2: The basic structure of our TST model, ATM, with the first stage training procedure. The green dashed line represents the loss of style classification to ensure that the style classifier can distinguish between different text styles. The black dotted line rerents text reconstruction loss to ensure the generated sentence has a similar semantic meaning as the input sentence.
the sequence-to-sequence model  $\theta$  to reconstruct the original input sentence  $X$  with the original style label  $l$ . (2) For the style controlling, we train a discriminator network  $\gamma$  to assist the sequence-to-sequence model network in better controlling the style of the generated sentence. The structure of the model is shown in Figure 2.
S2S-model To ease the explanation, we start with the sequence-to-sequence (S2S) model here. Explicitly, for an input sentence  $X = (x_{1},x_{2},\dots,x_{n})$  of length  $n$ ,  $X \in D$ , the S2S encoder  $\operatorname{Enc}(X;\theta_E)$  maps inputs to a sequence of continuous hidden representations  $H = (h_1,h_2,\dots,h_n)$ . Then, the S2S decoder  $\operatorname{Dec}(H;\theta_D)$  estimates the conditional probability for the output sentence  $Y = (y_{1},y_{2},\dots,y_{n})$  by auto-regressively factorized its as:
$$
p _ {\theta} (Y | X) = \prod_ {t = 1} ^ {n} p _ {\theta} \left(y _ {t} \mid H, y _ {1}, \dots , y _ {t - 1}\right) \tag {1}
$$
At each time step  $t$ , the probability of the next token is computed by a softmax classifier:
$$
p _ {\theta} \left(y _ {t} \mid H, y _ {1}, \dots , y _ {t - 1}\right) = \text {s o f t m a x} \left(o _ {t}\right) \tag {2}
$$
where  $o_t$  is logit vector outputted by decoder network. The standard S2S model without discriminator makes the output sequence  $Y$  the same as the input sequence  $X$ .
Discriminator Model By teacher forcing, S2S tends to ignore the style labels and collapses to a reconstruction model, which might copy the input sentence, hence failing to transfer the style. Therefore, to make the model learn meaningful style
Algorithm 1 ATM Pre-training Learning
Input: sequence-to-sequence model  $f_{\theta}$ , discriminator  $\gamma$ , and a dataset  $D_{i}$  with style  $l_{i}$  belong to  $L_{s}$
Output: well-trained parameter  $\theta ,\gamma$
1: Sample a batch of m sentences  $X_{1},X_{2},\ldots X_{m}$  from  $D_{i}$
2: while in first stage and not convergence do
3: Use  $f_{\theta}$  to generate new sentence
4:  $Y_{i} = f_{\theta}(X_{i},l_{i})$
5: Compute  $\mathcal{L}_{cls}(\gamma)$  for  $Y_{i}$  by Eq. (4);
6: Compute  $\mathcal{L}_{rec}(\theta)$  for  $Y_{i}$  by Eq. (3);
information, we apply a style discriminator  $\gamma$  for the style regularization. In summary, we use a style discriminator to provide the direction (gradient) for TST to conform to the target style. Our discriminator is a multi-layer perceptron with a sigmoid activation function to predict style labels or guide the direction of style transfer. Our model training involves a pre-training learning strategy and a domain adaptive meta-learning strategy.
3.2.3 First Stage: Pre-training Learning
In the first stage, we train the discriminator model to distinguish different text styles. In this stage, the discriminator models are equivalent to a text classifier. Inspired by (Lewis et al., 2019), we feed the hidden states from the last layer of the decoder into the classifier instead of the gumblesoftmax trick (Jang et al., 2017) for gradient backpropagation, which is more stable and better than gumblesoftmax(See Table 5). The loss function for the discriminator is simply the cross-entropy loss of the classification problem:
$$
\mathcal {L} _ {c l s} (\gamma) = - \underset {X _ {i} \sim D _ {S}} {\mathbb {E}} [ \log P (l _ {i} | X _ {i}, l _ {i}; \theta , \gamma) ] \tag {3}
$$
For the S2S model, we pre-train the S2S model to allow the generation model to learn to copy an input sentence  $X$  using teacher forcing. The loss function of the sequence-to-sequence model minimizes the negative log-likelihood of the training data:
$$
\mathcal {L} _ {\text {r e c}} (\theta) = - \underset {X _ {i} \sim D _ {S}} {\mathbb {E}} [ \log P (Y _ {i} | X _ {i}; \theta) ] \tag {4}
$$
In summary, we train the sequence model and the style classification model separately on the source domain to learn content preservation and style discrimination in the first stage. The first stage training procedure of the ATM is summarized in Algorithm 1.
Figure 3: Overview of our proposed DAML-ATM with second stage training strategy. In the meta-training phase, a temporary model  $(\theta_{old},\theta_{new})$  is learned from  $D_{tr}$ . In the meta-validation phase, the base model is updated by gradient descent with respect to the parameters  $\theta$  on  $D_{val}$ . In the final evaluation phase, the learned sequence encoder is fine-tuned on  $T_{tr}$  and tested on  $T_{te}$  from a unseen domain  $D_{new}$ .
3.2.4 Second Stage: Domain Adaptive Meta Learning with Adversarial Training
After the first stage of training, the style classifier has learned how to distinguish between different text styles. For style controlling, we adopt a method of adversarial training to avoid disentangling the content and style in the latent space. The discriminator model aims to minimize the negative log-likelihood of opposite style  $\tilde{l}_i$  when feeding to the sequence model sentence  $X_i$  with the style label  $l_i$ . In the second stage, we freeze the parameters of the discriminator. Therefore, style loss only works on the S2S model  $\theta$ , which forces the S2S model  $\theta$  to generate opposite styles of sentences:
$$
\mathcal {L} _ {\text {s t y l e}} (\theta) = - \underset {X _ {i} \sim D} {\mathbb {E}} \left[ \log P \left(\tilde {l} _ {i} \mid X _ {i}, l _ {i}; \theta , \gamma\right) \right] \tag {5}
$$
In the second stage, we use the DAML algorithm for domain adaptive TST, so the text reconstruction loss and the style discriminator loss are calculated over the meta-training samples in task  $T_{i}$  from  $D_{tr}$ . These two losses can be written as
$$
\begin{array}{r} \mathcal {L} _ {T _ {i}} ^ {\text {r e c}} (\theta) = - \underset {X _ {i} \sim T _ {i}} {\mathbb {E}} [ \log P (Y _ {i} | X _ {i}; \theta) ] \\ \mathcal {L} _ {T _ {i}} ^ {\text {s t y l e}} (\theta) = - \underset {X _ {i} \sim T _ {i}} {\mathbb {E}} [ \log P (\tilde {l} _ {i} | X _ {i}, l _ {i}; \theta , \gamma)) \end{array} \tag {6}
$$
We add different prefixes to the input in the second stage, which allows the S2S model to perceive
different TST tasks. The second stage of the algorithm is called domain adaptive meta-strategy, which consists of two core phases: a meta-training phase and a meta-validation phase, as shown in Figure 3.
Domain Adaptive Meta-Training
In the meta-training phase, our objective is to learn different domain-specific temporary models for each domain that are capable of learning the general knowledge of each domain. Inspired by feature-critic networks (Li et al., 2019b), we use a similar manner to adapt the parameters of the domain-specific temporary model:
$$
\theta_ {i} ^ {\text {o l d}} = \theta_ {i - 1} - \alpha \nabla \theta_ {i - 1} \mathcal {L} _ {T _ {i}} ^ {\text {r e c}} \left(\theta_ {i - 1}, \gamma_ {i - 1}\right) \tag {7}
$$
$$
\theta_ {i} ^ {n e w} = \theta_ {i - 1} ^ {o l d} - \alpha \nabla \theta_ {i - 1} \mathcal {L} _ {T _ {i}} ^ {s t y l e} (\theta_ {i - 1}, \gamma_ {i - 1})
$$
where  $i$  is the adaptation step in the inner loop, and  $\alpha$  is the learning rate of the internal optimization. At each adaptation step, the gradients are calculated with respect to the parameters from the previous step. For each domain of  $D_{tr}$ , it has different  $\theta^{old}$  and  $\theta^{new}$ . The base model parameters  $\theta_0$  should not be changed in the inner loop.
Algorithm 2 The training procedure of DAML-ATM
Input: D = {D1, ..., DK}, α, β
Output: optimal meta-learned model θ
1: Initialize the base sequence-to-sequence model θ and discriminator model γ by algorithm 1
2: while not converge do
3: Randomly split D = Dt ∪ Dval and Dt ∩ Dval = ∅
4: Meta-training:
5: for j in meta batches do //Outer loop
6: Sample a task Tj from Dval
7: for i in adaptation steps do //Inner loop
8: Sample a task Ti from Dt
9: Compute meta-training rec loss LrecTi
10: Compute meta-training style loss style
11: Compute adapted parameters with gradient descent for θi-1
12: θiold = θi-1 - α∇θi-1LtrTi(θi-1, γi-1)
13: θinew = θi-1old - α∇θi-1LstyleTi(θi-1, γi-1)
14: Meta-validation:
15: Compute meta-validation loss on Tj: LvalTi
16: Meta-optimization:
17: Perform gradient step w.r.t. θ
18: θ0 = θ0 - β∇θ0ETjLvalTi(θiold, θi new, γ)
Domain Adaptive Meta-Validation
After meta-training phase, DAML-ATM has already learned a temporary model  $(\theta_{i}^{old},\theta_{i}^{new})$  in the meta-training domains  $D_{tr}$ . The meta-validation phase tries to minimize the distribution divergence
between the source domains  $D_{tr}$  and simulated target domains  $D_{val}$  using the learned temporary model. In the meta-validation phase, each temporary model is calculated on the meta-validation domain  $D_{val}$  to get meta validation losses.
$$
\mathcal {L} _ {T _ {j}} ^ {v a l} = \mathcal {L} _ {T _ {j}} ^ {r e c} (\theta_ {i} ^ {o l d}, \gamma_ {0}) + \mathcal {L} _ {T _ {j}} ^ {s t y l e} (\theta_ {i} ^ {n e w}, \gamma_ {0}) \tag {8}
$$
Thus, the base model  $\theta$  is updated by gradient descent
$$
\theta_ {0} = \theta_ {0} - \beta \nabla \theta_ {0} \mathcal {L} _ {T _ {j}} ^ {\text {v a l}} \tag {9}
$$
where  $\beta$  is the meta-learning rate. Unlike the ordinary gradient descent process, the update mechanism of Eq. (9) involves updating one gradient by another gradient (w.r.t. the parameters of the temporary model). This process requires a second-order optimization partial derivative.
3.2.5 Final Evaluation Phase of DAML-ATM
In the final evaluation phase, we first initialize the model with the parameters learned during the above algorithm 2. Then, the model takes input as a new adaptation task  $T$ , which consists of a small indomain data  $S_{tr}$  for fine-tuning the model and a test set  $S_{te}$  for testing. The procedure is summarized in Algorithm 3. (Note that the discriminator is not needed for inference.)
Algorithm 3 The Final Evaluation Procedure of DAML-ATM
Input:  $\theta, \gamma$  learned from Algorithm 2, low resource training set  $S_{tr}$  and test set  $S_{te}$  of an unseen domain  $D_{new}$   
Output: Performance on  $S_{te}$
1: while not convergence do
2: Serialize a task  $T_{tr}$  from the unseen domain  $S_{tr}$
3: Update  $\theta = \theta -\beta \nabla_{\theta}\sum_{T_{tr}}(\mathcal{L}_{T_{tr}}^{rec}(\theta) + \mathcal{L}_{T_{tr}}^{style}(\theta))$
4: return optimal  $\theta^{*}$  for  $S_{te}$
5: Style accuracy, bleu, domain accuracy =  $f_{T_{te}}(\theta)$
Dataset	Domain	Train	Dev	Test	Human Reference
Yelp	Restaurant	444k	4k	1k	1k
Amazon	Product	554k	2k	1k	1k
IMDB	Movie	341k	2k	1k	No
Yahoo!	Q & A	5k	1k	1k	No
Table 1: Statistics of source and target datasets(non-parallel data). The style label set is {negative, positive}.
In this section, we first detail the experimental setups. Then, we present our experimental results over multiple target domains.
4.1 Datasets and Experimental Setups