SlowGuess/ACL_2020_OCR huggingface dataset:
Inkit Padhi, Pierre Dognin, Ke Bai‡, Cicero Nogueira dos Santos†* Vijil Chenthamarakshan, Youssef Mroueh, Payel Das
IBM Research,  $\ddagger$ Duke University,  $\dagger$ Amazon AWS AI
inkpad@ibm.com, ke.bai@duke.edu, cicnog@amazon.com
{pdognin,ecvijil,mroueh,daspa}@us.ibm.com
Generative feature matching network (GFMN) is an approach for training implicit generative models for images by performing moment matching on features from pre-trained neural networks. In this paper, we present new GFMN formulations that are effective for sequential data. Our experimental results show the effectiveness of the proposed method, SeqGFMN, for three distinct generation tasks in English: unconditional text generation, class-conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial approaches for text generation and text style transfer.
Generative feature matching networks (GFMNs) (dos Santos et al., 2019) has been recently proposed for learning implicit generative models by performing moment matching on features from pre-trained neural networks. This approach demonstrated that GFMN could produce state-of-the-art image generators while avoiding instabilities associated with adversarial learning. Similarly to training generative adversarial networks (GANs) (Goodfellow et al., 2014), GFMN training requires backpropagate through the generated data to update the model parameters. This backpropagation through the generated data, combined with adversarial learning instabilities, has proven to be a compelling challenge when applying GANs for discrete data such as text. However, it remains unknown if this is also an issue for feature matching networks since the effectiveness of GFMN for sequential discrete data has not yet been studied.
In this work, we investigate the effectiveness of GFMN for different text generation tasks. As a
first contribution, we propose a new formulation of GFMN for unconditional sequence generation, which we name Sequence-GFMN or SeqGFMN for short, by performing token level feature matching. SeqGFMN has a stable training because it does not concurrently train a discriminator, which in principle could easily learn to distinguish between one-hot and soft one-hot representations. As a result, we can use soft one-hot representations that the generator outputs during training without using the Gumbel softmax or REINFORCE algorithm as needed in GANs for text. Additionally, different from GANs (Zhu et al., 2018), SeqGFMN can produce meaningful text without the need of pre-training the generator with maximum likelihood estimation (MLE). We perform experiments using Bidirectional Encoder Representations from Transformers (BERT), GloVe, and FastText as our feature extractor networks. We use two different corpora, and assess both the quality and diversity of the generated texts with three different quantitative metrics: BLEU, Self-BLEU and Fréchet Inferent Distance (FID). Additionally, we show that the latent space induced by SeqGFMN contains semantic and syntactic structure, as evidenced by interpolations in the  $z$  space.
Our second contribution consists in proposing a new strategy for class-conditional generation with GFMN. The key idea here is to perform class-wise feature matching. We apply SeqGFMN to perform sentiment-based conditional generation using the Yelp Reviews dataset, and assess its performance using classification accuracy, BLEU, and Self-BLEU.
Finally, as a third contribution, we demonstrate that the feature matching loss is an effective approach to perform distribution matching in the context of unsupervised text style transfer (UTST). Most previous work on UTST adapts the autoencoder framework by adding an additional
*work done prior to joining Amazon
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3855-3863 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
loss term: adversarial loss or back-translation loss. Our method consists in replacing the adversarial and back-translation loss with style-wise feature matching. Our experimental results indicate that the feature matching loss produces better results than the traditionally used losses.
2 Feature Matching Nets for Text
2.1 SeqGFMN
Let  $G$  be a sequence generator implemented as a neural network with parameters  $\theta$ , and let  $E$  be a pretrained NLP feature extractor network with  $L$  hidden layers, that produces features at token-level for each token in a sequence of length  $T$ . The method consists of training  $G$  by minimizing the following token-level feature matching loss function:
$$
\min  _ {\theta} \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {M} \left| \left| \mu_ {p _ {d a t a}} ^ {j, t} - \mu_ {p _ {G}} ^ {j, t} (\theta) \right| \right| ^ {2} + \left| \left| \sigma_ {p _ {d a t a}} ^ {j, t} - \sigma_ {p _ {G}} ^ {j, t} (\theta) \right| \right| ^ {2} \tag {1}
$$
$$
\mu_ {p _ {d a t a}} ^ {j, t} = \mathbb {E} _ {x \sim p _ {d a t a}} E _ {j, t} (x) \in \mathbb {R} ^ {d _ {j}},
$$
$$
\mu_ {p _ {G}} ^ {j, t} (\theta) = \mathbb {E} _ {z \sim \mathcal {N} (0, I _ {n _ {z}})} E _ {j, t} (G (z; \theta)) \in \mathbb {R} ^ {d _ {j}},
$$
$$
\sigma_ {p _ {d a t a}, \ell} ^ {j, t} = \mathbb {E} _ {x \sim p _ {d a t a}} E _ {j, \ell , t} (x) ^ {2} - [ \mu_ {p _ {d a t a}} ^ {j, \ell , t} ] ^ {2},
$$
$$
\begin{array}{l} \sigma_ {p _ {G}, \ell} ^ {j, t} (\theta) = \mathbb {E} _ {z \sim \mathcal {N} (0, I _ {n _ {z}})} E _ {j, \ell , t} (G (z; \theta)) ^ {2} - [ \mu_ {p _ {G}} ^ {j, \ell , t} ] ^ {2}, \\ \ell = 1 \dots d _ {j}, \\ \end{array}
$$
where  $||.||^2$  is the  $L_{2}$  loss;  $x$  is a real data point sampled from the data distribution  $p_{data}$ ;  $z \in \mathbb{R}^{n_z}$  is a noise vector sampled from the normal distribution  $\mathcal{N}(0, I_{n_z})$ ;  $E_{j,t}(x)$  denotes the token-level  $t$  feature map at a hidden layer  $j$  from  $E$ ;  $M \leq L$  is the number of hidden layers used to perform feature matching;  $T$  is the maximum sequence length; and  $\sigma_{p_{data}}^2$  and  $\sigma_{p_G}^2$  are the variances of the features for real data and generated data respectively. Note that this loss function is quite different from both the MLE loss used in regular language models and the adversarial loss used in GANs.
In order to train  $G$ , we first precompute  $\mu_{pdata}^{j,t}$  and  $\sigma_{pdata,\ell}^{j,t}$  on the entire training data. During training, we generate a minibatch of fake data by passing the Gaussian noise vector through the generator. The fixed feature extractor  $E$  is used to extract features on the output of the generator at a per-token level. The loss is then computed, as mentioned in Eq. 1. The parameters  $\theta$  of the generator  $G$  are optimized using stochastic gradient
descent. Note that the network  $E$  is used for feature extraction only and is kept fixed during the training of  $G$ . Similar to (dos Santos et al., 2019), we use ADAM moving average, which allows us to use small minibatch sizes. Fig. 1 illustrates SeqGFMN training; note that we use mean matching only for brevity, in practice we match both mean and diagonal covariance.
In our SeqGFMN framework, the output of the generator  $G$  is a sequence  $\tilde{x}$  of soft one-hot representations,  $\{\tilde{w}_1,\tilde{w}_2,\dots,\tilde{w}_T\}$ , where each element  $\tilde{w}_i$  consists in the output of the softmax function at token  $i$ . In the feature extractor  $E$ , these soft one-hot representations are multiplied by an embedding matrix to generate soft embeddings, which are then fed to the following layers of  $E$ .
2.2 Class-Conditional SeqGFMN
Conditional generation is motivated by the assumption that if the training data can be clustered into distinct and meaningful classes, knowledge of such classes at training time would improve the overall performance of the model. For class-based text generation, some datasets provide such opportunity by labeling the training data with relevant classes (e.g., positive/negative sentiment for Yelp Reviews dataset), information that can be leveraged by our model to condition the generation.
For this to be effective, the extracted features used for SeqGFMN need to be sufficiently representative of the text generated yet still be different between classes. To account for the knowledge of latent classes, we extend the loss from Eq.1 for the case of two distinct classes:
$$
\begin{array}{l} \min _ {\theta} \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {M} | | \delta_ {c = 0} ^ {j, t} | | ^ {2} + | | \Delta_ {c = 0} ^ {j, t} | | ^ {2} + \\ \left| \left| \delta_ {c = 1} ^ {j, t} \right| \right| ^ {2} + \left| \left| \Delta_ {c = 1} ^ {j, t} \right| \right| ^ {2} \tag {2} \\ \end{array}
$$
where  $\delta_c^{j,t} = \mu_{p_{data}^c}^{j,t} - \mu_{p_G^c}^{j,t}(\theta)$  and  $\Delta_c^{j,t} = \sigma_{p_{data}^c}^{j,t} - \sigma_{p_G^c}^{j,t}(\theta)$  follows the same definition for means and variances as Eq.1, with the exception that they are now class-dependent. Given a class  $c$ , we allow for conditional generation by conditioning the noise vector  $z$  on  $c$ . Indeed, if  $z \sim \mathcal{N}(0, I_{n_z})$ , applying a class dependent linear transformation  $z_c = A_c z + b_c$  will change the noise distribution such that  $z_c \sim \mathcal{N}(b_c, A_c^\top A_c)$ .  $A_c$  and  $b_c$  are learned at training time so to minimize our loss. This enables the model to effectively sample
$$
\begin{array}{c} z _ {1} \\ \dots \\ z _ {N} \end{array} \longrightarrow \boxed {\mathrm {G e n e r a t o r}} \longrightarrow \begin{array}{c} \tilde {x} _ {1} \\ \mathrm {N N} \\ \tilde {x} _ {N} \end{array} \longrightarrow \boxed {\mathrm {F e a t u r e}} \longrightarrow \begin{array}{c} E _ {1, 1} (\tilde {x} _ {1}) \dots E _ {M, T} (\tilde {x} _ {1}) \\ \mathrm {e x t r a c t o r N N} \\ \dots \\ E _ {1, 1} (\tilde {x} _ {N}) \dots E _ {M, T} (\tilde {x} _ {N}) \end{array} \longrightarrow \mathcal {L} = \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {M} | | \boldsymbol {\mu} _ {\boldsymbol {p} _ {d a t a}} ^ {\boldsymbol {j}, t} - \frac {1}{N} \sum_ {i = 1} ^ {N} E _ {j, t} (\tilde {x} _ {i}) | | ^ {2}
$$
Figure 1: For each training iteration, Generator  $(G)$  outputs  $N$  sentences from noise signals  $z_{1}\dots z_{N}$ . A fixed feature extractor is used to extract token level features  $(E_{j,t})$  for the generated data.  $\mathcal{L}$  is the  $L_{2}$ -norm of the difference between extracted features means of generated and real data  $\pmb{\mu}_{\pmb{p}_{data}}^{j,t}$ , which is then backpropagated to update the parameters of  $G$ . The same strategy is used for variance terms in  $\mathcal{L}$  (here ignored for brevity).
a new input noise from distinct distributions, conditioned on the class  $c$ . Since the model can update the linear transformation parameters  $A_{c}$  and  $b_{c}$  to minimize its loss, the model can learn transformations that separate or disentangle between the different classes  $c$  naturally. For example, conditioning on sentiment where  $c = 0$  is the negative sentiment class and  $c = 1$  the positive class, amounts simply to learning two transformations  $(A_{0}, b_{0})$  and  $(A_{1}, b_{1})$ . This approach can be extended beyond learning linear transformations to allow for deep neural network to be employed. During training, a minibatch is composed of input noise samples conditioned on class  $c$ . Within our generator, we use a conditional batch normalization (condBN) from (Dumoulin et al., 2016). The conditional BN is a 2-stage process: First, we perform a standard BN of a minibatch regardless of  $c$  where  $y_{i} = \mathrm{BN}_{\gamma, \beta}(x_{i})$ , using notations from (Ioffe and Szegedy, 2015). Then  $y_{i}$  enters a second stage where  $w_{i} = \gamma_{c}y_{i} + \beta_{c}$  brings class dependency on  $c$  as proposed in (Dumoulin et al., 2016). This allows for the influence of class conditioning to carry over the whole model where conditional BN is used. Our models can have three distinct configurations: conditional input noise, conditional BN, or both conditional input noise and conditional BN.
2.3 Unsupervised Text Style Transfer (UTST) with SeqGFMN
Text style transfer consists of rewriting a sentence from a given style  $s_i$  (e.g., informal) into a different style  $s_j$  (e.g., formal) while maintaining the content and keeping the sentence fluent. The major challenge for this task is the lack of parallel data, and many recent approaches adapt the encoder-decoder framework to work with non-parallel data (Shen et al., 2017; Fu et al., 2018). This adaptation normally consists in using: (1) the reconstruction loss in an autoencoding fashion, which is intended to learn a conditional language model (decoder  $D$ ) while providing content preservation; together with (2) a classification loss produced by a style
classifier  $C$ , which is intended to guarantee the correct transfer. Balancing these two losses while generating good quality sentences is difficult, and several approaches such as adversarial discriminators (Shen et al., 2017) and cycle-consistency loss (Melnyk et al., 2017) have been employed in recent works. Here, we use feature matching as a way to alleviate this problem. Essentially, our unsupervised text style transfer approach is an encoder-decoder trained with the following three losses:
Reconstruction loss: Given an input sentence  $x^{s_i}$  from set  $X$  and its decoded sentence  $\hat{x}^{s_i} = D(E(x^{s_i}), s_i)$  (decoded in the same input style  $s_i$ ), the reconstruction loss measures how well the decoder  $D$  is able to reconstruct it:
$$
\mathcal {L} _ {\text {r e c}} = \mathbb {E} _ {x ^ {s _ {i}} \sim X} \left[ - \log p _ {D} \left(x ^ {s _ {i}} \mid E \left(x ^ {s _ {i}}\right), s _ {i}\right) \right]. \tag {3}
$$
Classification loss: This loss is formulated as :
$$
\begin{array}{l} \mathcal {L} _ {\text {c l a s s}} = \mathbb {E} _ {x ^ {s _ {i}} \sim X} \left[ - \log p _ {C} \left(s _ {i} \mid x ^ {s _ {i}}\right) \right] + \\ \mathbb {E} _ {\hat {x} ^ {s _ {i} \rightarrow s _ {j}} \sim \hat {X}} \left[ - \log p _ {C} \left(s _ {j} \mid \hat {x} ^ {s _ {i} \rightarrow s _ {j}}\right)\right]. \tag {4} \\ \end{array}
$$
where  $\hat{X}$  is the set of style transferred sentences generated by the current model. For the classifier, the first term provides supervised signal regarding style classification and the second term gives additional training signal from the transferred data, enabling the classifier to be trained in a semi-supervised regime. For the encoder-decoder the second term gives feedback on the current generator's effectiveness on transferring sentences to a different style.
Feature Matching loss: It is computed in a similar way as the class-conditional loss (Eq. 2). This loss consists of matching statistics of the features for each style separately. This means that when transferring from style  $s_i$  to  $s_j$ , we match the features of the resulting sentence with the features of real data that are from the target style  $s_j$ .
(Zhang et al., 2017a) proposes Adversarial Feature Matching for Text Generation by adding a reconstruction feature loss to the GAN objective. This
is different from our setup, as our discriminator is not learned, and our feature matching is per token and not on a global sentence level. Sequence GAN (SeqGAN) (Yu et al., 2017), MaliGAN (Che et al., 2017), and RankGAN (Lin et al., 2017) use a pretrained generator with MLE loss with a per token reward discriminator that is trained with reinforcement learning. SeqGFMN is similar to SeqGAN in the sense that it has a per token reward (per token feature matching loss). Still, it alleviates the need for pre-training the generator and the cumbersome training of a discriminator by relying on a fixed, state-of-the-art, text feature extractor such as BERT. Due to the discrete nature of the problem, training implicit models is tricky (de Masson d'Autume et al., 2019), which is addressed by using REINFORCE, actor-critic methods (Fedus et al., 2018), and Gumbel softmax trick(Kusner and Hernandez-Lobato, 2016).
For unsupervised text style transfer, different adaptations of the encoder-decoder framework have been proposed recently. (Shen et al., 2017; Fu et al., 2018) uses adversarial classifiers to decode to a different style/language. (Melnyk et al., 2017), (Nogueira dos Santos et al., 2018) proposed a method that combines a collaborative classifier with the back-transfer loss. (Prabhumoye et al., 2018) presented an approach that trains different encoders, one per style, by combining the encoder of a pre-trained NMT and style classifiers. The main difference between our approach and these previous work consists in the fact that we use the feature matching loss to perform distribution matching.
Datasets: We evaluate our proposed approach on three different english datasets: MSCOCO (Lin et al., 2014), EMNLP 2017 WMT News dataset (Bojar et al., 2017), and Yelp Reviews Dataset (Shen et al., 2017). Both COCO and WMT News datasets are used for unconditional models, while Yelp Reviews is employed to evaluate class-conditional generation and unsupervised text style transfer.
Feature Extractors for Textual Data: We experiment with different feature extractors that generate token-level representations. We use word embeddings from GloVe (Pennington et al., 2014) and FastText (Bojanowski et al., 2017) as representatives of shallow (cheap-to-train) architectures. As
a representative of large, deep feature extractor we use BERT (Devlin et al., 2018). Devlin et al. (2018) demonstrated that the features extracted by BERT can boost the performance of diverse NLP tasks. Our hypothesis is that BERT features are informative enough to allow the training of (cross-domain) text generators with the help of feature matching.
Metrics: In order to evaluate the diversity and quality of texts of the unconditional generators we use three metrics BLEU (Papineni et al., 2002), Self-BLEU(Zhu et al., 2018) and Fréchet Inferent Distance, FID(Heusel et al., 2017). Additionally, for class-conditional generation and unsupervised text style transfer, we report accuracy scores from a CNN sentiment classifier trained on the Yelp.
Unconditional Text Generation: In Tab. 1, we show quantitative results for SeqGFMN trained on COCO and WMT News using different feature extractors. As expected, BERT as a feature extractor gives better performance because of a more significant and richer features used.
We also present a comparison with other implicit generative models for text generation from scratch. We compare SeqGFMN with five different GAN approaches: SeqGAN (Yu et al., 2017), MaliGAN (Che et al., 2017), RankGAN (Lin et al., 2017), TextGAN (Zhang et al., 2017a) and RelGAN (Weili Nie and Patel, 2019). We do not use generator pre-training for any of the models. As reported in Tab. 1, SeqGFMN outperforms all GAN models in terms of BLEU and FID. The combination of low BLEU and low Self-BLEU for the different GANs indicates that the learned models generate random n-grams that do not appear in the test set. All GANs fail to learn reasonable models due to the challenges of learning a discrete data generator from scratch under the min-max game. Whereas, SeqGFMN can learn suitable generators without the need of generator pre-training.
Class-conditional Generation: Conditional generation experiments were conducted on Yelp Reviews dataset with sentiment labels (178K negative, 268K positive). For this experiment, we first pre-trained the Generator using a conditional denoising AE where class labels are provided only to the decoder  $D$ . The architecture of the encoder is the same as in (Zhang et al., 2017b) with three strided convolutional layers. Once pre-trained,  $D$  is used as initialization for our Generator  $G$ . The training is similar
Model	BLEU-2	BLEU-3	BLEU-4	BLEU-5	Self-BLEU	FID
COCO	Real Data	0.721	0.494	0.308	0.194	0.487	3.559
SeqGAN	0.044	0.019	0.012	0.010	0.026	13.167
MaliGAN	0.042	0.017	0.011	0.008	0.032	15.855
RankGAN	0.039	0.016	0.010	0.008	0.023	15.502
TextGAN	0.034	0.015	0.010	0.008	0.624	17.275
RelGAN	0.230	0.055	0.026	0.017	0.811	13.948
SeqGFMN (FastText)	0.389	0.153	0.089	0.059	0.644	6.371
SeqGFMN (Glove)	0.403	0.139	0.077	0.053	0.655	6.218
SeqGFMN (BERT)	0.695	0.476	0.277	0.186	0.802	5.610
WMT News	Real Data	0.852	0.596	0.356	0.199	0.289	0.365
SeqGAN	0.008	0.004	0.003	0.003	0.088	8.731
MaliGAN	0.070	0.021	0.012	0.008	0.018	9.057
RankGAN	0.188	0.055	0.024	0.015	0.973	12.306
TextGAN	0.053	0.018	0.010	0.008	0.644	9.945
RelGAN	0.076	0.026	0.015	0.012	0.451	8.809
SeqGFMN (FastText)	0.364	0.102	0.045	0.028	0.787	3.761
SeqGFMN (Glove)	0.385	0.106	0.047	0.029	0.735	4.033
SeqGFMN (BERT)	0.760	0.464	0.204	0.096	0.888	3.530
to the previous section except now sentiment class labels are passed to  $G$ , and class-dependent statistics of BERT features are used, as described in 2.2.
Table 1: Quantitative results for different implicit generators trained from scratch.
Model	Accu.	Class	BLEU3	Self-BLEU3
Baseline	-	-	0.415	0.509
Conditional	0.746	0	0.473	0.498
Noise+BN	1	0.413	0.472
Cond. BN	0.745	0	0.423	0.473
1	0.395	0.505
Cond. Noise	0.495	0	0.413	0.458
1	0.412	0.470
Tab. 2 presents results for our regular model (baseline) and the three conditional generators: Cond. Noise, Cond. Batch Normalization (BN), Cond. Noise+BN. We use 10K generated sentences for each sentiment class to compute classification accuracy. In terms of accuracy and BLEU-3 score, the Cond. Noise+BN model provides the best generator as it is able to capture and leverage the class information.
Unsupervised Text Style Transfer (UTST): In Table 3, we report BLEU and accuracy scores for SeqGFMN and six baselines: BackTranslation (Prabhumoye et al., 2018), which uses back-transfer loss; CrossAligned (Shen et al., 2017), MultiDecoder (Fu et al., 2018), and StyleEmbedding (Fu et al., 2018), which use adversarial loss; and TemplateBased (Li et al., 2018) and Del-Retrieval (Li et al., 2018), which uses rule-based methods. The BLEU score is computed between the transferred
sentences and the human-annotated transferred references, similar to (Li et al., 2018). And, the accuracy is based on our pre-trained classifier. Compared to the other models, SeqGFMN produces the best balance between BLEU and accuracy. Additionally, if we use back-transfer loss together with feature matching loss  $(SeqGFMN + BT)$  our model gets a significant improvement on both metrics.
Table 2: Comparison between Sentiment-dependent and class-agnostic (unconditional) SeqGFMN models.
Model	BLEU	Accuracy
BackTranslation	2.5	95.7
CrossAligned	9.1	74.1
MultiDecoder	14.6	50.1
StyleEmbedding	21.1	9.2
TemplateBased	22.6	81.1
Del-Retrieval	16.0	88.2
SeqGFMN	23.7	92.9
SeqGFMN + BT	24.5	96.4
Table 3: Comparison between SeqGFMN and other models for unsupervised text style transfer.
We presented new implicit generative models based on feature matching loss that are suitable for unconditional and conditional text generation. Our results demonstrated that backpropagating through discrete data is not an issue for the training via matching distributions at the token level. SeqGFMN can be trained from scratch without the need for RL or Gumbel Softmax. This approach has allowed us to create effective models for unconditional generation, class-conditional generation, and unsupervised text style transfer. We believe this work opens a new competitive avenue in the area of implicit generative models for sequential data.