SlowGuess/NAACL_2022_OCR huggingface dataset:
Sharan Narasimhan Suvodip Dey Maunendra Sankar Desarkar
Indian Institute of Technology Hyderabad, India
sharan.n21@gmail.com suvodip15@gmail.com maunendra@cse.iith.ac.in
Recent studies show that auto-encoder based approaches successfully perform language generation, smooth sentence interpolation, and style transfer over unseen attributes using unlabeled datasets in a zero-shot manner. The latent space geometry of such models is organised well enough to perform on datasets where the style is "coarse-grained" i.e. a small fraction of words alone in a sentence are enough to determine the overall style label. A recent study uses a discrete token-based perturbation approach to map "similar" sentences ("similar" defined by low Levenshtein distance/ high word overlap) close by in latent space. This definition of "similarity" does not look into the underlying nuances of the constituent words while mapping latent space neighbourhoods and therefore fails to recognise sentences with different style-based semantics while mapping latent neighbourhoods. We introduce EPAAEs (Embedding Perturbed Adversarial AutoEncoders) which completes this perturbation model, by adding a finely adjustable noise component on the continuous embeddings space. We empirically show that this (a) produces a better organised latent space that clusters stylistically similar sentences together, (b) performs best on a diverse set of text style transfer tasks than similar denoising-inspired baselines, and (c) is capable of fine-grained control of Style Transfer strength. We also extend the text style transfer tasks to NLI datasets and show that these more complex definitions of style are learned best by EPAAE. To the best of our knowledge, extending style transfer to NLI tasks has not been explored before.
The Text Style transfer (TST) task is a form of controlled language generation. The goal is to produce fluent style-altered sentences from a given base sentence, while also preserving its style-independent
content. The definition of "style" depends on the class labels of the end task. Our work focuses on the unsupervised scenario i.e. performing the training on completely unlabelled corpora. By inducing the latent space organization through input perturbation, TST can be performed using a simple vector arithmetic method (discussed in Section 6).
Background. Several well-known architectures use auxiliary objectives that serve as regularizers to ensure that the latent space geometry is smoothly interpolatable and learns high-level semantic features (refer Appendix D). Inspired by successes in denoising approaches in vision (Creswell and Bharath, 2019; Vincent et al., 2008), we look at input perturbation based approaches for unsupervised style transfer. Unlike text, in vision, freedom exists to finely control the degree of Gaussian "blur" on the continuous input image space. We conjecture that this notion of "controlling blur" may be useful in the text domain as well, serving as our motivation for our chosen model for Input Perturbation. Shen et al. (2020) map "similar" (defined by low Levenshtein distance/ high word overlap) sentences together in the latent space by introducing a simple denoising objective over an underlying Adversarial Autoencoder (AAE) (Makhzani et al., 2015). This noise model includes simple token manipulation
Figure 1: A TSNE plot of encoded latent vectors for all 1950 sentences in the Toy Dataset. EPAAE shows tighter and more organised clustering. Refer to B.1 and B.2 for larger plots with legend.
*Our code, data and outputs are available at https://github.com/sharan21/EPAAE
Human Language Technologies, pages 475 - 493
(token dropout and substitution) with some probability  $p$ , to reconstruct the original sentences from the perturbed inputs. We can reason intuitively that discrete word dropout allows sentences with high word overlap (or low Levenshtein distance) to have a higher chance of being perturbed into one another, thereby mapping them close in the latent space during the training time. However, this method allows for stylistically dissimilar sentences, albeit with high word overlap, to be mapped together in the latent space.
**Idea.** We argue that this negatively impacts the quality of final latent geometry and the results of the subsequent style transfer task. As an alternative, we explore Embedding Perturbation, where a noise vector is sampled, appropriately scaled and added to the embeddings of each input sentence, such that the resultant embeddings are constrained to live inside an E-dimensional hypersphere. The radius of this hypersphere is controllable using a tune-able hyper-parameter  $\zeta$ . This hypersphere constraint is partially inspired by concepts in adversarial robustness and ensures that each resultant noised word embedding is not altered to the extent of causing the underlying semantics of the sentence to change. We argue that this allows sentences with stylistically similar constituent embeddings to be mapped together, also encouraging the formation of style-preserving latent neighbourhoods (more on this in Section 4). The resulting latent representation from "embedding-perturbed" autoencoders consequently are better semantically organized and stylistically robust, enabling us to perform TST using an inductive method i.e. simple vector arithmetic on latent vectors.
Contributions. We show that this extended model of input perturbation with both discrete and continuous components, allows for overall better quality text style transfer, particularly in its ability to preserve style-independent "content" information. To expand the traditional definitions of styles such as Polarity, Formality, which are based on simple "intra-sentence" attributes, we introduce the "Discourse Style Transfer Task" by salvaging NLI datasets in which the flow of logic between sentences are captured using "Entailment", "Contradiction", and "Neutral" labels. This enables interesting applications such as discourse manipulation in which a pair of sentences agreeing with each other can be made to contradict, and vice versa. We also test our model on fine-grained styles present
in the Style-PTB dataset. We empirically show that our model performs the best on a diverse set of datasets with styles ranging from coarse-grained styles (like sentiment) to fine-grained styles (like tenses) and complex inter-sentence styles (like discourse or flow of logic).
Seminal work in TST. Autoencoder based approaches for TST on labelled non-parallel datasets are quite well explored (Shen et al., 2017; Fu et al., 2018). Some techniques involve implicit Style-Content disentanglement of the latent space using Back Translation (Prabhumoye et al., 2018) and adversarial learning (John et al., 2019). Li et al. (2018) achieve disentanglement using simple keyword replacement. Most studies look at simple non-parallel classification datasets, defining their style to be the class label. Studies also look at Syntax-Semantic disentanglement of the latent space (Chen et al., 2019; Bao et al., 2019). A  $\lambda_{1}$  penalty is imposed on the log variance of the perturbations to prevent it from vanishing. The latent vacancy problem (Xu et al., 2020) of the  $\beta$ -VAE is mitigated by introducing auxiliary losses and provided for one of the earliest methods for unsupervised TST. Similar to our work, Rubenstein et al. (2018) introduces the Latent noised AAE (LAAE), Gaussian perturbation is instead added to latent encodings to promote organization. Unsupervised work includes using a language model as a discriminator for a richer feedback mechanism (Yang et al., 2018), allowing it to increase performance in word substitution decipherment, sentiment modification, and related language translation. More seminal work related to autoencoders in the context of Style Transfer is mentioned in Appendix D.
Contemporary work in TST. More recent work, treat the style transfer problem as paraphrase generation and fine-tune pre-trained language models (Krishna et al., 2020). Malmi et al. (2020) trains masked language models or MLMs on the source and target domains to identify input tokens to be removed and replace them with tokens from the target MLM in an unsupervised manner. Liu et al. (2020) uses gradient-based update rules in the continuous latent space  $z$  from style and content predictor networks, enabling the transfer of fine-grained styles without using an adversarial approach. Reid and Zhong (2021) performs TST on sentiment and politeness datasets using token editing methods
(similar to Li et al. (2018)) using Levenshtein editing operations. Lee et al. (2021) also focuses on enhancing content preservation by introducing a method to remove style at the token level using reverse attention and fuse this content representation with style using a conditional layer normalization technique. Riley et al. (2021) adapts the T5 model (Raffel et al., 2020) for few shot text style transfer by extracting a style representation and performing conditioned decoding, using only a handful of examples at inference time.
The underlying language model is a generative auto-encoder which models an input distribution  $p(\mathbf{x})$  assumed to be from an underlying latent distribution  $p(\mathbf{z})$ . A deterministic encoder  $E$  representing the distribution  $q(\mathbf{z}|\mathbf{x})$ , in the form of an RNN, whose output is reparameterized by another distribution  $z_{i}\sim \mathcal{N}(\mu_{i}(\mathbf{x}),\sigma_{i}(\mathbf{x}))$  to give the aggregated posterior distribution  $q(\mathbf{z})$ . Various auxiliary loss functions are used to enforce the learned prior  $q(\mathbf{z})$  to match  $p(\mathbf{z})$ . The Generator  $G$  representing  $p(\mathbf{x}|\mathbf{z})$ , also in the form of an RNN, decodes back the sample from the learnt prior  $q(\mathbf{z})$  into its corresponding input from  $p(\mathbf{x})$ . Gradient descent is applied on the reconstruction loss of the autoencoder given by:
$$
\mathcal {L} _ {\mathrm {r e c}} \left(\theta_ {E}, \theta_ {G}\right) = E _ {p _ {\mathrm {d a t a}} (x)} \left[ - \log p _ {G} (x \mid E (x)) \right] \tag {1}
$$
We use the AAE (Makhzani et al., 2015) as our choice for the underlying generative autoencoder over which the input perturbation techniques were applied. AAE uses a discriminator  $D$  to enforce  $q(\mathbf{z})$  to match  $p(\mathbf{z})$ , a standard Gaussian, by learning to distinguish between samples from the two different distributions. This adversarial loss serves as a regularizer for the latent space, giving it the ability the organize itself better for smooth sentence interpolation.
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {a d v}} \left(\theta_ {E}, \theta_ {D}\right) = \mathbb {E} _ {p (z)} \left[ - \log D (z) \right] + \\ \mathbb {E} _ {p _ {\text {d a t a}} (x)} [ - \log (1 - D (E (x))) ] \tag {2} \\ \end{array}
$$
The final min-max objective is a sum of the reconstruction loss (given below) and  $\lambda$  weighted adversarial loss:
$$
\min  _ {E, G} \max  _ {D} \mathcal {L} _ {\operatorname {r e c}} \left(\theta_ {E}, \theta_ {G}\right) - \lambda \mathcal {L} _ {\operatorname {a d v}} \left(\theta_ {E}, \theta_ {D}\right) \tag {3}
$$
We found empirically that AAEs performed well and were stable during training. On the other hand,  $\beta$ -VAEs required careful tuning of the  $\beta$  hyperparameter to prevent posterior collapse and did not perform as well.
3.1 Finely-controlled continuous noise on embedding space
Figure 2: An abstract representation of the continuous embedding noise approach for two arbitrary embeddings  $e_i$  and  $e_{i+1}$  on an embedding space  $E \in R^2$
To further organize the latent geometry of the underlying AAE to encode style-based semantic similarity among sentences, we propose a perturbation-based approach on the continuous embedding space. Word embeddings of dimensionality  $E$ , of each input token  $x_{i}$  are denoted as  $e_{i}$ . Consider an input sentence of length  $l$  containing tokens  $x_{0},\dots ,x_{l - 1}$  having embeddings  $e_0,\dots ,e_{l - 1}$  respectively. Our objective is to blur every embedding vector  $e_{i}$  by adding some appropriately scaled noise vector  $n_i$  to produce a resultant noised embedding vector  $e_i^\prime$ , such that  $e_i^\prime$  does not lie too far away from  $e_i$  to not change the underlying semantics of the word completely. We do this by ensuring that each new  $e_i^\prime$  lives inside an  $E$ -dimensional hypersphere. The center of the hypersphere is the original embedding  $e_i$  and its radius is defined as  $|e_i|*s_i$ , where  $s_i\in \mathbb{R}^1$  is a random variable sampled from a distribution  $P(s)$ .  $P(s)$  is probability distribution of the form  $\mathcal{V}(\mu = 0,\sigma)$  where  $\mathcal{V}$  is some arbitrarily chosen distribution and  $\sigma$  as function of hyperparameter  $\zeta \in R$ . This distribution  $P(s)$  parameterized by  $Y,\zeta$  models the probability density cloud inside the embedding hypersphere, controlling its radius and interior densities. Figure 1 neatly summarizes the aforementioned embedding perturbation mechanism for a simple example with only two individual word embeddings. In
practice, we use a vectorized representation of the above mechanism to blur the embeddings of a minibatch of sentences of size  $L$  in constant  $O(1)$  time. The embedding perturbation method is summarized below in a vectorized form:
$$
z \sim \mathcal {N} (\mu , \sigma^ {2})
$$
$$
\hat {z} = z / | z |
$$
$$
\begin{array}{l} s \sim Y (\mu = 0, \sigma^ {2} = (\zeta / 3) ^ {2}) \tag {4} \\ n = \left(s \odot | e |\right) ^ {T} \hat {z} \\ e ^ {\prime} = e + n \\ r = \zeta * | e | \\ \end{array}
$$
where  $z, \hat{z}, s, n$  and  $e$  are vectorized representations of  $z_{i}, \hat{z}_{i}, s_{i}, n_{i}$  and  $e_{i}$  respectively for a mini-batch of sentences of size  $l$ .  $\odot$  and  $*$  denote element wise and scalar multiplication respectively.  $|x|$  denotes the magnitude of a vector/batch of vectors  $x$ .  $r$  is the vectorized mini-batch representation of  $r_{i}$ , the expected radius of hypersphere  $i$ . We choose the Gaussian distribution as our choice of the probability distribution  $\mathcal{V}$ , as on testing it produced the best results†. To couple the radius  $r_{i}$  of hypersphere to our hyperparameter  $\zeta$  to enable fine-grained control using  $\zeta$ , we constrain the probability density of  $Y$  to live inside the hypersphere within three standard deviations. To achieve this, we equate  $3\sigma$  to  $\zeta$ , and consequently set the variance  $\sigma^2$  of  $\mathcal{V}$  to be  $(\zeta / 3)^2$ .
3.2 Discrete word dropout probability
The "Denoising Autoencoder" or DAAE (Shen et al., 2020) considered discrete token-level perturbations such as token masking, substitution and deletion. We consider token deletion as discrete noise. Drop probability  $p = 0.3$  is found to be the best in both their experiments and ours.
Token deletion is the only type of input perturbation that can alter the number of tokens in a sentence. Any continuous model for input noise cannot mathematically generalize the effects of this kind of discrete word dropout. Furthermore, during our experiments, we find that for some datasets, both discrete and continuous noise components are required to produce the overall best model. In such a case, we first perform token deletion and then subject the leftover token embeddings to perturbation. We refer to this generalized noise model as
"Embeddings-Perturbed Adversarial Autoencoder" or "EPAAE" parameterized by  $\zeta$  and  $p$ .
4 Semantic Similarity in Latent Space Neighbourhoods
We contrast and compare the resultant latent space neighbourhoods of DAAEs and EPAAEs.
4.1 Preliminary reasoning
Here, we first investigate the question - Does token deletion during training group truly put semantically similar sentences together in latent space  $Z$ ? Intuitively we can reason that the answer may be in the affirmative if the drop probability is small. For example, the sentences "The food was good" and "The food was superb" might get perturbed into a common version, i.e. "The food was" and therefore be mapped nearby in  $Z$  during training. As analysed and concluded in (Shen et al., 2020), latent neighbourhoods in the latent space of DAAEs successfully cluster sentences with high word overlap (low Levenshtein/Edit distance) together. However, the Levenshtein distance metric is not an accurate measure of the true semantic similarity between sentences. A pair of sentences with high word overlap might convey different ideas w.r.t the underlying style-based semantics of the dataset. For example, in the context of the Yelp dataset, the sentences "The food was good" and "The food was bad" are stylistically opposing (style being polarity in this case) and yet still get mapped close by in  $Z$  for DAAEs.
4.2 Testing the hypothesis with a Toy Dataset
We conduct specifically curated experiments on a synthetic dataset to verify our hypothesis that EPAAEs map stylistically similar sentences together.
Details of Toy dataset: Inspired by Yelp, each sentence in this dataset either represents a positive or negative sentiment. It also contains different sentiment independent components, such as the identity of the person and the subject of the review. Each sentence is of the format: "The <identity_token> said the <subject_token> is <decision_token>" where identity, subject and decision token classes are the only variable parameters in each sentence. The entire set of all permutations of these token classes forms the dataset. For example, the decision class is further subdivided into two subclasses, i.e. positive/ negative sentiment,
We also tried the uniform distribution with varying values for  $\zeta$
Token Class	Sub Classes	Token Options
(0) Identity	(0) Male	boy, man, husband, boyfriend, waiter
(1) Female	girl, woman, wife, girlfriend, waitress
(1) Subject	(0) Food	food, meal, dinner, breakfast, lunch, pasta, chicken
(1) Others	service, staff, experience, atmosphere, location, place
(2) Decision	(0) Positive	good, great, excellent, decent, amazing, wonderful, reasonable
(1) Negative	bad, worst, horrible, spicy, bland, expensive, disgusting, mediocre
Table 1: Structure of token classes and division of subclasses with token options used to create Toy Dataset
DAAE (p = 0.3)	L2 Norm	EPAAE (ζ = 3.0)	L2 Norm
the man said the pasta is spicy	0	the man said the pasta is spicy	0
the man said the pasta is good	2.79	the man said the staff is spicy	2.62
the girl said the pasta is spicy	2.88	the man said the pasta is bad	2.69
the man said the pasta is wonderful	2.89	the man said the pasta is bland	2.77
the man said the service is spicy	3	the girl said the pasta is spicy	2.84
the man said the pasta is bad	3.02	the husband said the pasta is spicy	2.89
each subclass containing 7 and 8 choosable tokens, respectively. The resultant dataset consisted of 1950 sentences. The details of the subclasses and the representative tokens inside each subclass are shown in Table 1. We produce output labels for each sentence by using a 3-bit representation, one bit for each of three token classes, where the values of each bit represent the token label subclass within that class. For example, a label of 5 is encoded as 101 corresponding to a sentence with Female, Food and Negative labels. There are  $2^{3} = 8$  labels in total labelled 0-7.
Qual. Analysis of Latent space. We consider two models, DAAE with token deletion probability  $p = 0.3$  and EPAAE with  $\zeta = 2.5$ . Both models are initially pre-trained over the unlabelled Yelp dataset. The synthetic dataset is used during inference time only. We encode all 1950 sentences into their respective latent space vectors and use TSNE plots to visualise the latent space of each model (Figure 1). We pick a random query sentence, e.g.
Table 2: Nearest Neighbour experiment performed on EPAAE and DAAE. Sentences in red indicate dissimilarity in semantics with respect to cluster center
Dataset	Model	Mean L2 Norm for Label Flip	Mean Hops for Label Flip
Toy Dataset	DAAE	5.725	14.979
EPAAE	4.324	19.587
Yelp	DAAE	9.386	5.850
EPAAE	8.211	6.513
SNLI	DAAE	9.794	3.185
EPAAE	10.373	3.528
DNLI	DAAE	9.320	6.027
EPAAE	11.681	4.882
SciTail	DAAE	10.224	5.750
EPAAE	10.676	6.253
PP Removal	DAAE	8.195	6.722
EPAAE	8.774	7.409
Tenses	DAAE	7.686	8.802
EPAAE	10.924	5.426
Voices	DAAE	6.805	8.367
EPAAE	9.657	8.726
Table 3: Quantitative metrics to capture model's ability to preserve style-based semantics inside Latent Neighbourhoods. Bold indicates model with higher value for that corresponding metric in that specific dataset, which is suggestive of ability to better preserve style-based semantics.
"The man said the pasta is spicy", and encode it into  $Z$ . We then retrieve the top five nearest encoding to the query and observe their decoded outputs to check for the preservation of style-based semantics (Table 2) around the query. We find that the EPAAE maps the latent neighbourhood such that stylistically similar (positive/negative sentiment in this context) are grouped. This is not the case with the DAAE, evident from Table 2 in which neighbours 1 and 3 have a differing sentiment from the query. This offers an explanation as to why the DAAE is not able to produce tightly confined clusters.
Quant. Analysis of Latent space. To further validate our hypothesis that EPAAEs better preserve style-based semantics in latent neighbours, we generate useful quantitative metrics over a k-nearest neighbours experiment, done on the test split across all datasets. We document these metrics in Table 3. Column 3 conveys the mean distance to the closest neighbour with the opposite label. Column 4 conveys the mean number of hops to reach to the closest neighbour with opposite labels. In all but a few datasets, the DAAE reports a smaller "mean hops for label flip", supporting our hypothesis that stylistically dissimilar sentences are mapped closer together in the latent neighbour than our proposed model. We also see this trend mostly holds true for the "Mean L2 Norm for Label Flip" metric in Column 4 as well, providing further evidence to our hypothesis.
5 Setup, Datasets and Metrics
Baselines. This work focuses on simple denoising approaches for Unsupervised TST and subsequently constrains our choice of baselines to follow these criteria. We consider three other autoencoder based models for our experimentation, i.e. Denoising AAE (DAAE), Latent-Noised AAE (LAAE) and the  $\beta$ -VAE.
Hyperparameters and Setup. Details on hyperparameter selection can be found in A.2. Other common hyperparameters (detailed in A.1) related to encoder/decoder architectures remained identical across all models. Training is completely unsupervised, and labels are only used during inference time. Details on computation time, number of parameters and infrastructure used can be found in Appendix. E.
In this section, we briefly describe the different kinds of datasets used for experimentation. Further details is provided in in Appendix C.
Complexity of Styles in Datasets: Current studies mainly focus on high-level styles to validate the approaches. To validate our hypothesis that the EPAAE can perform fine-grained style transfer due to semantics learnt from embeddings, we consider three tasks: sentiment, discourse and fine-grained text style transfer. As prepossessing, we remove non-essential special characters and lowercase all sentences. Except for the Yelp dataset, no pruning is done based on sentence length.
Sentiment Style Datasets: We use the preprocessed version from (Shen et al., 2017) of the Yelp dataset. The sentiment labels (positive, negative) were considered as style.