SlowGuess/NAACL_2022_OCR huggingface dataset:
Ghazi Felhi, Joseph Le Roux  
LIPN
Université Sorbonne Paris Nord - CNRS UMR 7030
F-93430, Villetaneuse, France
{felhi, leroux}@lipn.fr
Djamé Seddah
INRIA Paris
django.seddah@inria.fr
We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers.
In the attention of Transformers, keys handle information selection while values specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values.
We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syntax transfer capabilities when compared to supervised models and that comparable supervised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available<sup>1</sup>.
Disentanglement, a process aimed at obtaining neural representations with identified meaning, is a crucial component of research on interpretability (Rudin et al., 2022). A form of disentanglement that received a lot of interest from the NLP community is the separation between syntax and semantics in neural representations (Chen et al., 2019; Bao et al., 2019; Zhang et al., 2019; Chen et al., 2020; Huang and Chang, 2021; Huang et al., 2021). Previous works perform disentanglement using paraphrase pairs as information for semantics, and/or constituency parses as information for syntax. The
dependence of models on labeled data is known to entail high cost (see Seddah et al., 2020 on syntactic annotation), and to often require new labels to handle problems such as concept drift (Lu et al., 2019) and domain adaptation (Farahani et al., 2021).
In light of the above, we propose an unsupervised model which directs syntax and semantics into different neural representations without semantic or syntactic information. In the Transformer architecture (Vaswani et al., 2017), the attention mechanism is built upon a query from a set  $Q$ , which pools values  $V$  through keys  $K$ . For each query, values are selected according to their matching score computed by the similarity between their corresponding keys and the query. Building on an analogy between the  $(K, V)$  couple and syntactic roles with their lexical realizations (explicit in §4.2) we present QKVAE², a Transformer-based Variational Autoencoder (VAE; Kingma and Welling, 2014).
To build our model, we modify a previous Transformer-based VAE, called the Attention-Driven VAE (ADVAE; Felhi et al., 2021). Using Cross-Attention, our model encodes sentences into two latent variables:  $z^{sem}$  to infer values for  $V$ , and  $z^{syn}$  to assign keys in  $K$  for values in  $V$ . These keys and values are then used in the Attention mechanism of a Transformer Decoder to generate sentences. We show that  $z^{syn}$  tends to contain syntactic information, while  $z^{sem}$  tends to represent semantic information. Additionally, comparisons with a supervised model show that it needs a considerable amount of data to outperform our model on syntactic and semantic transfer metrics.
Our contributions can be summarized as follows: We describe QKVAE, a model designed to disentangle syntactic information from semantic information by using separate latent variables for keys and values in Transformers Attention.
${}^{2}$  A contraction of the  $\left( {Q,K,V}\right)$  triplet with the VAE acronym.
$^{1}$  github.com/ghazi-f/QKVAE
Human Language Technologies, pages 5763 - 5776
- We run experiments on a dataset for English which empirically show that the two types of latent variables have strong preferences respectively for syntax and semantic.
- We also show that our model is capable of transferring syntactic and semantic information between sentences by using their respective latent variables. Moreover, we show that our model's syntax transfer capabilities are competitive with supervised models when they use their full training set (more than 400k sentences), and that a supervised model needs a fairly large amount of labeled data (more than 50k samples) to outperform it on both semantic and syntactic transfer.
We broadly divide works on explainability in NLP into two research directions. The first seeks post hoc explanations for black-box models, and led to a rich literature of observations on the behavior of Neural Models in NLP (Tenney et al., 2019; Jawahar et al., 2019; Hu et al., 2020; Kodner and Gupta, 2020; Marvin and Linzen, 2020; Kulmizev et al., 2020; Rogers et al., 2020). Along with these observations, this line of works also led to numerous advances in methodology concerning, for instance, the use of attention as an explanation (Jain and Wallace, 2019; Wiegreffe and Pinter, 2020), the validity of probing (Pimentel et al., 2020), or contrastive evaluation with minimal pairs (Vamvas and Sennrich, 2021). The second research direction on explainability in NLP seeks to build models that are explainable by design. This led to models with explicit linguistically informed mechanisms such as the induction of grammars (RNNG; Dyer et al., 2016, URNNG; Kim et al., 2019) or constituency trees (ON-LSTM; Shen et al., 2019, ONLSTM-SYD; Du et al., 2020).
Disentangled representation learning is a subfield of this second research direction which aims at separating neural representations into neurons with known associated meanings. This separation was performed on various characteristics in text such as style (John et al., 2020; Cheng et al., 2020), sentiment and topic (Xu et al., 2020), or word morphology (Behjati and Henderson, 2021). In works on disentanglement, consequent efforts have been put in the separation between syntax and semantics, whether merely to obtain an interpretable specialization in the embedding space (Chen et al., 2019;
Bao et al., 2019; Ravfogel et al., 2020; Huang et al., 2021), or for controllable generation (Zhang et al., 2019; Chen et al., 2020; Huang and Chang, 2021; Li et al., 2021). However, all these works rely on syntactic information (constituency parses and PoS tags) or semantic information (paraphrase pairs). To the best of our knowledge, our work is the first to present a method that directs syntactic and semantic information into assigned embeddings in the challenging unsupervised setup.
From a broader machine learning perspective, using knowledge of the underlying phenomena in our data, we design our model QKVAE with an inductive bias that induces understandable behavior in an unsupervised fashion. Among the existing line of applications of this principle (Rezende et al., 2016; Hudson and Manning, 2018; Locatello et al., 2020; Tjandra et al., 2021), ADVAE (Felhi et al., 2021), the model on which QKVAE is based, is designed to separate information from the realizations of different syntactic roles without supervision on a dataset of regularly structured sentences.
In this section, we go over the components of our model, namely VAEs, attention in Transformers, and ADVAE, the model on which QKVAE is based.
3.1 VAEs as Language Models
Given a set of observations  $w$ , VAEs are a class of deep learning models that train a generative model  $p_{\theta}(w) = \int_z p(z)p_{\theta}(w|z)dz$ , where  $p(z)$  is a prior distribution on latent variables  $z$  that serve as a seed for generation, and  $p_{\theta}(w|z)$  is called the decoder and generates an observation  $w$  from each latent variable value  $z$ . Since directly maximizing the likelihood  $p_{\theta}(w)$  to train a generative model is intractable, an approximate inference distribution  $q_{\phi}(z|w)$ , called the encoder, is used to formulate a lower-bound to the exact log-likelihood of the model, called the Evidence Lower-Bound (ELBo):
$$
\begin{array}{l} \log p _ {\theta} (w) \geq \\ \mathbb {E} _ {(z) \sim q _ {\phi} (z | w)} \left[ \log p _ {\theta} (w | z) \right] - \\ \operatorname {K L} \left[ q _ {\phi} (z | w) | | p (z) \right] = \operatorname {E L B o} (w; z) \tag {1} \\ \end{array}
$$
Early works on VAEs as language models have shown that, contrary to non-generative sequence-to-sequence (Sutskever et al., 2014) models, they learn a smooth latent space (Bowman et al., 2016). In fact, this smoothness enables decoding an interpolation of latent codes (i.e. a homotopy) coming
from two sentences to yield a well-formed third sentence that clearly shares characteristics (syntactic, semantic...) with both source sentences. This interpolation will be used as a control baseline in our experiments.
3.2 Attention in Transformers.
The inductive bias responsible for the disentangle-ment capabilities of our model is based on the design of Attention in Transformers (Vaswani et al., 2017). In attention mechanisms, each element of a series of query vectors  $Q = \{q_{1},\dots ,q_{|Q|}\}$  performs a soft selection of values  $V = \{v_{1},\ldots ,v_{|V|}\}$  whose compatibility with the query is given by their corresponding key vector in  $K = \{k_{1},\dots ,k_{|V|}\}$  via dot product. For each  $q_{i}\in Q$ , the series of dot products is normalized and used as weights for a convex interpolation of the values. Formally, the result is compactly written as:
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {S o f t m a x} (Q K ^ {T}) V \quad (2)
$$
Here, we stress that  $K$  is only capable of controlling what information is selected from  $V$ , while  $V$  is responsible for the value of this information. Using the above operators and the embedding level concatenation operator Cat, Multi-Head Attention (MHA) in Transformers is defined as follows:
$$
\mathrm {M H A} (\tilde {Q}, \tilde {K}, \tilde {V}) = \operatorname {C a t} (h e a d _ {1}, \dots h e a d _ {H}) W ^ {O}
$$
$$
s. t: h e a d _ {i} = \mathrm {A t t e n t i o n} (\tilde {Q} W _ {i} ^ {Q}, \tilde {K} W _ {i} ^ {K}, \tilde {V} W _ {i} ^ {V})
$$
Where  $W^{O}, W_{i}^{Q}, W_{i}^{K}$ , and  $W_{i}^{V}$  are trainable parameter matrices. In turn, Self-Attention (SA) and Cross-Attention (CA) are defined, for sets of elements called source  $S$  and target  $T$ , as follows:
$$
\mathrm {S A} (T) = \mathrm {M H A} (T, T, T)
$$
$$
\operatorname {C A} (T, S) = \operatorname {M H A} (T, S, S)
$$
The above SA mechanism is used to exchange information between elements of target  $T$ , while in CA, targets  $T$  pull (or query for) information from each element of the source  $S$ . Transformer Encoders (Enc) are defined as the composition of layers each consisting of an attention followed by a Feed-Forward Network F:
$$
\operatorname {E n c} (T) = \tilde {T} _ {D ^ {e n c}}, \text {s . t .} \tilde {T} _ {d} = \left\{ \begin{array}{l} T \text {i f} d = 0, \text {e l s e :} \\ \mathrm {F} (\mathrm {S A} (\tilde {T} _ {d - 1})) \end{array} \right.
$$
Transformer Decoders (Dec) are defined with instances of SA, CA and F:
$$
\operatorname {D e c} (T, S) = \tilde {T} _ {D ^ {d e c}}, \text {s . t .}:
$$
$$
\tilde {T} _ {d} = \left\{ \begin{array}{l} T \text {i f} d = 0, \text {e l s e :} \\ \mathrm {F} (\mathrm {C A} (\mathrm {S A} (\tilde {T} _ {d - 1}), S)) \end{array} \right.
$$
where  $D^{enc}$  and  $D^{dec}$  above are respectively the number of layers of Enc and Dec. For autoregressive decoding, Vaswani et al. (2017) define a version of Dec we will call  $\overline{\mathrm{Dec}}$ . In this version, the result of each  $QK^T$  (Eq. 2) in Self-Attention is masked so that each  $t_i$  in  $T$  only queries for information from  $t_j$  with  $j \leq i$ . Even though  $\overline{\mathrm{Dec}}$  yields a sequence of length equal to that of target  $T$ , in the following sections we will consider its output to be only the last element of  $\tilde{T}_{D^{dec}}$  in order to express auto-regressive generation in a clear manner.
3.3 ADVAE
ADVAE is a Variational Autoencoder for unsupervised disentanglement of sentence representations. It mainly differs from previous LSTM-based (Bowman et al., 2016) and Transformer-based (Li et al., 2020b) VAEs in that it uses Cross-Attention to encode and decode latent variables, which is the cornerstone of our model. In ADVAE, Cross-Attention is used to:  $i$  ) encode information from sentences into a fixed number of vectorial latent variables;  $ii$  ) decode these vectorial latent variables by using them as sources for the target sentences generated by a Transformer Decoder.
Formally, let us define  $M^{\mu}, M^{\sigma}$ , and  $M^{w}$  to be linear layers that will respectively be used to obtain the latent variables' means and standard deviations, and the generated words' probabilities,  $L$  the number of vectorial latent variables  $z = \{z_{1},\ldots ,z_{L}\}$ , and finally  $E = \{e_1,\dots ,e_L\}$  and  $D = \{d_{1},\ldots ,d_{L}\}$  two sets of  $L$  trainable embeddings. Embeddings  $e_i$  and  $d_{i}$  serve as fixed identifiers for the latent variable  $z_{i}$  respectively in the encoder and in the decoder.
Given input token sequence  $w$ , the encoder  $q_{\phi}(z|w) = \prod_{l}q_{\phi}(z_{l}|w)$  first yields parameters  $\mu_l$  and  $\sigma_l$  to be used by the diagonal Gaussian distribution of each of the latent variables  $z_{l}$  as follows:
3We omit residual connections and layer normalizations after each SA or CA for simplicity.
4To simplify equations, we omit word embedding look-up tables and positional embeddings.
v	child	to wear	cloak	winter
k1	nsubj	root	dobj	∅
k2	agent	root	nsubjpass	pobj
Table 1: Example of interpretable values for the  $v$  and  $k$  in our model with  $L = 4$ . We display a sentence transiting from the active form to the passive form, to illustrate how different keys arranging the same values can lead to the same minimal semantic units being rearranged according to a different syntactic structure. We also stress that a different set of keys may omit or bring forth an element from the values vector (e.g. "winter" here above).
$$
\tilde {z} = \operatorname {D e c} (e; \operatorname {E n c} (w))
$$
$$
\forall l \quad \text {s . t .} 1 \leq l \leq L:
$$
$$
\mu_ {l} = M ^ {\mu} (\tilde {z} _ {l}), \sigma_ {l} = \mathrm {S o f t P l u s} (M ^ {\sigma} (\tilde {z} _ {l}))
$$
$$
z _ {l} \sim \mathcal {N} \left(\mu_ {l}; \sigma_ {l}\right) \tag {3}
$$
Cross-Attention is also used by the ADVAE decoder to dispatch information from the source latent variable samples to the target generated sequence. Accordingly, using a beginning-of-sentence token  $w_0$ ,  $p_{\theta}(w|z) = \prod_i p_{\theta}(w_i|w_{<i}, z)$  yields probabilities for the categorical distribution of the generated tokens  $w$  by decoding latent variables  $z$  concatenated with their embeddings  $d$ :
$$
y = \operatorname {C a t} (d; z)
$$
$$
\forall i \quad \text {s . t .} 1 \leq i \leq | w |:
$$
$$
\tilde {w} _ {i} = \overline {{\operatorname {D e c}}} (w _ {0}, \dots , w _ {i - 1}; \operatorname {E n c} (y))
$$
$$
w _ {i} \sim \text {C a t e g o r i c a l} (\text {S o f t m a x} (M ^ {w} (\tilde {w} _ {i})))
$$
4 QKVAE: Using separate latent variables for Keys and Values
In this section, we describe the architecture of our model, the behavior it entails, and how we deal with the optimization challenges it poses.
4.1 QKVAE architecture
The modification we bring to ADVAE is aimed at controlling how information is selected from the latent space with the value of a newly introduced latent variable. We call this latent variable  $z^{syn}$ , and refer to the latent variables already formulated in ADVAE as  $z^{sem} = \{z_1^{sem}, \dots, z_L^{sem}\}$ .  $z^{syn}$  is obtained with the same process as each  $z_l^{sem}$  (Eq. 3), i.e. by adding an additional identifier embedding  $e_s$ , and matrices  $M^{\mu s}$  and  $M^{\sigma s}$  to obtain its mean and standard-deviation parameters.
For the QKVAE Decoder, we modify the Transformer Decoder Dec into QKVDec so as to use Multi-Head Attention with separate inputs for keys and values instead of Cross-Attention :
$$
\operatorname {Q K V D e c} \left(T; S _ {K}; S _ {V}\right) = \tilde {T} _ {D ^ {Q K V}}, \text {s . t .}:
$$
$$
\tilde {T} _ {d} = \left\{ \begin{array}{l} T \text {i f} d = 0, \text {e l s e :} \\ \mathrm {F} (\mathrm {M H A} (\mathrm {S A} (\tilde {T} _ {d - 1}), S _ {K}, S _ {V}) \end{array} \right.
$$
where  $D^{QKV}$  is the number of layers. Similar to  $\overline{\mathrm{Dec}}$ , we define  $\overline{\mathrm{QKVDec}}$  to be the auto-regressive version of QKVDec. The QKVAE decoder yields probabilities for the generated tokens by using this operator on values given by  $z^{sem}$  concatenated with embeddings  $d$ , and keys given by a linear transformation on  $z^{syn}$ :
$$
v = \operatorname {C a t} (d; z ^ {s e m}), k = M ^ {s} (z ^ {s y n})
$$
$$
\forall i \quad \text {s . t .} \quad 1 \leq i \leq | w |:
$$
$$
\tilde {w} _ {i} = \overline {{\mathrm {Q K V D e c}}} (w _ {0}, \dots , w _ {i - 1}; k; v)
$$
where  $M^s$  is a linear layer. While ADVAE already uses Cross-Attention to encode and decode latent variables, our model uses separate variables to obtain keys and values for Multi-Head Attention in its decoder.
4.2 QKVAE Behavior
In the Multi-Head Attention of our decoder,  $z^{syn}$  controls keys, and  $z^{sem}$  controls values. In other words, the value of each  $z_{l}^{sem}$  is called to be passed to the target sequence according to its key which is given by the variable  $z^{syn}$ . Therefore, given a query,  $z^{syn}$  decides which content vector  $z_{l}^{sem}$  participates most to the value of the generated token at each generation step. To better get a gist of the kind of behavior intended by this construction, we assume in Table 1 for explanatory purposes, that our decoder has one layer and one attention head, that the value of each  $k^{l}$  in key matrices  $k_{1}$  and  $k_{2}$  corresponds to syntactic roles, and that each  $v^{l}$  informs on the realization of the corresponding syntactic role. Table 1 displays the resulting sentence when each of  $k_{1}$  and  $k_{2}$  are coupled with  $v$ .
The output of  $M^s$  is reshaped to obtain a matrix of keys.
In the examples in Table 1, the generator uses a query at each generation step to pick a word in a manner that would comply with English syntax. Therefore, the key of each value should inform on its role in the target structure, which justifies syntactic roles as an adequate meaning for keys.
Although our model may stray from this possibility and formulate non-interpretable values and keys, keys will still inform on the roles of values in the target structure, and therefore influence the way values are injected into the target sequence. And given the fact that our model uses multiple layers and attention heads and the continuous nature of keys in Attention (as opposed to discrete syntactic role labels), our model performs a multi-step and continuous version of the behavior described in Table 1.
Injecting values into the structure of a sentence requires the decoder to model this structure. Previous works have shown that this is well within the capabilities of Transformers. Specifically, Hewitt and Manning (2019) showed that Transformers embed syntactic trees in their inner representations, Clark et al. (2019) showed that numerous attention heads attend to specific syntactic roles, and we (Felhi et al., 2021) showed that Transformer-based VAEs can capture the realizations of syntactic roles in latent variables obtained with Cross-Attention.
4.3 Balancing the Learning of  $z^{sem}$  and  $z^{syn}$
Similar to ADVAE, we use a standard Normal distribution as a prior  $p(z) = p(z^{sem})p(z^{syn})$  and train QKVAE with the  $\beta$ -VAE objective (Higgins et al., 2017) which is simply ELBo (Eq. 1) with a weight  $\beta$  on its Kullback-Leibler (KL) term. Higgins et al. (2017) show that a higher  $\beta$  leads to better unsupervised disentanglement. However, the KL term is responsible for a phenomenon called posterior collapse where the latent variables become uninformative and are not used by the decoder (Bowman et al., 2016). Therefore, higher values for  $\beta$  cause poorer reconstruction performance (Chen et al., 2018). To avoid posterior collapse, we follow Li et al. (2020a): i) We pretrain our model as an autoencoder by setting  $\beta$  to 0; ii) We linearly increase  $\beta$  to its final value (KL annealing; Bowman et al., 2016) and we threshold each dimension of the KL term with a factor  $\lambda$  (Free-Bits strategy; Kingma et al., 2016).
In preliminary experiments with our model, we observed that it tends to encode sentences using
only  $z^{sem}$ . As we use conditionally independent posteriors $^6$ $q(z^{syn}|w)$  and  $q(z^{sem}|w)$  for our latent variables, their KL terms (Eq. 1) can be written separately, and they can therefore be weighted separately with different values of  $\beta$ . Using a lower  $\beta$  for  $z^{syn}$  as was done by (Chen et al., 2020) $^7$  did not prove effective in making it informative for the model. Alternatively, linearly annealing  $\beta$  for  $z^{sem}$  before  $z^{syn}$  did solve the issue. This intervention on the learning process was inspired by the work of Li et al. (2020c) which shows that latent variables used at different parts of a generative model should be learned at different paces.
Data To compare our model to its supervised counterparts, we train it with data from the English machine-generated paraphrase pairs dataset ParaNMT (Wieting and Gimpel, 2018). More specifically, we use the 493K samples used by Chen et al.  $(2020)^{8}$  to train their model VGVAE. Since our model is unsupervised, we only use the reference sentences (half the training set) to train our model. Using the development and test sets of ParaNMT, Chen et al. (2020) also provide a curated set of triplets formed by a target sentence (target), a semantic source (sem_src), and a syntactic source (syn_src). The semantic source is a paraphrase of the target sentence, while the syntactic source is selected by finding a sentence that is syntactically close to the target (i.e. edit distance between the sequence of PoS Tags of both sentences is low $^{9}$ ) and semantically different from the paraphrase (has low BLEU score with it). Contrary to paraphrases in the training set of ParaNMT, paraphrases from this set were manually curated. These triplets are divided into a development set of 500 samples and a test set of 800 samples. We display results on the test set in the main body of the paper. The results on the development set, which lead to the same conclusions, are reported in Appendix A.
Training details & hyper-parameters Encoders and Decoders in QKVAE are initialized with pa
These posteriors are ADVAE encoders (Eq. 3).
7Although not explicitly mentioned in the paper, this is performed in their companion source code.
$^{8}$ https://drive.google.com/open?id=1HHDlUT_WpedL6zNYpcN94cLwed_yyrP
<sup>9</sup>We follow Chen et al. (2020) by using this evaluation data, although edit distance between PoS tags might not be a good proxy for syntactic similarity.
rameters from BART (Lewis et al., 2020). After manual trial and error on the development set, we set the sizes of  $z^{syn}$  and  $z^{sem}$  to 768, and  $L$  to 4. Further Hyper-parameters are in Appendix B. We train 5 instances of our model and report the average scores throughout all experiments.