Venue: NAACL 2022

Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

Ghazi Felhi, Joseph Le Roux

LIPN

Universite Sorbonne Paris Nord - CNRS UMR 7030

F-93430, Villetaneuse, France

{felhi, leroux}@lipn.fr Djame Seddah

INRIA Paris

Paris, France djame.seddah@inria.fr

###### Abstract

We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers.

In the attention of Transformers, _keys_ handle information selection while _values_ specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values.

We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syntax transfer capabilities when compared to supervised models and that comparable supervised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available1.

Footnote 1: github.com/ghazi-fQKVAE

## 1 Introduction

Disentanglement, a process aimed at obtaining neural representations with identified meaning, is a crucial component of research on interpretability Rudin et al. (2022). A form of disentanglement that received a lot of interest from the NLP community is the separation between syntax and semantics in neural representations Chen et al. (2019); Bao et al. (2019); Zhang et al. (2019); Chen et al. (2020); Huang and Chang (2021); Huang et al. (2021). Previous works perform disentanglement using paraphrase pairs as information for semantics, and/or constituency parses as information for syntax. The dependence of models on labeled data is known to entail high cost (see Seddah et al., 2020 on syntactic annotation), and to often require new labels to handle problems such as concept drift Lu et al. (2019) and domain adaptation Farahani et al. (2021).

In light of the above, we propose an unsupervised model which directs syntax and semantics into different neural representations without semantic or syntactic information. In the Transformer architecture Vaswani et al. (2017), the attention mechanism is built upon a _query_ from a set \(Q\), which pools _values_\(V\) through _keys_\(K\). For each query, values are selected according to their matching score computed by the similarity between their corresponding keys and the query. Building on an analogy between the \((K,V)\) couple and syntactic roles with their lexical realizations (explicited in SS4.2) we present QKVAE2, a Transformer-based Variational Autoencoder VAE; Kingma and Welling (2014).

Footnote 2: A contraction of the \((Q,K,V)\) triplet with the VAE acronym.

To build our model, we modify a previous Transformer-based VAE, called the Attention-Driven VAE (ADVAE; Felhi et al., 2021). Using Cross-Attention, our model encodes sentences into two latent variables: \(z^{sem}\) to infer values for \(V\), and \(z^{syn}\) to assign keys in \(K\) for values in \(V\). These keys and values are then used in the Attention mechanism of a Transformer Decoder to generate sentences. We show that \(z^{syn}\) tends to contain syntactic information, while \(z^{sem}\) tends to represent semantic information. Additionally, comparisons with a supervised model show that it needs a considerable amount of data to outperform our model on syntactic and semantic transfer metrics.

Our contributions can be summarized as follows:

* We describe QKVAE, a model designed to disentangle syntactic information from semantic information by using separate latent variables for keys and values in Transformers Attention.

 

* We run experiments on a dataset for English which empirically show that the two types of latent variables have strong preferences respectively for syntax and semantic.
* We also show that our model is capable of transferring syntactic and semantic information between sentences by using their respective latent variables. Moreover, we show that our model's syntax transfer capabilities are competitive with supervised models when they use their full training set (more than 400k sentences), and that a supervised model needs a fairly large amount of labeled data (more than 50k samples) to outperform it on both semantic and syntactic transfer.

## 2 Related Work

We broadly divide works on explainability in NLP into two research directions. The first seeks _post hoc_ explanations for black-box models, and led to a rich literature of observations on the behavior of Neural Models in NLP Tenney et al. (2019); Jawahar et al. (2019); Hu et al. (2020); Kodner and Gupta (2020); Marvin and Linzen (2020); Kulmizev et al. (2020); Rogers et al. (2020). Along with these observations, this line of works also led to numerous advances in methodology concerning, for instance, the use of attention as an explanation Jain and Wallace (2019); Wiegreffe and Pinter (2020), the validity of probing Pimentel et al. (2020), or contrastive evaluation with minimal pairs Vamvas and Sennrich (2021). The second research direction on explainability in NLP seeks to build models that are explainable by design. This led to models with explicit linguistically informed mechanisms such as the induction of grammars RNNG; Dyer et al. (2016), URNNG; Kim et al. (2019) or constituency trees (ON-LSTM; Shen et al. (2019), ONLSTM-SYD; Du et al. (2020).

Disentangled representation learning is a subfield of this second research direction which aims at separating neural representations into neurons with known associated meanings. This separation was performed on various characteristics in text such as style John et al. (2020); Cheng et al. (2020), sentiment and topic Xu et al. (2020), or word morphology Behjati and Henderson (2021). In works on disentanglement, consequent efforts have been put in the separation between syntax and semantics, whether merely to obtain an interpretable specialization in the embedding space Chen et al. (2019); Bao et al. (2019); Ravfogel et al. (2020); Huang et al. (2021), or for controllable generation Zhang et al. (2019); Chen et al. (2020); Huang and Chang (2021); Li et al. (2021). However, all these works rely on syntactic information (constitucency parses and PoS tags) or semantic information (paraphrase pairs). To the best of our knowledge, our work is the first to present a method that directs syntactic and semantic information into assigned embeddings in the challenging unsupervised setup.

From a broader machine learning perspective, using knowledge of the underlying phenomena in our data, we design our model QKVAE with an inductive bias that induces understandable behavior in an unsupervised fashion. Among the existing line of applications of this principle Rezende et al. (2016); Hudson and Manning (2018); Locatello et al. (2020); Tjandra et al. (2021), ADVAE Felhi et al. (2021), the model on which QKVAE is based, is designed to separate information from the realizations of different syntactic roles without supervision on a dataset of regularly structured sentences.

## 3 Background

In this section, we go over the components of our model, namely VAEs, attention in Transformers, and ADVAE, the model on which QKVAE is based.

### VAEs as Language Models

Given a set of observations \(w\), VAEs are a class of deep learning models that train a generative model \(p_{\theta}(w)=\int_{z}p(z)p_{\theta}(w|z)dz\), where \(p(z)\) is a prior distribution on latent variables \(z\) that serve as a seed for generation, and \(p_{\theta}(w|z)\) is called the decoder and generates an observation \(w\) from each latent variable value \(z\). Since directly maximizing the likelihood \(p_{\theta}(w)\) to train a generative model is intractable, an approximate inference distribution \(q_{\phi}(z|w)\), called the encoder, is used to formulate a lower-bound to the exact log-likelihood of the model, called the Evidence Lower-Bound (ELBo):

\[\log p_{\theta}(w)\geq\\ \mathbb{E}_{(z)\sim q_{\phi}(z|w)}\left[\log p_{\theta}(w|z) \right]-\\ \mathrm{KL}[q_{\phi}(z|w)||p(z)]=\mathrm{ELBO}(w;z) \tag{1}\]

Early works on VAEs as language models have shown that, contrary to non-generative sequence-to-sequence Sutskever et al. (2014) models, they learn a smooth latent space Bowman et al. (2016). In fact, this smoothness enables decoding an interpolation of latent codes (_i.e._ a homotopy) coming 

from two sentences to yield a well-formed third sentence that clearly shares characteristics (syntactic, semantic...) with both source sentences. This interpolation will be used as a control baseline in our experiments.

### Attention in Transformers.

The inductive bias responsible for the disentanglement capabilities of our model is based on the design of Attention in Transformers Vaswani et al. [2017]. In attention mechanisms, each element of a series of query vectors \(Q=\{q_{1},\ldots,q_{|Q|}\}\) performs a soft selection of values \(V=\{v_{1},\ldots,v_{|V|}\}\) whose compatibility with the query is given by their corresponding key vector in \(K=\{k_{1},\ldots,k_{|V|}\}\) via dot product. For each \(q_{i}\in Q\), the series of dot products is normalized and used as weights for a convex interpolation of the values. Formally, the result is compactly written as:

\[\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}(QK^{T})V \tag{2}\]

Here, we stress that \(K\) is only capable of controlling what information is selected from \(V\), while \(V\) is responsible for the value of this information. Using the above operators and the embedding level concatenation operator \(\mathrm{Cat}\), Multi-Head Attention (\(\mathrm{MHA}\)) in Transformers is defined as follows:

\[\mathrm{MHA}(\tilde{Q},\tilde{K},\tilde{V})=\mathrm{Cat}(head_{1},.. head_{H})W^{O}\] \[s.t:\;head_{i}=\mathrm{Attention}(\tilde{Q}W_{i}^{Q},\tilde{ K}W_{i}^{K},\tilde{V}W_{i}^{V})\]

Where \(W^{O}\), \(W_{i}^{Q}\), \(W_{i}^{K}\), and \(W_{i}^{V}\) are trainable parameter matrices. In turn, Self-attention (\(\mathrm{SA}\)) and Cross-Attention (\(\mathrm{CA}\)) are defined, for sets of elements called source \(S\) and target \(T\), as follows:

\[\mathrm{SA}(T)=\mathrm{MHA}(T,T,T)\] \[\mathrm{CA}(T,S)=\mathrm{MHA}(T,S,S)\]

The above \(\mathrm{SA}\) mechanism is used to exchange information between elements of target \(T\), while in \(\mathrm{CA}\), targets \(T\) pull (or _query_ for) information from each element of the source \(S\). Transformer Encoders (\(\mathrm{Enc}\)) are defined as the composition of layers each consisting of an attention followed by a Feed-Forward Network F:3

Footnote 3: We omit residual connections and layer normalizations after each \(\mathrm{SA}\) or \(\mathrm{CA}\) for simplicity.

Transformer Decoders (\(\mathrm{Dec}\)) are defined with instances of \(\mathrm{SA}\), \(\mathrm{CA}\) and \(\mathrm{F}\):

\[\mathrm{Dec}(T,S)=\tilde{T}_{D^{dec}},\text{s.t. }:\]

where \(D^{enc}\) and \(D^{dec}\) above are respectively the number of layers of \(\mathrm{Enc}\) and \(\mathrm{Dec}\). For autoregressive decoding, Vaswani et al. [2017] define a version of \(\mathrm{Dec}\) we will call \(\overline{\mathrm{Dec}}\). In this version, the result of each \(QK^{T}\) (Eq. 2) in Self-Attention is masked so that each \(t_{i}\) in \(T\) only queries formation from \(t_{j}\) with \(j\leq i\). Even though \(\overline{\mathrm{Dec}}\) yields a sequence of length equal to that of target \(T\), in the following sections we will consider its output to be only the last element of \(\tilde{T}_{D^{dec}}\) in order to express auto-regressive generation in a clear manner.

### Advae

ADVAE is a Variational Autoencoder for unsupervised disentanglement of sentence representations. It mainly differs from previous LSTM-based Bowman et al. [2016] and Transformer-based Li et al. [2020b] VAEs in that it uses Cross-Attention to encode and decode latent variables, which is the cornerstone of our model. In ADVAE, Cross-Attention is used to: _i)_ encode information from sentences into a fixed number of vectorial latent variables; _ii)_ decode these vectorial latent variables by using them as sources for the target sentences generated by a Transformer Decoder.

Formally, let us define \(M^{\mu}\), \(M^{\sigma}\), and \(M^{w}\) to be linear layers that will respectively be used to obtain the latent variables' means and standard deviations, and the generated words' probabilities, \(L\) the number of vectorial latent variables \(z=\{z_{1},\dots,z_{L}\}\), and finally \(E=\{e_{1},\dots,e_{L}\}\) and \(D=\{d_{1},\dots,d_{L}\}\) two sets of \(L\) trainable embeddings. Embeddings \(e_{i}\) and \(d_{i}\) serve as fixed identifiers for the latent variable \(z_{i}\) respectively in the encoder and in the decoder.

Given input token sequence \(w\), the encoder \(q_{\phi}(z|w)=\prod_{l}q_{\phi}(z_{l}|w)\) first yields parameters \(\mu_{l}\) and \(\sigma_{l}\) to be used by the diagonal Gaussian distribution of each of the latent variables \(z_{l}\) as follows4: 

\[\tilde{z}= \mathrm{Dec}(e;\mathrm{Enc}(w))\] \[\forall~{}l~{}~{}\text{ s.t. }1\leq l\leq L:\] \[\mu_{l}=M^{\mu}(\tilde{z}_{l}),~{}~{}\sigma_{l}=\mathrm{SoftPlus}( M^{\sigma}(\tilde{z}_{l}))\] \[z_{l}\sim\mathcal{N}(\mu_{l};\sigma_{l}) \tag{3}\]

Cross-Attention is also used by the ADVAE decoder to dispatch information from the _source_ latent variable samples to the _target_ generated sequence. Accordingly, using a beginning-of-sentence token \(w_{0}\), \(p_{\theta}(w|z)=\prod_{i}p_{\theta}(w_{i}|w_{<i},z)\) yields probabilities for the categorical distribution of the generated tokens \(w\) by decoding latent variables \(z\) concatenated with their embeddings \(d\):

\[y=\mathrm{Cat}(d;z)\] \[\forall~{}i~{}~{}\text{s.t. }1\leq i\leq|w|:\] \[\tilde{w}_{i}=\overline{\mathrm{Dec}}(w_{0},\dots,w_{i-1}; \mathrm{Enc}(y))\] \[w_{i}\sim\mathrm{Categorical}(\mathrm{Softmax}(M^{w}(\tilde{ w}_{i})))\]

## 4 QKVAE: Using separate latent variables for Keys and Values

In this section, we describe the architecture of our model, the behavior it entails, and how we deal with the optimization challenges it poses.

### QKVAE architecture

The modification we bring to ADVAE is aimed at controlling how information is selected from the latent space with the value of a newly introduced latent variable. We call this latent variable \(z^{syn}\), and refer to the latent variables already formulated in ADVAE as \(z^{sem}=\{z^{sem}_{1},\dots,z^{sem}_{L}\}\). \(z^{syn}\) is obtained with the same process as each \(z^{sem}_{l}\) (Eq. 3), _i.e._ by adding an additional identifier embedding \(e_{s}\), and matrices \(M^{\prime*}\) and \(M^{\prime*}\) to obtain its mean and standard-deviation parameters.

For the QKVAE Decoder, we modify the Transformer Decoder \(\mathrm{Dec}\) into \(\mathrm{QKVDec}\) so as to use Multi-Head Attention with separate inputs for keys and values instead of Cross-Attention :

\[\mathrm{QKVDC}(T;S_{K};S_{V})=\tilde{T}_{D^{QKV}},~{}\text{s. t. }:\] \[\tilde{T}_{d}=\left\{\begin{array}{ll}T&\text{if $d=0$, else:}\\ \mathrm{F}(\mathrm{MHA}(\mathrm{SA}(\tilde{T}_{d-1}),S_{K},S_{V})\end{array}\right.\]

where \(D^{QKV}\) is the number of layers. Similar to \(\overline{\mathrm{Dec}}\), we define \(\overline{\mathrm{QKVDec}}\) to be the auto-regressive version of \(\mathrm{QKVDec}\). The QKVAE decoder yields probabilities for the generated tokens by using this operator on values given by \(z^{sem}\) concatenated with embeddings \(d\), and keys given by a linear transformation on \(z^{syn}\):

\[v=\mathrm{Cat}(d; z^{sem}),~{}~{}k=M^{s}(z^{syn})\] \[\forall~{}i~{}~{}\text{s.t. }1\leq i\leq|w|:\] \[\tilde{w}_{i}= \overline{\mathrm{QKVDec}}(w_{0},\dots,w_{i-1};k;v)\] \[w_{i}\sim\mathrm{Categorical}(\mathrm{Softmax}(M^{w}(\tilde{w}_{ i})))\]

where \(M^{s}\) is a linear layer.5 While ADVAE already uses Cross-Attention to encode and decode latent variables, our model uses separate variables to obtain keys and values for Multi-Head Attention in its decoder.

Footnote 5: The output of \(M^{s}\) is reshaped to obtain a matrix of keys.

### QKVAE Behavior

In the Multi-Head Attention of our decoder, \(z^{syn}\) controls keys, and \(z^{sem}\) controls values. In other words, the value of each \(z^{sem}_{l}\) is called to be passed to the target sequence according to its key which is given by the variable \(z^{syn}\). Therefore, given a query, \(z^{syn}\) decides which content vector \(z^{sem}_{l}\) participates most to the value of the generated token at each generation step. To better get a gist of the kind of behavior _intended_ by this construction, we assume in Table 1 for explanatory purposes, that our decoder has one layer and one attention head, that the value of each \(k^{l}\) in key matrices \(k_{1}\) and \(k_{2}\) corresponds to syntactic roles, and that each \(v^{l}\) informs on the realization of the corresponding syntactic role. Table 1 displays the resulting sentence when each of \(k1\) and \(k2\) are coupled with \(v\).

\begin{table}
\begin{tabular}{c c c c c c c} \hline \(v\) & child & to wear & cloak & winter & & \\ \(k1\) & nsubj & root & dobj & \(\emptyset\) & \(\longrightarrow\) & decoded (\(v\), \(k1\)): & A child wears a cloak. \\ \(k2\) & agent & root & nsubjpass pobj & \(\longrightarrow\) & decoded (\(v\), \(k2\)): & A cloak is worn, in winter, by a child \\ \hline \end{tabular}
\end{table}
Table 1: Example of interpretable values for the \(v\) and \(k\) in our model with \(L=4\). We display a sentence transiting from the active form to the passive form, to illustrate how different _keys_ arranging the same _values_ can lead to the same minimal semantic units being rearranged according to a different syntactic structure. We also stress that a different set of _keys_ may omit or bring forth an element from the _values_ vector (_e.g._ “winter” here above).

 

In the examples in Table 1, the generator uses a query at each generation step to pick a word in a manner that would comply with English syntax. Therefore, the key of each value should inform on its role in the target structure, which justifies syntactic roles as an adequate meaning for keys.

Although our model may stray from this possibility and formulate non-interpretable values and keys, keys will still inform on the _roles_ of values in the target structure, and therefore influence the way values are injected into the target sequence. And given the fact that our model uses multiple layers and attention heads and the continuous nature of keys in Attention (as opposed to discrete syntactic role labels), our model performs a multi-step and continuous version of the behavior described in Table 1.

Injecting values into the structure of a sentence requires the decoder to model this structure. Previous works have shown that this is well within the capabilities of Transformers. Specifically, Hewitt and Manning (2019) showed that Transformers embed syntactic trees in their inner representations, Clark et al. (2019) showed that numerous attention heads attend to specific syntactic roles, and we (Felhi et al., 2021) showed that Transformer-based VAEs can capture the realizations of syntactic roles in latent variables obtained with Cross-Attention.

### Balancing the Learning of \(z^{sem}\) and \(z^{syn}\)

Similar to ADVAE, we use a standard Normal distribution as a prior \(p(z)=p(z^{sem})p(z^{syn})\) and train QKVAE with the \(\beta\)-VAE objective (Higgins et al., 2017) which is simply ELBo (Eq. 1) with a weight \(\beta\) on its Kullback-Leibler (KL) term. Higgins et al. (2017) show that a higher \(\beta\) leads to better unsupervised disentanglement. However, the KL term is responsible for a phenomenon called _posterior collapse_ where the latent variables become uninformative and are not used by the decoder (Bowman et al., 2016). Therefore, higher values for \(\beta\) cause poorer reconstruction performance (Chen et al., 2018). To avoid posterior collapse, we follow Li et al. (2020a): _i)_ We pretrain our model as an autoencoder by setting \(\beta\) to 0; _ii)_ We linearly increase \(\beta\) to its final value (KL annealing; Bowman et al., 2016) and we threshold each dimension of the KL term with a factor \(\lambda\) (Free-Bits strategy; Kingma et al., 2016).

In preliminary experiments with our model, we observed that it tends to encode sentences using only \(z^{sem}\). As we use conditionally independent posteriors6\(q(z^{syn}|w)\) and \(q(z^{sem}|w)\) for our latent variables, their KL terms (Eq. 1) can be written seperately, and they can therefore be weighted separately with different values of \(\beta\). Using a lower \(\beta\) for \(z^{sym}\) as was done by (Chen et al., 2020)7 did not prove effective in making it informative for the model. Alternatively, linearly annealing \(\beta\) for \(z^{sem}\) before \(z^{syn}\) did solve the issue. This intervention on the learning process was inspired by the work of Li et al. (2020c) which shows that latent variables used at different parts of a generative model should be learned at different paces.

Footnote 6: These posteriors are ADVAE encoders (Eq. 3).

Footnote 7: Although not explicitly mentioned in the paper, this is performed in their companion source code.

## 5 Experiments

### Setup

DataTo compare our model to its supervised counterparts, we train it with data from the English machine-generated paraphrase pairs dataset ParaNMT (Wieting and Gimpel, 2018). More specifically, we use the 493K samples used by Chen et al. (2020)8 to train their model VGVAE. Since our model is unsupervised, we only use the reference sentences (half the training set) to train our model. Using the development and test sets of ParaNMT, Chen et al. (2020) also provide a curated set of triplets formed by a target sentence (_target_), a semantic source (_sem_src_),and a syntactic source (_syn_src_). The semantic source is a paraphrase of the target sentence, while the syntactic source is selected by finding a sentence that is syntactically close to the target (_i.e._ edit distance between the sequence of PoS Tags of both sentences is low9) and semantically different from the paraphrase (has low BLEU score with it). Contrary to paraphrases in the training set of ParaNMT, paraphrases from this set were manually curated. These triplets are divided into a development set of 500 samples and a test set of 800 samples. We display results on the test set in the main body of the paper. The results on the development set, which lead to the same conclusions, are reported in Appendix A.

Footnote 8: [https://drive.google.com/open?id=IHHDIUT_WpedLfG2xNYpcn84cl.wed_yyrP](https://drive.google.com/open?id=IHHDIUT_WpedLfG2xNYpcn84cl.wed_yyrP)

Footnote 9: We follow Chen et al. (2020) by using this evaluation data, although edit distance between PoS tags might not be a good proxy for syntactic similarity.

Training details & hyper-parametersEncoders and Decoders in QKV 

rameters from BART Lewis et al. (2020). After manual trial and error on the development set, we set the sizes of \(z^{syn}\) and \(z^{sem}\) to 768, and \(L\) to 4. Further Hyper-parameters are in Appendix B. We train 5 instances of our model and report the average scores throughout all experiments.

BaselinesWe compare our system to 4 previously published models, where 2 are supervised and 2 are unsupervised: _i) VGVAE Chen et al. (2020):_ a VAE-based paraphrase generation model with an LSTM architecture. This model is trained using paraphrase pairs and PoS Tags to separate syntax and semantics into two latent variables. This separation is used to separately specify semantics and syntax to the decoder in order to produce paraphrases; _ii) SynPG Huang and Chang (2021):_ A paraphrase generation Seq2Seq model based on a Transformer architecture which also separately encodes syntax and semantics for the same purpose as VGVAE. This model is, however, trained using only source sentences with their syntactic parses, without paraphrases; _iii) Optimus Li et al. (2020):_ A large-scale VAE based on a fusion between BERT Devlin et al. (2019) and GPT-2 Radford et al. (2019) with competitive performance on various NLP benchmarks; _iv) ADVAE:_ This model is QKVAE without its syntactic variable. The size of its latent variable is set to 1536 to equal the total size of latent variables in QKVAE.

Official open-source instances10 of the 4 models above are available, which ensures accurate comparisons. The off-the-shelf instances of VGVAE and SynPG are trained on ParaNMT with GloVe11Pennington et al. (2014) embeddings. We fine-tune a pre-trained Optimus on our training set following instructions from the authors. Similar to our model, we initialize ADVAE with parameters from BARTLewis et al. (2020) and train 5 instances of it on ParaNMT with \(L=4\).

Footnote 10: VGVAE: github.com/mingdachen/syntactic-template-generation/; SynPG: github.com/uclanlp/syng; Optimus: github.com/ChunyuanLI/Optimus; ADVAE: github.com/ghazi-f/ADVAE

### Syntax and Semantics Separation in the Embedding Space

We first test whether \(z^{syn}\) and \(z^{sem}\) respectively specialize in syntax and semantics. A syntactic (resp. semantic) embedding should place syntactically (resp. semantically) similar sentences close to each other in the embedding space.

Using the (_target, sem_src, syn_src_) triplets, we calculate for each embedding the probability that _target_ is closer to _sem_src_ than it is to _syn_src_ in the embedding space. For simplicity, we refer to the syntactic and semantic embeddings of all models as \(z^{syn}\) and \(z^{sem}\). For Gaussian latent variables, we use the mean parameter as a representation (respectively the mean direction parameter from the von Mises-Fisher distribution of the semantic variable of VGVAE). We use an L2 distance for Gaussian variables and a cosine distance for the others. Since Optimus and ADVAE do not have separate embeddings for syntax and semantics _i)_ We take the whole embedding for Optimus; _ii)_For ADVAE, we measure the above probability on the development set for each latent variable \(z_{l}\) (Eq. 3). Then, we choose the latent variable that places _target_ sentences closest to their _sem_src_ (resp. _syn_src_) as a semantic (resp. syntactic) variable. The results are presented in Table 2.

Table 2 clearly shows for QKVAE, SynPG, and VGVAE that the syntactic (resp. semantic) variables lean towards positioning sentences in the embedding space according to their syntax (resp. semantics). Surprisingly, the syntactic variable of our model specializes in syntax (_i.e._ has low score) as much as that of SynPG. The generalist latent variable of Optimus seems to position sentences in the latent space according to their semantics. Accordingly, we place its score in the \(z^{sem}\) column. Interestingly, the variables in ADVAE have very close scores and score well below 50, which shows that the entire ADVAE embedding leans more towards syntax. This means that, without the key/value distinction in the Attention-based decoder, the variables specialize more in structure than in content.

\begin{table}
\begin{tabular}{c c c} \hline  & \multicolumn{2}{c}{\(z^{sem}\uparrow\)} & \(z^{syn}\downarrow\) \\ \cline{2-3} \multicolumn{2}{c}{_Supervised Models_} \\ \hline VGVAE & 99.9 & 14.8 \\ SynPG & 93.4 & 26.5 \\ \hline \multicolumn{2}{c}{_Unsupervised Models_} \\ \hline Optimus & 91.8 & - \\ ADVAE & 39.5 & 40.0 \\ QKVAE & 89.2 & 26.4 \\ \hline \end{tabular}
\end{table}
Table 2: The probability*100 that an embedding places a target sentence closer to its semantic source than it is to its syntactic source in the embedding space. Arrows (\(\uparrow\!\!/\downarrow\)) indicate whether higher or lower scores are better.

 



### Syntactic and Semantic Transfer

Similar to Chen et al. (2020), we aim to produce sentences that take semantic content from _sem_src_ sentences and syntax from _syn_src_ sentences. For each of SynPG, VGVAE, and QKVAE we simply use the syntactic embedding of _syn_src_, and the semantic embedding of _sem_src_ as inputs to the decoder to produce new sentences. Using the results of the specialization test in the previous experiment, we do the same for ADVAE by taking the 2 latent variables that lean most to semantics (resp. syntax) as semantic (resp. syntactic) variables. The output sentences are then scored in terms of syntactic and semantic similarity with _sem_src_, _syn_src_ and _target_.

Control and reference baselinesBeside model outputs, we also use our syntactic and semantic comparison metrics, explicited below, to compare _syn_src_ and _sem_src_ sentences to one another and to _target_ sentences. Additionally, using Optimus, we embed _sem_src_ and _syn_src_, take the dimension-wise average of both embeddings, and decode it. As VAEs are known to produce quality sentence interpolations Bowman et al. (2016); Li et al. (2020), the scores for this sentence help contrast a naive fusion of features in the embedding space with a composition of well identified disentangled features.

Transfer metricsWe measure the syntactic and semantic transfer from source sentences to output sentences. _i) Semantics:_ For semantics, previous works Chen et al. (2020); Huang and Chang (2021) rely on lexical overlap measures such as BLEU Papineni et al. (2001), ROUGE Lin (2004), and Meteor Denkowski and Lavie (2014). As will be shown in our results, the lexical overlap signal does not capture semantic transfer between sentences when this transfer is too weak to produce paraphrases. Therefore, we use Meteor (\(M\)) in conjunction with ParaBART Huang et al. (2021) a model where BART Lewis et al. (2020) is fine-tuned using syntactic information to produce neural representations that represent maximally semantics and minimally syntax. We measure the cosine similarity between sentences according to ParaBART embeddings (\(PB\)). _ii) Syntax:_ We use the script of Chen et al. (2020) to produce a syntactic tree edit distance (STED) between the constituency trees of sentences, as was done to assess VGVAE. Additionally, following the evaluation procedure designed by Huang and Chang (2021) for SynPG, we measure the Template Matching Accuracy between sentences, where the template is the constituency tree truncated at the second level (TMA2). TMA2 is the percentage of sentence pairs where such templates match exactly. We extend this measure by also providing it at the third level (TMA3). Results are presented in Tables 3 and 4. In both Tables, the comparison scores between sentences and _syn_src_ that are not significantly12 different from the same

\begin{table}
\begin{tabular}{c c|c||c|c||c|c} \hline  & \multicolumn{2}{c}{_sem\_src_} & \multicolumn{2}{c}{_syn\_src_} & \multicolumn{2}{c}{_target_} \\  & \(MT^{\dagger}\) & \(PBI\uparrow\) & \(M\downarrow\downarrow\) & \(PB\downarrow\) & \(M^{\dagger}\) & \(PB\uparrow\) \\ \hline \multicolumn{5}{c}{_Control and Reference baselines_} \\ \hline _sem\_src_ & 100 & 1.0 & 6.9 & 0.14 & 28.8 & 0.84 \\ _syn\_src_ & 6.9 & 0.14 & 100 & 1.0 & 12.1 & 0.16 \\ Optimus & 12.4 & 0.34 & 15.9 & 0.39 & 10.8 & 0.32 \\ \hline \multicolumn{5}{c}{_Supervised Models_} \\ \hline VGVAE & 17.6 & 0.58 & 15.3 & 0.18 & 24.9 & 0.58 \\ SynPG & 45.9 & 0.87 & 8.0 & 0.13 & 25.2 & 0.75 \\ \hline \multicolumn{5}{c}{_Unsupervised Models_} \\ \hline ADVAE & 8.0 & 0.19 & 8.3\({}^{\dagger}\) & 0.17 & 7.4 & 0.19 \\ QKVAE & 12.8 & 0.35 & 11.0 & 0.19 & 12.6 & 0.34 \\ \hline \end{tabular}
\end{table}
Table 4: Semantic transfer results. \(M\) is the Meteor score, and \(PB\) is the ParaBart cosine similarity.

\begin{table}
\begin{tabular}{l l l l||c c||c c c} \hline \hline  & \multicolumn{2}{c}{_sem\_src_} & \multicolumn{2}{c}{_syn\_src_} & \multicolumn{2}{c}{_target_} \\  & \(STD\dagger\) & \(TMA2\downarrow\) & \(TMA3\ddagger\) & \(STD\bot\) & \(TMA2\dagger\) & \(TMA3\dagger\) & \(STD\bot\) & \(TMA2\dagger\) & \(TMA3\dagger\) \\ \hline \multicolumn{5}{c}{_Control and Reference baselines_} \\ \hline _sem\_src_ & 0.0 & 100 & 100 & 13.0 & 40.3 & 4.8 & 12.0 & 39.6 & 7.0 \\ _syn\_src_ & 13.0 & 40.3 & 4.8 & 0.0 & 100 & 100 & 5.9 & 84.3 & 45.8 \\ Optimus & 11.6 & 50.0 & 15.9 & 9.2 & 61.6 & 23.6 & 10.2 & 58.9 & 21.8 \\ \hline \multicolumn{5}{c}{_Supervised Models_} \\ \hline VGVAE & 13.1 & 39.9 & 5.4 & 3.3 & 86.4 & 64.1 & 6.7 & 80.4 & 44.6 \\ SynPG & 11.7 & 41.9 & 18.0 & 13.5 & 74.1 & 10.5 & 13.1 & 69.1 & 13.3 \\ \hline \multicolumn{5}{c}{_Unsupervised Models_} \\ \hline ADVAE & 11.9 & 47.3 & 14.0 & 10.3 & 54.3\({}^{\dagger}\) & 19.2\({}^{\dagger}\) & 11.1 & 52.3 & 17.0 \\ QKVAE & 12.7 & 40.2 & 7.8 & 7.2 & 68.2 & 39.5 & 8.9 & 63.9 & 28.1 \\ \hline \end{tabular}
\end{table}
Table 3: Syntactic transfer results. _STED_ is the Syntactic Tree Edit Distance, and _TMA2/3_ is the exact matching between constituency trees truncated at the \(2^{nd}\)/\(3^{rd}\) level.



scores produced with regard to _sem_src_ are marked with 1.

Footnote 12: This was not observed by Chen et al. (2020), as they only compared outputs from VGVAE to the target paraphrases.

Sanity checks with metrics and baselinesWe notice in Table 4 that using Meteor as a semantic similarity measure results in various inconsistencies. For instance, paraphrases _target_ have a higher Meteor score with the syntactic sources than with interpolations from _Optimus_. It can also be seen that the Meteor score between outputs from VGVAE and both syntactic and semantic sources are rather close 13. In contrast, ParaBART score behaves as expected across comparisons in Table 4. Consequently, we retain ParaBART score as a semantic similarity measure. In the following, we use the scores between _sem_src_, _syn_src_, and _target_ (first two rows in Tables 4 and 3) as reference scores for unrelated sentences, paraphrase pairs, and syntactically similar sentences.

Footnote 13: This was not observed by Chen et al. (2020), as they only compared outputs from VGVAE to the target paraphrases.

Comparing the supervised baselinesVGVAE and SynPG greatly differ in scores. It can be seen that SynPG copies a lot of lexical items from its semantic input (high Meteor score) which allows for higher semantic similarity scores. However, Table 3 shows that SynPG transfers syntax from _syn_src_ at a high level (high TMA2, but low TMA3). In contrast, VGVAE transfers syntax and semantics in a balanced way and achieves the best syntax transfer scores overall (lowest STED with _syn_src_ and _target_).

Analysing the scores of QKVAEThe semantic similarity scores \(PB\) of QKVAE outputs with _target_ and _sem_src_ are close to those of Optimus outputs. Although these scores are low compared to supervised models, they are notably higher than semantic similarity scores between unrelated sentences (_e.g.__syn_src_ and _sem_src_). However, in contrast to Optimus, QKVAE outputs display low PB scores with _syn_src_, which show that they draw very little semantic information from the syntactic sources. Concerning syntactic transfer in Table 3, QKVAE outputs share syntactic information with _synt_src_ on all levels (low STED, and high TMA2 and TMA3). Our model is even competitive with SynPG on TMA2, and better on TMA3 and STED. As expected, the scores comparing QKVAE outputs to _sem_src_ show that they share very little syntactic information. On the other hand, ADVAE shows poor transfer performance on syntax and semantics, with only slight differences between scores w.r.t _syn_src_ and scores w.r.t _sem_src_.

### Comparing our Model to a Supervised Model with Less Data

Since VGVAE displays balanced syntactic and semantic transfer capabilities, we use it for this experiment where we train it on subsets of sizes in \(\{10K,25K,50K,100K\}\) from its original training data. Our goal is to find out how much labeled data is needed for VGVAE to outperform our unsupervised model on both transfer metrics.

In Figure 1, we plot for QKVAE and instances of VGVAE the _STED_ of their outputs w.r.t _syn_src_ and the \(PB\) of these outputs w.r.t _sem_src_. All values are averages over 5 runs, with standard deviations plotted as ellipses. Figure 1 shows that to outperform QKVAE on syntactic and semantic transfer, VGVAE needs more than 50K labeled samples.

## 6 Discussion and conclusion

In Table 5, we display example outputs of SynPG, VGVAE, and QKVAE along with their syntactic sources, semantic sources, and targets. We generally observed that the outputs of QKVAE range from paraphrases (line 6) to broadly related sentences (line 3). As was shown by our quantitative results, outputs from VAE-based models (VGVAE and QKVAE) share relatively few lexical items with the semantic input. This can be seen in the qualitative examples where they often swap words in the semantic source with closely related words (_e.g._

Figure 1: Plotting STED w.r.t _syn_ref_ and the PB cosine similarity w.r.t _sem_ref_ for VGVAE with different amounts of labeled data and for QKVAE. Points are scaled proportionally to the amount of training data. The vertical and horizontal diameters of each ellipse are equal to the standard deviation of the associated data points and axes.

 

"armed division" to "military force" in line 1, or "lunch boxes" to "snacks" in line 2). We attribute this quality to the smoothness of the latent space of VAEs which places coherent alternative lexical choices in the same vicinity. The examples above also show that our model is capable of capturing and transferring various syntactic characteristics such as the passive form (line 1), the presence of subject-verb inversion (lines 3, 4, and 7), or interjections (lines 4 and 6).

We presented QKVAE, an unsupervised model which disentangles syntax from semantics without syntactic or semantic information. Our experiments show that its latent variables effectively position sentences in the latent space according to these attributes. Additionally, we show that QKVAE displays clear signs of disentanglement in transfer experiments. Although the semantic transfer is moderate, syntactic transfer with QKVAE is competitive with SynPG, one of its supervised counterparts. Finally, we show that VGVAE, a supervised model, needs more than 50K samples to outperform QKVAE on both syntactic and semantic transfer.

We plan to extend this work in three directions: i) Finding ways to bias representations of each \(z_{l}^{sem}\) towards understandable concepts; _ii)_ Applying QKVAE to non-textual data since it is data agnostic (e.g. to rearrange elements of a visual landscape.); _iii)_ Investigating the behavior of QKVAE on other languages.

## Acknowledgments

This work is supported by the PARSITI project grant (ANR-16-CE33-0021) given by the French National Research Agency (ANR), the _Laboratoire d'excellence "Empirical Foundations of Linguistics"_ (ANR-10-LABX-0083), as well as the ONTORULE project. It was also granted access to the HPC resources of IDRIS under the allocation 20XX-AD011012112 made by GENCI.

## References

* 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_, pages 6008-6019.
* Behjati and Henderson [2021] Melika Behjati and James Henderson. 2021. Inducing Meaningful Units from Character Sequences with Slot Attention. _Arxiv_.
* 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings_, pages 10-21.
* 2019 Conference of the North American Chapter of the Association for

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline _sem\_src_ & _syn\_src_ & SynPG & VGVAE & QKVAE & _target_ \\ \hline \hline we have de-stored the 49th armored diffusion. & concomitant usage is not recommended. & we have destroyed 49th armored division. & amrored division has’t destroyed. & this military force will be destroyed. & 49th armored division has been destroyed . \\ \hline let the fire burn and put a piece of hot iron in it. & sing a song song song in it burn a hot piece of iron and fire. & don’t put the fire in turn a hot piece of iron and fire. & burn the fire. put the iron on burns. & come offe. get a fire on it. & keep his fire going. keep a piece of hot iron on it. \\ \hline they took the lunch boxes ’? & have you given me your hands ’? & do they bokes took the lunch ’? & have they taken them your snacks ? & have you heard of some lunch ’? & have you heard of the lunch boxes ’? \\ \hline does it have a coach ’? & that’s a phone switcher, right ’? & how does it have a coach? & that’s a coach, right ’? & that’s a coach, huh? & it has a coach, no ? \\ \hline an old lady in a cemetery. & that is a bad time for a war. & there’s a lady in an old cometary. & that’s an old lady in the cemetery. & this is a strange place for a woman. & there is an old lady in the cemetery. \\ \hline don’t be afraid. & there are still pzo. & you don’t be afraid. & there aren’t be afraid to be. & there will be no need to worry. & there is no need to be afraid . \\ \hline isn’t there a door open ’? & the machines are still good, right ’? & a isn’t open door there ’? & the doors aren’t open, right ’? & the door will be open, okay? & there is a door open, right ’? \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\ \end{tabular}
\end{table}
Table 5: Syntactic sources (_syn_src_), semantic sources (_sem_src_), the sentences produced when using them with different models, and the corresponding correct paraphrases (_target_).

 

Computational Linguistics: Human Language Technologies - Proceedings of the Conference_, 1:2453-2464.
* 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_, 1, pages 5972-5984.
* Workshop Track Proceedings_.
* Cheng et al. (2020) Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. 2020. Improving Disentangled Text Representation Learning with Information-Theoretic Guidance. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7530-7541.
* Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What Does BERT Look at? An Analysis of BERT's Attention. In _BlackBoxNLP@ACL_.
* Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
* 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Proceedings of the Conference_, 1:4171-4186.
* Du et al. (2020) Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J O'Donnell, Yoshua Bengio, and Yue Zhang. 2020. Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, page 6611-6628.
* Proceedings of the Conference_, pages 199-209.
* Farahani et al. (2021) Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. 2021. A Brief Review of Domain Adaptation. In _Advances in Data Science and Information Engineering_, pages 877-894.
* Felih et al. (2021) Ghazi Felih, Joseph Le Roux, and Djame Seddah. 2021. Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles. In _1st CrIGen: Controllable Generative Modeling in Language and Vision Workshop at NeurIPS 2021_.
* 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Proceedings of the Conference_, 1:4129-4138.
* Conference Track Proceedings_, pages 1-22.
* Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P Levy. 2020. A Systematic assessment of syntactic generalization in neural language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1725-1744.
* Huang et al. (2021) James Y. Huang, Kuan-Hao Huang, and Kai-Wei Chang. 2021. Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1372-1379, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Huang and Chang (2021) Kuan-hao Huang and Kai-wei Chang. 2021. Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1022-1033, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Hudson and Manning (2018) Drew A. Hudson and Christopher D. Manning. 2018. Compositional Attention Networks for Machine Reasoning. _International Conference on Learning Representations_, 333(6045):975-8.
* 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference_, pages 11-20.
* Jawahar et al. (2019) Ganesh Jawahar, Benoit Sagot, and Djame Seddah. 2019. What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3651-3657, Florence, Italy. Association for Computational Linguistics 

Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Veethomova. 2020. Disentangled representation learning for non-parallel text style transfer. In _ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_, pages 424-434.
* 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Proceedings of the Conference_, 1:1105-1117.
* Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kingma et al. (2016) Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. _Advances in neural information processing systems_, 29:4743-4751.
* Conference Track Proceedings_, MI, pages 1-14.
* Kodner and Gupta (2020) Jordan Kodner and Nitish Gupta. 2020. Overestimation of syntactic representation in neural language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, page 1757-1762.
* Kulmizev et al. (2020) Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, page 4077-4091.
* Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pretraining for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Stroudsburg, PA, USA. Association for Computational Linguistics.
* 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference_, pages 3603-3614.
* Li et al. (2020b) Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xuijun Li, Yizhe Zhang, and Jianfeng Gao. 2020b. Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4678-4699, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Li et al. (2021) Dingcheng Li, Hongliang Fei, Shaogang Ren, and Ping Li. 2021. A deep decomposable model for disentangling syntax and semantics in sentence representation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 4300-4310, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Li et al. (2020c) Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. 2020c. Progressive Learning and Disentanglement of Hierarchical Representations. _International Conference on Learning Representations_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In _Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL 2004, Barcelona, Spain_, pages 74-81.
* Locatello et al. (2020) Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot attention. _Advances in Neural Information Processing Systems_, 2020-Decem(NeurlIPS):1-27.
* Lu et al. (2019) Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. 2019. Learning under Concept Drift: A Review. _IEEE Transactions on Knowledge and Data Engineering_, 31(12):2346-2363.
* Marvin and Linzen (2020) Rebecca Marvin and Tal Linzen. 2020. Targeted syntactic evaluation of language models. _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018_, pages 1192-1202.
* Papineni et al. (2001) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, volume 371, pages 311-318, Morristown, NJ, USA. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Pimentel et al. (2020) Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-Theoretic Probing for Linguistic Structure. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4609-4622, Stroudsburg, PA, USA. Association for Computational Linguistics 

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
* Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Jacob Goldberger, and Yoav Goldberg. 2020. Unsupervised distillation of syntactic information from contextualized word representations. In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 91-106, Online. Association for Computational Linguistics.
* Rezende et al. (2016) Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. 2016. Unsupervised learning of 3D structure from images. In _Advances in Neural Information Processing Systems_, Nips, pages 5003-5011.
* Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. _Transactions of the Association for Computational Linguistics_, 8:842-866.
* Rudin et al. (2022) Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. 2022. Interpretable machine learning: Fundamental principles and 10 grand challenges. _Statistics Surveys_, 16:1-80.
* Seddah et al. (2020) Djame Seddah, Farah Essaidi, Annal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suarez, Benoit Sagot, and Abhishek Srivastava. 2020. Building a User-Generated Content North-African Arabizi Treebank: Tackling Hell. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1139-1150. Association for Computational Linguistics.
* Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. _35th International Conference on Machine Learning, ICML 2018_, 10:7322-7330.
* Shen et al. (2019) Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In _7th International Conference on Learning Representations, ICLR 2019_, pages 1-14.
* Volume 2_, 155(1-2):105-145.
* Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
* Tjandra et al. (2021) Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. 2021. Unsupervised learning of disentangled speech content and style representation. _Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH_, 4:3191-3195.
* Vamvas and Sennrich (2021) Jannis Vamvas and Rico Sennrich. 2021. On the Limits of Minimal Pairs in Contrastive Evaluation. In _Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 58-68, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In _NeurIPS_, Nips.
* 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference_, pages 11-20.
* 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_, 1:451-462.
* Xu et al. (2020) Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. 2020. On Variational Learning of Controllable Representations for Text without Supervision. _The 37th International Conference on Machine Learning (ICML 2020)_.
* Zhang et al. (2019) Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, and Lawrence Carin. 2019. Syntax-Infused Variational Autoencoder for Text Generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2069-2078, Stroudsburg, PA, USA. Association for Computational Linguistics.

 

Results on the development set

We hereby display the scores on the development set. The encoder scores concerning the specialization of latent variables are in Table 6, while the transfer scores are in Table 7 for semantics, and Table 8 for syntax. The values on the development set concerning the comparison of QKVAE with VGVAE trained on various amounts of data is in Figure 2.

## Appendix B Hyper-parameters

Hyper-parameter valuesThe \(\beta\) weight on the KL divergence is set to 0.6 for \(z^{c}\) and to 0.3 for \(z^{s}\), and the \(\lambda\) threshold for the Free-Bits strategy is set to 0.05. KL annealing is performed between steps 3K and 6K for \(z^{sem}\), and between steps 7K and 20K for \(z^{syn}\). The model is trained using Adafactor Shazeer and Stern (2018), a memory-efficient version of Adam Kingma and Ba (2015). Using a batch size of 64, we train for 40 epochs, which takes about 30 hours on a single Nvidia GEForce RTX 2080 GPU. We use 4 layers for both Transformer encoders and decoders. The encoders (resp. decoders) are initialized with parameters from the 4 first layers (resp. 4 last layers) of BART encoders (resp. decoders). In total, our model uses 236M parameters.

Manual Hyper-parameter searchGiven that the architecture for Transformer layers is fixed by BART, we mainly explored 3 parameters: number of latent variables \(L\), number of Transformer layers, values for \(\beta\). Our first experiments have shown that setting \(L\) to 8 or 16 does not yield good re

\begin{table}
\begin{tabular}{l c c} \hline  & \multicolumn{2}{c}{\(z^{sem}\uparrow\)} & \(z^{syn}\downarrow\) \\ \cline{2-3} \multicolumn{3}{c}{_Supervised Models_} \\ \hline VGVAE & 99.0 & 16.4 \\ SynPG & 91.6 & 31.2 \\ \hline \multicolumn{3}{c}{_Unsupervised Models_} \\ \hline Optimus & 89.4 & - \\ ADVAE & 41.0 & 40.3 \\ QKVAE & 86.7 & 27.0 \\ \end{tabular}
\end{table}
Table 6: The probability*100 that an embedding places a target sentence closer to its semantic source than it is to its syntactic source in the embedding space. (development set results)

Figure 2: Plotting STED w.r.t _syn_ref_ and the PB cosine similarity w.r.t _semi_ref_ for VGVAE with different amounts of labeled data and for QKVAE. Points are scaled proportionally to the amount of training data. The vertical and horizontal diameters of each ellipse are equal to the standard deviation of the associated data points and axes.

\begin{table}
\begin{tabular}{l l|l||l|l||l|l|l} \hline  & \multicolumn{3}{c}{_sem_src_} & \multicolumn{3}{c}{_synt_src_} & \multicolumn{3}{c}{_target_} \\  & & _STED_\(\dagger\) & _TMA2\(\downarrow\)_ & _TMA3\(\downarrow\)_ & _STED_\(\downarrow\) & _TMA3\(\uparrow\)_ & _STED_\(\downarrow\) & _TMA2\(\uparrow\)_ & _TMA3\(\uparrow\)_ \\ \hline \multicolumn{6}{c}{_Control/Celling baselines_} \\ \hline _sem\_src_ & 0.0 & 100 & 100 & 11.9 & 46.4 & 6.8 & 10.9 & 47.0 & 7.3 \\ _syn\_src_ & 11.9 & 46.4 & 6.8 & 0.0 & 100 & 100 & 6.0 & 81.6 & 45.0 \\ Optimus & 9.7 & 58.2 & 20.6 & 9.2\({}^{\dagger}\) & 61.6\({}^{\dagger}\) & 22.6\({}^{\dagger}\) & 9.9 & 59.6 & 18.4 \\ \hline \multicolumn{6}{c}{_Supervised Models_} \\ \hline VGVAE & 11.9 & 45.4 & 6.8 & 3.2 & 84.2 & 58.2 & 6.7 & 77.6 & 39.0 \\ SynPG & 9.3 & 49.4 & 21.4 & 12.2 & 73.0 & 12.2 & 12.2 & 68.6 & 13.0 \\ \hline \multicolumn{6}{c}{_Unsupervised Models_} \\ \hline ADVAE & 10.1 & 53.4 & 18.6 & 9.8\({}^{\dagger}\) & 55.0\({}^{\dagger}\) & 17.4\({}^{\dagger}\) & 10.5 & 52.8 & 15.4 \\ QKVAE & 11.4 & 45.0 & 9.1 & 6.8 & 66.4 & 37.4 & 8.6 & 63.0 & 26.9 \\ \hline \end{tabular}
\end{table}
Table 8: Syntactic transfer results (development set results) 

sults, which is probably due to the fact that a high \(L\) raises the search space for possible arrangements of values with keys, and consequently makes convergence harder. Concerning the number of layers, we observed that results with the full BART model (6 layers) have high variance over different runs. Reducing the number of layers to 4 solved this issue. In regards to \(\beta\), we observed that it must be \(0.6\) or less for the model to produce adequate reconstructions and that it is beneficial to set it slightly lower for \(z^{syn}\) than for \(z^{sem}\) so as to absorb more syntactic information with \(z^{syn}\).