Year: 2021

Abstract:
Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,. . . ) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that i) disentanglement of syntactic roles can be induced without supervision, ii) ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAEs, iii) realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available1.

heading: 1 INTRODUCTION
text: A disentangled representation of data describes information as a combination of separate understandable factors. This separation provides better transparency, but also better transfer performance (Higgins et al., 2018; Dittadi et al., 2021). When it comes to disentanglement, Variational Autoencoders (VAEs; Kingma & Welling, 2014) were extensively proven effective (Higgins et al., 2017; Chen et al., 2018; Rolinek et al., 2019). and were used throughout several recent works (Chen et al., 2019; Li et al., 2020b; John et al., 2020). In NLP, disentanglement has been mostly performed to separate the semantics (or content) in a sentence from characteristics such as style and structure in order to generate paraphrases (Chen et al., 2019; John et al., 2020; Bao et al., 2020; Huang & Chang, 2021; Huang et al., 2021). We show in our work that the information in the content itself can be separated with a VAE-based model. In contrast to the aforementioned works, we use neither supervision nor input syntactic information for this separation. We demonstrate this ability by controlling the lexical realization of core syntactic roles. For example, the subject in a sentence can be encoded separately and controlled to generate the same sentence with another subject. Our framework includes a model and an evaluation protocol aimed at measuring the disentanglement of syntactic roles.
The model we introduce is an Attention-Driven VAE (ADVAE), which we train on the SNLI raw text dataset (Schmidt et al., 2020). It draws its inspiration from attention-based machine translation models (Bahdanau et al., 2015; Luong et al., 2015). Such models translate sentences between languages with
1URL to be disclosed upon publication. Our code is provided as supplemental material during the submission process.
different underlying structures and can be inspected to show a coherent alignment between spans from both languages. Our ADVAE uses Transformers (Vaswani et al., 2017), an attention-based architecture, to map sentences from a language to independent latent variables, then map these variables back to the same sentences. Although ADVAE could be used to study other attributes, we motivate it (§4.1) and therefore study it for the alignment of syntactic roles with latent variables.
Evaluating disentanglement with regard to spans is challenging. After training the model and only for evaluation, we use linguistic information (from an off-the-shelf dependency parser) to first extract syntactic roles from sentences, and then study their relation to latent variables. To study this relation on the ADVAE decoder, we repeatedly i) generate a sentence from a sampled latent vector ii) perturb this latent vector at a specific location iii) generate a sentence from this new vector and observe the difference. On the encoder side, we study the attention values to see whether each latent variable is focused on a particular syntactic role in input sentences. The latter procedure is only possible through the way our ADVAE uses attention to produce latent variables. To the best of our knowledge, we are the first to use this transparency mechanism to obtain quantitative results for a latent variable model.
We first justify our focus on syntactic roles in §3, then we go over our contribution, which is threefold: i) We introduce the ADVAE, a model that is designed for unsupervised disentanglement of syntactic roles, and that enables analyzing the interaction between latent variables and observations through the values of attention (§4), ii) We design an experimental protocol for the challenging assessment of disentanglement over realizations of syntactic roles, based on perturbations on the decoder side and attention on the encoder side (§5), iii) Our empirical results show that our architecture disentangles syntactic roles better than standard sequence VAEs and Transformer VAEs and that it is capable of controlling realizations of syntactic roles separately during generation (§6).

heading: 2 RELATED WORKS
text: Linguistic information in neural models Accounting for linguistic information provided better inductive bias in the design of neural NLP systems during recent years. For instance, successful attempts at capturing linguistic information with neural models helped improve grammar induction (RNNG; Dyer et al., 2016), constituency parsing and language modeling (ON-LSTM; Shen et al., 2019, ONLSTM-SYD; Du et al., 2020), as well as controlled generation (SIVAE; Zhang et al., 2019). Many ensuing works have also dived into the linguistic capabilities of the resulting models, the types of linguistic annotations that emerge best in them, and syntactic error analyses (Hu et al., 2020; Kodner & Gupta, 2020; Marvin & Linzen, 2020; Kulmizev et al., 2020). Based on the Transformer architecture, the self-supervised model BERT (Devlin et al., 2019) has also been subject to studies showing that the linguistic information it captures is organized among its layers in a way remarkably close to the way a classical NLP pipeline works (Tenney et al., 2020). Furthermore, (Clark et al., 2019), showed that many attention heads in BERT specialize in dependency parsing. We refer the reader to (Rogers et al., 2020) for an extensive review of Bert-related studies. However, such studies most often rely on structural probes (Jawahar et al., 2019; Liu et al., 2019; Hewitt & Manning, 2019) to explain representations, probes which are not without issues, as shown by Pimentel et al. (2020). In that regard, the generative capabilities and the attention mechanism of our model offer an alternative to probing: analysis is performed directly on sentences generated by the model and on internal attention values.
Disentanglement in NLP The main line of work in this area revolves around using multitask learning to separate concepts in neural representations (e.g. style vs content (John et al., 2020), syntax vs semantics (Chen et al., 2019; Bao et al., 2020)). Alternatively, Huang & Chang (2021) and Huang et al. (2021) use syntactic trees as inputs to separate syntax from semantics, and generate paraphrases without a paraphrase corpus. Towards less supervision, Cheng et al. (2020) only uses style information to separate style from content in representations. Literature on unsupervised disentanglement in NLP remains sparse. Examples are the work of Xu et al. (2020) on categorical labels (sentiment and topic), and that of Behjati & Henderson (2021) on representing morphemes using character-level Seq2Seq models. The work of Behjati & Henderson (2021) is closest to ours as it uses Slot Attention (Locatello et al., 2020), which, like ADVAE, is a cross-attention-based representation technique. Our contribution depart from previous work since i) syntactic parses are not used as learning signals but as a way to interpret our model, and ii) cross-attention enables our model to link a fixed number of latent variables to text spans.

heading: 3 SYNTACTIC ROLES AND DEPENDENCY PARSING
text: We present in Figure 1 an example sentence with its dependency parse2, its Part-of-Speech (PoS) tags, and a flat predicative structure with PropBank-like semantic roles (Palmer et al., 2005). Dependency parsing yields a tree, where edges are labeled with syntactic roles (or relations or functions) such as nominal subject (nsubj). The lexical realizations of these syntactic functions are textual spans and correspond to syntactic constituents. For instance, the lexical realization of the direct object (dobj) of the verb holds in this sentence is the span his nice guitar, with guitar as head. In short, the spans corresponding to subtrees consist of tokens that are more dependent of each other than of the rest of the sentence. As a consequence, and because a disentanglement model seeks independent substructures in the data, we expect such a model to converge to representations that display separation in realizations of frequent syntactic roles.
In our work, we focus3 within the same framework. on nominal subjects, verbal roots of sentences, and direct or prepositional objects. These are core (as opposed to oblique; see Nivre et al. (2016) for details on the distinction) syntactic roles, since they directly relate to the predicative structure. In fact in most cases, as illustrated in Figure 1, the verbal root of a sentence is its main predicate, the nominal subject its agent (ARG0) and the direct or prepositional object its patient (ARG1).

heading: 4 MODEL DESCRIPTION
text: The usual method to obtain sentence representations from Transformer models uses only a Transformer encoder either by taking an average of the token representations or by using the representation of a special token (e.g [CLS] in BERT(Devlin et al., 2019)). Recently, the usage of both Transformer encoders and decoders has also been explored in order to obtain representations whether by designing classical Autoencoders (Lewis et al., 2019; Siddhant et al., 2019; Raffel et al., 2020), or VAEs (Li et al., 2020a). Our model, the ADVAE, differs from these models in that it uses targets in Machine Translation (MT) Transformers (i.e an encoder and a decoder) to produce sentence representations. Producing representations with Cross-Attention has been introduced by Locatello et al. (2020) as part of the Slot Attention modules in the context of unsupervised object discovery. However, in contrast to Locatello et al. (2020), we simply use Cross-Attention as it is found in Vaswani et al. (2017), i.e. without normalizing attention weights over the query axis, or using GRUs(Cho et al., 2014) to update representations. As will be shown through our experiments, this is sufficient to disentangle syntactic roles. We explain the observation that motivates our work in §4.1, we then describe in §4.2 the minimal changes we apply to MT Transformers, and finally, we present the objective we use in §4.3. The parallel between our model and MT Transformers is illustrated in Figure 2.

heading: 4.1 THE INTUITION BEHIND OUR MODEL
text: Consider s = (sj)1≤j≤Ns and t = (tj)1≤j≤Nt , two series of tokens forming respectively a sentence in a source language and a sentence in a target language. Given s, attention-based translation models
2Following the ClearNLP constituent to dependency conversion, close to Stanford Dependencies de Marneffe & Manning (2008). See https://github.com/clir/clearnlp-guidelines/blob/master/ md/components/dependency_conversion.md.
3Future research that takes interest in the finer-grained disentanglement of content may simply study a larger array of syntactic roles. Using our current system we display results including all syntactic roles in Appendix G.
are capable of yielding t while also providing information about the alignment between the groups of tokens (of different sizes) in both sentences (Bahdanau et al., 2015; Luong et al., 2015)). This evidence suggests that attention-based architectures are capable of factoring information from groups of words according to a source structure, and redistributing it according to a target structure.
The aim of our design is to use, as a target, a set of NZ independent latent variables that will act as fixed placeholders for the information in sentences. We stress that NZ is fixed and independent of the input sentence size Ns. Combining Transformers, an attention-based MT model, and the VAE framework for disentanglement, our ADVAE is intended to factor information from independent groups of words into separate latent variables. In the following sections, we will refer to this set of independent latent variables as the latent vector z = (zi)1≤i≤NZ and to each zi as a latent variable.

heading: 4.2 MODEL ARCHITECTURE
text: Inference model: This is the inference model qφ (encoder in Fig. 2.b) for our latent variables z = (zi)1≤i≤NZ . It differs from an MT Transformer in two ways. First it uses as input a sentence s, and NZ learnable vectors (eenczi )1≤i≤NZ instead of the target tokens t used in translation. These learnable vectors will go through Cross-Attention without Self-Attention. We stress that these learnable vectors are input-independent. Second its output is not used to select a token from a vocabulary but rather passed to a linear layer (resp. a linear layer followed by a softplus non-linearity) to yield the mean parameters (µi)1≤i≤NZ (resp. the standard deviation parameters (σi)1≤i≤NZ ) to parameterize the diagonal Gaussian distributions (q(i)φ (zi|s))1≤i≤NZ . The Transformer Decoder is therefore replaced in Fig 2.b by a Transformer Encoder that uses Cross-attention to factor information from the sentence. The distribution of the whole latent vector is simply the product of Gaussians qφ(z1, . . . , zNZ |s) = ∏NZ i q (i) φ (zi|s).
Generation model: Our generation model consists of an autoregressive decoder (Fig. 2.c) pθ(s|z1, . . . , zNZ ) = ∏Ns j pθ(sj |s<j , z1, . . . , zNZ ) where s<i is the series of tokens preceding
si, and a prior assuming independent standard Gaussian variables, i.e. p(z1, . . . , zNZ ) = ∏NZ i p(zi).
Each latent variable zi is concatenated with an associated learnable vector edeczi (z-Identifier in Fig. 2.c) instead of going through positional encoding. From there on, the latent variables are used like source tokens in an MT Transformer.

heading: 4.3 OPTIMIZATION OBJECTIVE
text: We train our ADVAE using the β-VAE (Higgins et al., 2017) objective, which is the Evidence Lower-Bound (ELBo) with a controllable weight on its Kullback-Leibler (KL) term:
log pθ(s) ≥ E(z)∼qφ(z|s) [log pθ(s|z)]− βDKL[qφ(z|s)||p(z)] (1) In Eq. 1, s is a sample from our dataset, z is our latent vector and the distributions pθ(s) =∫ pθ(s|z)p(z)dz and qφ(z|s) are respectively the generation model and the inference model. We use a standard Gaussian distribution as prior p(z) and a diagonal Gaussian distribution as the approximate inference distribution qφ(z|s). The weight β is used (as in Chen et al., 2018, Xu et al., 2020, Li et al., 2020b) to control disentanglement, but also to find a balance between the expressiveness of latent variables and the generation quality.

heading: 5 EVALUATION PROTOCOL
text: In order to quantify disentanglement, we first measure the interaction between latent variables and syntactic roles. To do so, we extract core syntactic roles from sentences according to the procedure we describe in §5.1. Subsequently, for the ADVAE decoder, we repeatedly perturb latent variables and measure their influence on the realizations of the syntactic roles in generated sentences (§5.2). For the ADVAE encoder, we use attention to determine the syntactic role that participates most in producing the value of each latent variable (§5.3).
Given these metrics, we measure disentanglement taking inspiration from the Mutual Information Gap (MIG; Chen et al., 2018) in §5.4. MIG consists in measuring the difference between the first and second latent variables with the highest mutual information with regard to a target factor. It is intended to quantify the extent to which a target factor is concentrated in a single variable. This metric assumes knowledge of the underlying distribution of the target information in the dataset.However, there is no straightforward or agreed-upon way to set this distribution for text spans, and therefore to calculate MIG in our case. As a workaround, we use the influence metrics defined in §5.2 and §5.3 as a replacement for mutual information to quantify disentanglement.

heading: 5.1 SYNTACTIC ROLE EXTRACTION
text: We use the Spacy4 dependency parser (Honnibal & Montani, 2017) trained on Ontonotes5 (Weischedel et al., 2013). For each sentence the realization of verb is the root of the dependency tree if its POS tag is VERB. Realizations of subj (subject), dobj (direct object), and pobj (prepositional object) are spans of subtrees whose roots are labelled resp. nsubj, dobj, and pobj.
In the rare cases where multiple spans answer the requirement for a syntactic role, we take the first one as the subsequent spans are most often part of a subordinate clause. A realization of a syntactic role in R = {verb, subj, dobj, pobj} is empty if no node in the dependency tree satisfies its extraction condition.5

heading: 5.2 LATENT VARIABLE INFLUENCE ON DECODER
text: Intuitively, we repeatedly compare the text generated from a sampled latent vector to the text generated using the same vector where only one latent variable is resampled. Thus we can isolate the effect of each latent variable on output text and gather statistics.
More precisely, we sample T dec latent vectors (z(l))1≤l≤Tdec = (z (l) i )1≤l≤Tdec,1≤i≤NZ . Then for each zl, and for each i we create an altered version z̃(li) = (z̃(li)i′ )1≤i′≤NZ where we resample only the ith latent variable (i.e. ∀i′ 6= i, z̃(li)i′ = z (l) i′ ).
Generating the corresponding sentences6 with pθ(s|z) yields a list of original sentences (s(l))1≤l≤Tdec , and a matrix of sentences displaying the effect of modifying each latent variable
4https://spacy.io/models/en#en_core_web_sm 5Examples of syntactic role extractions can be found in Appendix D. 6Throughout this work, we use greedy sampling (sampling the maximum-probability word at each step), for
all generated sentences.
(s̃(li))1≤l≤Tdec,1≤i≤NZ . For each syntactic role r ∈ R, we will denote the realization extracted from a sentence s with ρr(s).
To measure the influence of a variable zi on the realization of a syntactic role r, denoted Γdecri , we estimate the probability that a change in this latent variable incurs a change in the span corresponding to the syntactic role. We first discard, for the influence on a role r, sentence pairs (s(l), s̃(li)) where it appears or disappears, because the presence of a syntactic role is a property of its parent word, (e.g. the presence or absence of a dobj is controlled by the transitivity of the verb) hence not directly connected to the representation of the role r itself. As they are out of the scope of our work, we report measures of these structural changes (diathesis) in Appendix C, and leave their extensive study to future works. We denote the remaining number of samples T ′decri .
In the following, we use operator 1{.}, which is equal to 1 when the boolean expression it contains is true and to 0 when it is false. This process yields a matrix Γdec of shape (|R|, NZ) which summarizes interactions in the decoder between syntactic roles and latent variables:
Γdecri = T ′decri∑ l=1 1{ρr(s(l)) 6= ρr(s̃(li))} T ′decri
(2)

heading: 5.3 ENCODER INFLUENCE ON LATENT VARIABLES
text: We compute this on a held out set of size T enc of sentences (s(l)j )1≤l≤T enc,1≤j≤N s(l) . Each sentence s(l) of sizeNs(l) generates an attention matrix (a (l) ij )1≤i≤NZ ,1≤j≤Ns(l) . Attention values are available in the Transformer Encoder with cross-attention computing the inference model7, and quantify the degree to which each latent variable embedding eenczi draws information from each token sj to form the value of zi.
For the encoder, we consider the influence of a syntactic role on a latent variable to be the probability for the attention values of the latent variable to reach their maximum on the index of a token in that syntactic role’s realization. The indices of tokens belonging to a syntactic role r in a sentence s(l) are denoted argr(s(l)). For each syntactic role r and sentence s(l), we discard inputs where this syntactic role cannot be found, and denote the remaining number of samples T ′encr . The resulting measure of influence of syntactic role r on variable zi is denoted Γencri . The whole process yields matrix Γ
enc of shape (|R|, NZ) which summarizes interactions in the encoder between syntactic roles and latent variables:
Γencri = T ′encr∑ l=1 1{arg max j (a (l) ij ) ∈ argr(s (l))} T ′encr (3)

heading: 5.4 DISENTANGLEMENT METRICS
text: For Γ∗ (either Γdec or Γenc) each line corresponds to a syntactic role in the data. The disentanglement metric for role r is the following:
∆Γ∗r = Γ ∗ rm1 − Γ ∗ rm2 (4)
s.t. m1 = arg max 1≤i≤NZ Γ∗ri, m2 = arg max 1≤i≤NZ ,j 6=m1 Γ∗ri
We calculate total disentanglement scores for syntactic roles using Γdec, Γenc as follows: Ddec = ∑ r∈R ∆Γencr , Denc = ∑ r∈R ∆Γencr (5)
In summary, the more each syntactic role’s information is concentrated in a single variable, the higher the values of Ddec and Denc. However, similar to MIG, these metrics do not say whether variables
7For simplicity, attention values are averaged over attention heads and transformer layers. This also allows drawing conclusions with regard to the tendency of the whole attention network, and not just particular specialized heads as was done in Clark et al. (2019). Nevertheless, we display per-layer results in Appendix J.
capturing our concepts of interest are distinct. Therefore, we also report the number of distinct variables that capture the most each syntactic role (i.e the number of distinct values of m1 in Eq. 4 when looping over r). This is referred to as NΓenc for the encoder and NΓdec for the decoder.

heading: 6 EXPERIMENTS
text: Dataset Previous unsupervised disentanglement works (Higgins et al., 2017; Kim & Mnih, 2018; Li et al., 2020b) tend to use relatively homogeneous and low complexity data. The data has low complexity if it varies along clear factors which correspond to what the model aims to disentangle. Similarly, we use a dataset where samples exhibit low variance in terms of syntactic structure while providing a high diversity of realizations for the syntactic roles composing the sentences, which is an adequate test-bed for unsupervised disentanglement of syntactic roles’ realizations. This dataset is the plain text from the SNLI dataset (Bowman et al., 2015) extracted8 by Schmidt et al. (2020). The SNLI data is a collection of premises (on average 8.92 ± 2.66 tokens long) made for Natural Language Inference. We use 90K samples as a training set, 5K for development, and 5K as a test set.
Setup Our objective is to check whether the architecture of our ADVAE induces better syntactic role disentanglement. We compare it to standard Sequence VAEs (Bowman et al., 2016) and to a Transformer-based baseline that doesn’t use cross-attention. Instead of cross-attention, this second baseline uses mean-pooling over the output of a Transformer encoder for encoding. For decoding, it uses the latent variable as a first token in a Transformer decoder, as is done for conditional generation with GPT-2 Santhanam & Shaikh (2019). These comparisons are performed using the same β-VAE objectives, and the decoder disentanglement scores as metrics. Training specifics and hyper-parameter settings are detailed in Appendix E. For each of the two baselines, the latent variables we vary during the decoder’s evaluation are the mono-dimensional components of its latent vector. It is easier to pack information about the realizations of multiple syntactic roles into Dz dimensions than into a single dimension. Consequently, the single dimensions we study for the baselines should be at an advantage to separate information into different variables.
Scoring disentanglement on the encoder side will not be possible for the baselines above as it requires attention values. To establish that our model effectively tracks syntactic roles, we compare it to a third baseline that locates each syntactic role through its median position across the dataset. This baseline is fairly strong on a language where word order is rigid (i.e configurational language) such as English. We refer to this Position Baseline as PB.
The scores are given for different values of β (Eq. 1). Raising β lowers the expressiveness of latent variables, but yields better disentanglement (Higgins et al., 2017). Following Xu et al. (2020), we set β to low values to avoid posterior collapse. In our case, we observed that the models do not collapse for β < 0.5. Therefore, we display results for β ∈ {0.3, 0.4}. We stop at 0.3 as lower values for β result in poorer generation quality. For our model we report performance for instances with NZ = 4 (ours-4) and NZ = 8 (ours-8).
Results The global disentanglement metrics are reported in Table 1.9
On the decoder side, the Sequence VAE exhibits disentanglement scores in the range of those reported for our model for β = 0.3, and higher for β = 0.4. However, NΓdec shows that it struggles to factor the realizations of different syntactic roles in different latent variables, and the higher score shown for β = 0.4 is accompanied by a lower tendency to separate the information from different syntactic roles. The Transformer VAE baseline assigns different latent variables to the different syntactic role (high NΓdec), but suffers from very low specialization for these latent variables (low Ddec). In contrast, our model is consistently able to separate 3 out of 4 syntactic roles, and while a higher β raises its Ddec, it does not decrease its NΓdec . As ours-8 has more latent variables, this encourages the model to further split the information in each syntactic role between more latent variables10. The fact that ADVAEs perform better than both Sequence VAEs and classical Transformer VAEs shows that its disentanglement capabilities are due to the usage of Cross-Attention to obtain latent variables, and not only to the usage of Transformers. On the encoding side, our models consistently score above the baseline, showing that our latent variables actively follow the syntactic roles.
In Figures 3 and 4, we display the influence matrices Γenc and Γdec for an instance of our ADVAE with NZ = 4 as heatmaps. The vertical axes correspond to the latent variables. As can be seen, our model successfully associates latent variables to verbs and subjects but chooses not to separate direct objects and prepositional objects into different latent variables. Upon further inspection of the same heatmaps for the VAE baseline, it appears that it most often uses a single latent variable for verb and subj, and another for dobj and pobj.
One can also notice in Figures 3 and 4, that the encoder matrix is sparser than the decoder matrix (which is consistent with the higher encoder disentanglement scores in Table 1). This is to be expected as the decoder pθ(s|z) adapts the realizations of syntactic roles to each other after they are sampled separately from p(z). The reason for this is that the language modeling objective requires some coherence between syntactic roles (conjugating verbs with subjects, changing objects that are semantically inadequate for a verb, etc). This co-adaptation, contradicts the independence of our latent variables. It will be further discussed in the following paragraph.
10Results for a larger grid of Nz values are reported in Appendix K, and show that latent variables still clearly relate to syntactic roles, but in groups.
Changing the Realizations of Syntactic Roles Here, we display of few qualitative examples of how the realizations of syntactic roles can be separately changed using an instance of our ADVAE.
As a first example, we generate a sentence from a random latent vector, then resample for each syntactic role the corresponding disentangled latent variable to observe the change on the subsequently generated altered sentence. The results of this manipulation are in Table 211. As can be seen, some examples exhibit changes that only affect the target syntactic role (example 1). However, the model often produces co-adaptations that go past the target syntactic role either for semantic soundness (example 2, resampled verb adapts the object), or simply for lack of generalization from the SNLI data used for training.
A second example we display is a swap of syntactic role realizations between sentences. A few examples are given in Table 3. Similar to Table 2, the model often yields the expected result. Coadaptation is best seen here, as taking a syntactic role to a sentence with which it is incompatible results in unexpected changes (example 3).
Further investigations As this is a first step in this research direction, we conducted this study on a dataset of relatively regular sentences. Running similar experiments on a dataset with more complicated and diverse sentence structures such as in Yelp (Appendix B) results in the same comparative patterns. However, disentanglement scores are much lower. This calls for future iterations to improve upon ADVAE and our evaluation protocol to better model structure in order to scale to User Generated Content (UGC). Our experiments also enabled underlining an inherent issue to syntactic role disentanglement: co-adaptation. The independence between our latent variables causes the decoder pθ(s|z) to correct the incoherence between independently sampled syntactic role realizations. Using structured latent variables to learn relations between syntactic roles seems to be the natural solution to this problem. An investigation of a hierarchical version of the ADVAE (Appendix A) showed, however, that a drop-in replacement of the independent prior with a structured prior is not sufficient in order to absorb co-adaptation into the latent variable model. Our future works will, therefore, also include the investigation of training techniques that can achieve improved results with structured latent variables.

heading: 7 CONCLUSION
text: We introduce a framework to study the disentanglement of syntactic roles and show that it is possible to learn a representation of sentences that exhibits separation in the realizations of these syntactic functions without supervision. Our framework includes: i) Our model, the ADVAE, which maps syntactic roles to separate latent variables more often than standard Sequence VAEs and with better concentration than standard Transformer VAEs, and allows for the use of attention to study the interaction between latent variables and spans, ii) An evaluation protocol to quantify disentanglement between latent variables and spans both in the encoder and in the decoder.
Our study constitutes a first step in a promising process towards unsupervised explainable modeling and fine-grained control over the predicate-argument structure of sentences. Although we focused on syntactic roles realizations, this architecture as well as the evaluation method are generic and could be applied to other tasks. The architecture could be used at the document level (e.g. disentangling discourse relations), while the evaluation protocol could be applied to other spans such as constituents.
11More Examples are available in Appendix H

heading: A A HIERARCHICAL VERSION OF OUR ADVAE
text: As we stated, our ADVAE aims to factor sentences into independent latent variables. However, given the dependency structure of sentences, realizations of syntactic roles are known to be interdependent to some degree in general. Therefore one may think that a structured latent variable model would be better suited to model the realizations of syntactic roles. In fact, such a model could absorb the language modeling co-adaptation between syntactic roles. For instance, instead of sampling an object and a verb from p(z) that are inadequate, then co-adapting them through pθ(s|z), a structured pθ(z) could produce an adequate object for the verb. For this experiment, rather than using an independent prior p(z), we use a structured prior pθ(z) = p(z0) ∏L l=1 pθ(z
l|zl−1) where p(z0) is a standard Gaussian, and all subsequent L−1 hierarchy levels are parameterized by learned conditional diagonal Gaussians. The model used for each pθ(zl|zl−1) is shown in Figure 5 below:
We display the results for L = 2 and L = 3 in Table 4. For both models, we set NZ to 4.
The results show lower mean disentanglement scores, and high standard deviations compared to the standard version of our ADVAE. By inspecting individual training instances of this hierarchical model, we found that some instances achieve disentanglement with close scores to those of the standard ADVAE, while others completely fail (which results in the high variances observed in Table 4). Unfortunately, hierarchical latent variable models are notoriously difficult to train (Zhao et al., 2017). Our independent latent variable model is therefore preferable to the structured one due to these empirical results. More advanced hierarchical latent variable training techniques (such as Progressive Learning and Disentanglement (Li et al., 2020b)) may, however, provide better results.

heading: B EXPERIMENTING WITH THE YELP DATASET
text: We investigated the behavior of our ADVAE of on the user-generated reviews from the Yelp dataset used in Li et al. (2018) using the same procedure we used for SNLI. The length of sentences from this dataset (8.88± 3.64) is similar to the length of sentences from the SNLI dataset. Similar to the experiments in the main body of the paper, we display the disentanglement scores in Table 5, and the influence metrics of one of the instances of our model as heatmaps in Figures 6 and 7.
Table 5: Disentanglement results for the Yelp dataset
Model β Denc NΓenc Ddec NΓdec
Sequence VAE 0.3 - - 0.44(0.09) 2.20(0.45)0.4 - - 1.21(0.06) 2.25(0.50) PB - 0.33(-) 2.00(-) - -
ours-4 0.3 0.48(0.07) 2.00(0.00) 0.18(0.02) 2.50(0.58)0.4 0.54(0.04) 3.00(0.00) 0.23(0.03) 2.40(0.55) ours-8 0.3 0.44(0.04) 3.80(0.45) 0.17(0.04) 2.80(0.84)0.4 0.57(0.26) 3.40(0.55) 0.15(0.10) 2.40(0.89)
Figure 6: Encoder influence heatmap for Yelp(Γenc).
Figure 7: Decoder influence heatmap for Yelp(Γdec).
Although the results show similar trends, they are weaker than what we obtained for SNLI. Given the difference between SNLI and Yelp (displayed in Appendix D) there are two clear reasons for this decrease. The first is that Yelp is a dataset where it is harder to locate the syntactic roles. This is illustrated by the fact that the PB baseline obtains a much lower score. The second is that our syntactic role extraction heuristics are tailored for regular sentences with verbal roots, which subjects the evaluation metrics on Yelp to a considerable amount of noise. Nevertheless, the comparisons between a VAE, an ADVAE, and PB retain the same conclusions, but with lower margins and some overlapping standard deviations.
Through manual inspection of examples, we observed that the various structural characteristics (enumerations, sentences with nominal roots, presence of coordinating conjunctions, etc) were captured by different variables. This indicates that future iterations of our model need to provide ways to separate structural information from content-related information.

heading: C MEASURING THE EFFECT OF LATENT VARIABLES ON THE STRUCTURE OF SENTENCES
text: In Figure 8, for each latent variable and each syntactic role, we report the probability that resampling the latent variable causes the appearance/disappearance of the syntactic role. The instance we use here is the same as the one we use for the heatmaps in the main body of the paper. According to the heatmaps in Figures 3 and 4, latent variable 3 is the one associated with the verb. As can be seen in the present heatmap in Figure 8, this same variable is the one that has the most influence on the appearance/disappearance of direct and prepositional objects, and this is a pattern that proved to be consistent across our different runs. This constitutes empirical justification for our choice of discarding these cases from our decoder influence metrics.

heading: D EXAMPLE SENTENCES FROM YELP AND SNLI AND THEIR CORRESPONDING SYNTACTIC EXTRACTIONS
text: Table 6 shows some samples from SNLI and Yelp reviews. Samples from Yelp Reviews exhibit a clearly higher structural diversity. On the other hand, most SNLI samples are highly similar in structure. Our syntactic role extraction heuristics were tailored for sentences with verbal roots. As a result, it can be seen that they struggle with sentences with nominal roots as well as other forms of irregular utterances present in Yelp. For SNLI, our extractions mostly yield the expected results, allowing for a reliable global assessment of our models.

heading: E TRAINING DETAILS AND HYPER-PARAMETER SETTINGS
text: Our ADVAE’s hyper-parameters Our model has been set to be large enough to reach a low reconstruction error during the initial reconstruction phase of the training. We use 2-layer Transformers with 4 attention heads and a hidden size of 192. Contrary to Vanilla VAEs, our model seems to perform better with high values of NZ . Therefore, we set our latent vector to a size of 768, and divide it into 96-dimensional variables for our NZ = 8 model and to 192-dimensional latent variables for our NZ = 4 model. No automated hyper-parameter selection has been done afterward.
Sequence VAE hyper-parameters As is usually done for this baseline (Xu et al., 2020), we set both the encoder and the decoder to be 2-layer LSTMs. We run this model for hidden LSTM sizes in [256, 512], and latent vector sizes in [16, 32]. The results for the model scoring the highest Ddec are then reported. Even though selection has been done according to Ddec, we checked the remaining instances of our baselines and they also yielded low NΓdec values.
Transformer VAE hyper-parameters We set the hidden sizes and number of layers for this baseline similarly to ADVAE, since it is also a Transformer. We run this model for latent vector sizes in [16, 32] and display the highest scoring model, as is done for the Sequence VAE.
Training phases All our models are trained using ADAM(Kingma & Ba, 2015) with a batch size of 128 and a learning rate of 2e-4 for 20 epochs. The dropout is set to 0.3. To avoid posterior collapse, we train all our models for 3000 steps with β = 0 (reconstruction phase), then we linearly increase β to its final value for the subsequent 3000 steps. Following Bowman et al. (2016), we also use word-dropout. We set its probability to 0.1.
Evaluation For the evaluation, T dec is set to 2000, and T enc is equal to the size of the test set.

heading: F DISENTANGLEMENT SCORES FOR EACH SYNTACTIC ROLE
text: The full disentanglement scores are reported in Table 7 for the decoder, and in Table 8 for the encoder.

heading: G DISENTANGLEMENT HEATMAPS OVER THE ENTIRE RANGE OF SYNTACTIC ROLES AND POS TAGS
text: We report decoder and encoder heatmaps for all the syntactic roles following the Stanford Dependencies (SD; De Marneffe & Manning, 2008) annotation scheme of Ontonotes, which was used to train our Spacy2 parser, in Figures 9 and 10. For the sake of extensiveness and to make sure we did not draw results from some parser biases, we also report the same heatmaps but using UDPipe 2.0 (Straka, 2018), which uses UD type annotations12, in Figures 13 and 14. Finally, we also report
12A widely adopted annotation scheme derived from Stanford Dependencies.
heatmaps for interaction with PoS Tags extracted with Spacy2 in Figures 11 and 12. As was done in the main body of the paper, the span corresponding to each syntactic role (in both annotation schemes) was taken to be the series of words included in its corresponding subtree. In contrast, the span corresponding to each PoS tag was just taken to be the tagged word. Results from UD parsing extraction lead to the same conclusions as from our initial SD results.
The instance of our ADVAE for which we display the above heatmaps is the same one for which we display the heatmaps in Figures 3 and 4 in the main body of the paper. As shown in those Figures, it mostly uses variable 3 for verbs, variables 2 for subjects, and variable 1 for objects. The remaining variable (0) also seems to capture some interaction with objects. The heatmaps show that our ADVAE tends to group syntactic roles into latent variables in a way that aligns with the predicative structure of sentences. In fact, variable 2 displays the highest influence on the PoS tag VERB as well as its surroundings as a predicate argument such as adverbs and adverbial phrases. Similarly, latent variable 2 displays a high influence on subjects (nominal or clausal), numeral modifiers, adjectival modifiers, and auxiliaries (for conjugation). Moreover, Variable 1 highly influences the direct and prepositional objects, which we study in the main body of the paper, but also diverse clausal modifiers and obliques which often play similar roles to direct and prepositional objects in a predicate structure.

heading: H ADDITIONAL EXAMPLES OF RESAMPLED REALIZATIONS FOR EACH SYNTACTIC ROLE
text: Table 9 contains a wide array of examples where the latent variable corresponding to each syntactic role is resampled.

heading: I RECONSTRUCTION AND KULLBACK-LEIBLER VALUES ACROSS EXPERIMENTS
text: The values for the reconstruction loss, the DKL divergence, and the upper bound on perplexity concerning the experiments in the main body of the paper are reported in Table 10. Since our models are VAE-based, one can only obtain the upper bound on the perplexity and not its exact value. These upper bound values are obtained using an importance sampling-based estimate of the negative log-likelihood, as was done in Wu et al. (2020). We set the number of importance samples to 10.
It can be seen that the behavior of ADVAEs is very different from classical Sequence VAEs and Transformer VAEs. On the plus side, they are capable of sustaining much more information in their latent variables as shown by their higher DKL, and they do better at reconstruction. The upper bound estimate of their perplexity is however higher. A high DKL makes it more difficult for the importance sampling-based perplexity estimate to reach the true value of the model’s perplexity. This may be the reason behind the higher values observed for ADVAEs.

heading: J LAYER-WISE ENCODER ATTENTION
text: In the main body of the paper, we use attention values that are averaged throughout the network. We hereby display the encoder heatmaps obtained by using attention values from the first layer (Fig. 15), the second layer (Fig. 16), or an average on both layers (Fig. 17) for comparison.
As can be seen, the first layer alone provides the most sparse heatmap, and thus, the clearest correspondence between syntactic roles and latent variables. This corroborates the claims of Tenney et al. (2020) about syntax being most prominently processed in the early layers of Transformers.
K ADVAE RESULTS FOR A LARGER GRID OF Nz VALUES
We display in Table 11 the quantitative results of ADVAE on SNLI for Nz in {2, 4, 6, 8}. For ours-2, it is normal that it only separates syntactic role realizations into a maximum of 2 latent variables, as seen from the values of NΓenc and NΓenc , since 2 is its total number of latent variables.
As observed in the main body of the paper, the increase of the number of latent variables used in ADVAE leads to dispatching the influence on the realization of a single syntactic role to multiple latent variables. This in turn,leads to for Denc and Ddec that we observe . In Figures 18 and 19, we
respectively display the encoder and decoder heatmaps of ADVAE with 16 latent variables. As can be seen in these figures, each latent variable still highly specializes in a specific syntactic role. This is seen more clearly on the encoder heatmap due to co-adaptation harming the clarity of the decoder heatmap. This specialization seems to be shared among groups (e.g. variables 4 and 8 specialize in the subject, as indicated by the green squares on the figure). This causes the difference of influence between the most influential variable and the second most influential one to be low, and thus decreases the values of Denc and Ddec.
Table 11: Disentanglement quantitative results on SNLI for a larger grid of Nz values.
Model β Denc NΓenc Ddec NΓdec
ours-2 0.3 2.01(0.07) 2.00(0.00) 0.92(0.06) 2.00(0.00)0.4 0.33(0.15) 1.60(0.55) 0.13(0.09) 1.20(0.45) ours-4 0.3 1.30(0.09) 3.00(0.00) 0.68(0.22) 2.80(0.45)0.4 1.46(0.33) 3.00(0.00) 0.81(0.05) 3.00(0.00) ours-8 0.3 1.36(0.13) 3.40(0.89) 0.60(0.10) 3.00(0.00)0.4 1.44(0.79) 3.40(0.55) 0.63(0.35) 2.80(0.45)
ours-16 0.3 0.60(0.31) 3.60(0.55) 0.33(0.30) 2.60(0.55)0.4 0.65(0.16) 3.40(0.55) 0.56(0.28) 2.60(0.55)
Figure 19: Decoder influence heatmap for ADVAE with 16 latent variables on SNLI (Γdec). Squares with similar colors highlight groups of latent variables that relate to the same syntactic role.

Review summaries:
This paper proposes a new model ADVAE, which uses a sequence of latent variables which are constructed using cross-attention, which are then used to condition the inference model. The findings show that certain latent variables correlated with different syntactic roles (measured using a dependency parser).  The work claims that the latent variables, in this case, are able to disentangle content effectively, which I am inclined to agree with, however, I feel that the experimental setup is lacking to solidly support this hypothesis (which I detail in the “cons” and “questions” section). 

The paper proposes a method for unsupervised disentanglement of text components and shows its ability to identify semantic roles. To this end, a neural network is trained to compress the input into a fixed number of independent latent variables which are regularized to be standard Gaussians via the VAE framework. The inference network consists of a Transformer encoder-decoder network, where the decoder inputs correspond to the latent variables, which cross-attend to the outputs of the Transformer encoder, i.e., the encoded sentence. The idea behind this architecture is that attention-based seq2seq architectures align source and target sequences with each other.  The model is evaluated on disentanglement of semantic roles. To this end, they investigate how resampling of individual latent variables impacts the semantic roles in the text generated by the decoder, and how syntactic roles are aggregated into latent vectors via attention. They find that their proposed architecture is more successful at disentangling semantic roles into the latent variables than standard VAEs.

This paper propose a framework to obtain the disentanglement of syntactic roles as latent variables for sentence representations. The model is an attention-driven VAE which maps syntactic roles to separate latent variables using an encoder-decoder framework. In the second part of the paper, the authors introduce an evaluation protocol to quantify disentanglement between latent variables and spans both in the encoder and in the decoder, which includes syntactic role extraction, latent variable influence on decoder, encoder influence on latent variables and disentanglement metrics.

## Summary  - This paper proposes a probabilistic model called Attention-Driven Variational Autoencoder (ADVAE). This model is another instance of $\beta$-VAE whose encoder and encoders are composed of Transformers rather than previous neural architectures such as RNN. - The authors aim to disentangle the semantics of latent variables according to some syntactic roles (e.g., nouns and verbs) defined by syntax. To achieve this goal, they suggest employing the combination of Transformer and the existing $\beta$-VAE framework which is known to be effective for disentangling the role of each latent variable in VAE. - Moreover, this work presents a new way of quantifying syntactic disentanglement between latent variables, relying on the information obtained from the attention matrices of the Transformer architecture. - The experiments show that the proposed method is quantitatively better than the normal VAE, and that swapping the value of a specific latent variable can impact the generation of the target word (decided by the syntactic role of the latent variable we choose).