Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles
Authors: Ghazi Felhi
Joseph Le Roux
Djamé Seddah
Venue: ICLR 2022 Submitted (2022)
Keywords: NLP
disentanglement
unsupervised learning
controllable generation.

Abstract: Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,$\dots$)  and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles, and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on disturbing individual latent variables for the decoder. Our experiments on raw English text from the SNLI dataset show that $\textit{i)}$ disentanglement of syntactic roles can be induced without supervision, $\textit{ii)}$  ADVAE separates more syntactic roles than classical sequence VAEs, $\textit{iii)}$ realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation.  The code for our work is publicly available.

Under review as a conference paper at ICLR 2022
TOWARDS UNSUPERVISED CONTENT DISENTANGLEMENT IN SENTENCE REPRESENTATIONS VIA SYNTACTIC ROLES
Anonymous authors Paper under double-bling review
ABSTRACT
Linking neural representations linguistic factors is crucial in order build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects.... and their realizations are essential markers since they can be understood as decomposition of predicative structures and thus the meaning of sentences. Starting from deep probabilistic generative model with attention, measure the interaction between latent variables and realizations syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose an Attention Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAES enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that i) disentanglement of syntactic roles can be induced without supervision, ii) ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAES, iii) realizations syntactic roles can be separately modified in sentences by mere intervention the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available"
INTRODUCTION
A disentangled representation of data describes information combination of separate under- standable factors. This separation provides better transparency, but also better transfer performance (Higgins et al. 2018 Dittadi et al. 2021). When it comes to disentanglement, Variational Autoen-
coders (VAEs; Kingma & Welling 2014) were extensively proven effective (Higgins et al. 2017
Chen et al.] 2018] et al.) 2019). and were used throughout several recent works (Chen et al.
2019 Li et al. |2020b John et al. 2020). . In NLP, disentanglement has been mostly performed to sep-
sentence from characteristics such style and structure in order to generate paraphrases (Chen et al. 2019 John et al.) [220 Bao et al. 2020 Huang & Chang 2021 HHuag et al. (202I). We show in our work that the information in the content itself can be separated with a VAE-based model. In contrast to the aforementioned works, we use neither supervision nor input syntactic information for this separation. We demonstrate this ability by controlling the lexical realization of core syntactic roles. For example, the subject in sentence can be encoded separately and controlled to generate the same sentence with another subject. Our framework includes a model and an evaluation protocol aimed at measuring the disentanglement of syntactic roles.
arate the semanties (or content)
The model we introduce is an Attention-Driver VAE (ADVAE), which we train on the SNLI raw text dataset (Schmidt et al. 2020). It draws its inspiration from attention-based machine translation models (Bahdanau et al. 2013 Luong et al. 2013). Such models translate sentences between languages with
URI to be disclosed upon publication. Our code is provided as supplemental material during the submission process.

---

Under review as a conference paper at ICLR 2022
different underlying structures and can be inspected to show coherent alignment between spans from both languages. Our ADVAE uses Transformers (Vaswani et al. (017). an attention based architecture, to map sentences from language to independent latent variables, then map these variables back to the same sentences. Although ADVAE could be used to study other attributes, we motivate it ($4.1) and therefore study it for the alignment of syntactie roles with latent variables.
Evaluating disentanglement with regard to spans is challenging. After training the model and only for evaluation, we use linguistic information (from an off-the-shelf dependency parser) to first extract syntactic roles from sentences, and then study their relation to latent variables. To study this relation on the ADVAE decoder, we repeatedly i) generate sentence from a sampled latent vector ii) perturb this latent vector at specific location iii) generate a sentence from this new vector and observe the difference. On the encoder side, we study the attention values to see whether each latent variable focused on particular syntactic role in input sentences. The latter procedure is only possible through the way our ADVAE uses attention to produce latent variables. To the best of our knowledge, we are the first to use this transparency mechanism to obtain quantitative results for latent variable model.
We first justify our focus on syntactic roles in $ then we go over our contribution, which is threefold: i) We introduce the ADVAE, model that is designed for unsupervised disentanglement of syntactic roles, and that enables analyzing the interaction between latent variables and observations through the values of attention ( ii) We design experimental protocol for the challenging assessment of disentanglement over realizations of syntactic roles, based on perturbations on the decoder side and attention on the encoder side (55), iii) Our empirical results show that architecture disentangles syntactic roles better than standard sequence VAEs and Transformer VAEs and that it is capable of controlling realizations of syntactic roles separately during generation ($))
2 RELATED WORKS
Linguistic information in neural models Accounting for linguistic information provided better inductive bias in the design of neural NLP systems during recent years. For instance, successful attempts at capturing linguistic information with neural models helped improve grammar induction (RNNG; Dyer et al. 2016), constituency parsing and language modeling (ON-LSTM: [Shen et al.) 2019 ONLSTM-SYD; Du et al. 2020), as well as controlled generation (SIVAE: Zhang et al. 2019). Many ensuing works have also dived into the linguistic capabilities of the resulting models, the types of linguistic annotations that emerge best in them, and syntactic error analyses (Hu et al., 2020
Kodner& & Gupta/ 2020; Marvin Linzen 2020 Kulmizev et al. 2020). Based on the Transformer architecture, the self- superviee model BERT (Devlin et al. 2019) has also been subject to studies showing that the linguistic information captures is organized among its layers in way remarkably close to the way classical NLP pipeline works (Tenney et al. 2020). Furthermore. (Clark et al.) 2019), showed that many attention heads in BERT specialize in dependency parsing. We refer the reader (Rogers al. 2020) for extensive review of Bert-related studies. However, such studies most often rely on structural probes (Jawahar et al. 2019: Liu et al. (2099] Hewitt & Manning (019)) to explain representations, probes which are not without issues, as shown by Pimentel et al. (2000). In that regard, the generative capabilities and the attention mechanism of our model offer an alternative to probing: analysis is performed directly on sentences generated by the model and on internal attention values.
Disentanglement in NLP The main line of work in this area revolves around using multitask learning to separate concepts in neural representations (e.g. style vs content (John et al. 22020). syntax vs semantics (Chen et al., 2019 Bao et al. 2020)). Alternatively, Huang & Chang (2021) and Huang al. (202T) use syntactic trees as inputs separate syntax from semantics, and generate paraphrases without paraphrase corpus. Towards less supervision, Cheng et (2020) only uses style information to separate style from content in representations. Literature on unsupervised disentanglemen
in NLP remains sparse. Examples are the work of Xu et al.] (2020) on categorical labels (sentiment and topic), and that of Behjati & Henderson (2021) 1) on representing morphemes using character-level Seq2Seq models. The work of [Behjati & Henderson (2021) is closest to ours as it uses Slot Attention (Locatello et al. 2020), which, like ADVAE, is a cross- cisss-ttteniionniae contribution depart from previous work since syntactic parses not used as learning signals but as a way to interpret our model, and ii) cross-attention enables our model to link a fixed number of latent variables to text spans.
representation technique. Our
2

---

Under review as a conference paper at ICLR 2022
3
SYNTACTIC ROLES AND DEPENDENCY PARSING
ROOT
doby
Dependency
poss
Parse
DET
ADJ
NOUN
VERB
DET
ADJ
NOUN
PoS Tags
A
talented
musician
holds
his
nice
guitar
Predicative
ARGi)
ARGI]
Structure
Figure 1: A sentence and its syntactic roles. The correspondence between syntactic roles and elements of the predicative structure is highlighted with colors.
We present in Figure 1 an example sentence with its dependency parse" its Part-of- Speech (PoS) tags, and flat predicative structure with PropBank-like semantic roles (Palmer al. (2005.. Dependency parsing yields tree, where edges are labeled with syntactic roles (or relations or functions) such nominal subject (nsubj). The lexical realizations of these syntactic functions and correspond to syntactic constituents. For instance. the lexical realization of the direct object (dobj) of the verb holds in this sentence is the span his nice guitar, with guitar as head. In short, the spans corresponding to subtrees consist of tokens that are more dependent of each other than of the rest of the sentence. As consequence, and because a disentanglement model seeks independent substructures in the data, we expect such model to converge to representations that display separation in realizations of frequent syntactic roles.
textual spans
In our work, we focus within the same framework. on nominal subjects, verbal roots of sentences, and direct or prepositional objects. These are core (as opposed to oblique; see Nivre et al. (2016) for details the distinction) syntactic roles, since they directly relate to the predicative structure. In fact in most cases, as illustrated in Figure [ the verbal root of a sentence its main predicate, the nominal subject its agent (ARGO) and the direct prepositional object its patient (ARGI).
4 MODEL DESCRIPTION
The usual method to obtain sentence representations from Transformer models uses only a Trans- former encoder either by taking average of the token representations or by using the representation of special token (e.g [CLS] in BERT(Devlin et al. 2019)). Recently, the usage of both Transformer encoders and decoders has also been explored in order to obtain representations whether by designing classical Autoencoders (Lewis et al.] 2019: Siddhant et al.] 2019: Raffel et al.) 22200), or VAEs (Li al. 2020a). Our model, the ADVAE, differs from these models in that it uses targets in Machine Translation (MT) Transformers (i.e an encoder and a decoder) to produce sentence representations. has been introduced by Locatello et al. (2020) as part
Producing representations with
of the Slot Attention modules in the context of unsupervised object discovery. However, in contrast to Locatello et al. (2020), we simply use Cross-Attention as is found in Vaswani et al. (2017).
i.e. without normalizing attention weights over the query axis, or using GRÜsíCho al. 2014) to update representations. As will be shown through our experiments, this is sufficient to disentangle syntactic roles. We explain the observation that motivates our work in I we then describe in $4.2
the minimal changes we apply to MT Transformers, and finally, we present the objective we use in The parallel between our model and MT Transformers is illustrated in Figure]
$4.3
4.1 THE INTUITION BEHIND OUR MODEL
Consider (8;) ISEEN, and L = (lj) two series of tokens forming respectively a sentence source language and sentence target language. Given s, attention- baase translation models
Folllowigg the CleaNNLP constituent dependency conversion, close to Stanford Dependencies de Marneffe gu de inee /bl ob ob/master]
Manning (2008). See https:://i thub. wtt.:/ooocccllloo..oo.cc
md/ gogmponnents/deeeddden.
cot.ereovvciic..eecoo in.
Future research that takes interest in the finer-grained disentanglement of content may simply study larger
array of syntactic roles. Using our current system we display results including all syntactic roles in Appendix G
3

---

Under review as a conference paper at ICLR 2022
fy. b
Ph phy MNg
di 02g onyy N2
S1. 92 mir SX
Transformer Decoder
Transformet Enender
Transformer Decoder
MLP
MLP
MLP
Masked Self Attention Cross-atiention to align source informa- tion with target
Cross-attention to align source information with latent variable informa- fion
Masked Self Attention
Cross-atiention
Transformer Encoder
Transformer Encoder
Transfomer Encoder
align latent variable information with source
Positional Esccoding
Positional Encoding
Positional Encoding
Positional Encoding
z- z-leentifier
41 Sg.... Sy.
E11 on Pu
St
Sg. SN.
it 2gg on RNy
St1 84g ooe 8N.-1
(a) An MT Transformer
(b) Our encoder
(c) Our decoder
In blue, we highlight in (b) the difference between our encoder and a source-to-target MT model, and in (c) the difference between our decoder and a target-to-soure MT model. The input at the bottom right for the Transformer Decoders in (a) and (c) is the series of previous words for autoregressive generation. The input to our model a series of words 8, at the bottom left of (b), and its output is the reconstruction of these words in the same language, at the top right of (c).
Figure 2:
are capable of yielding while also providing information about the alignment between the groups of tokens (of different sizes) in both sentences (Bahdanau et al.) [2015] Luong et al., (2015).. This evidence suggests that attention-basee architectures are capable of factoring information from groups of words according to source structure, and redistributing it according to target structure.
The aim of our design is to use, as a target, set of Nz independent latent variables that will act as fixed placeholders for the information in sentences. We stress that Nz is fixed and independent of the input sentence size N.. Combining Transformers, an attention-based MT model, and the VAE framework for disentanglement, our ADVAE is intended to factor information from independent groups of words into separate latent variables. the following sections, we will refer to this set of independent latent variables the latent vector (2)1siiN2 and to each 2; latent variable.
4.2 MODEL ARCHITECTURE
This is the inference model 4s (encoder in Fig. 2b) for our latent variables (z)) It differs from MT Transformer in two ways. First it uses as input sentence 8, and N2 learnable vectors ( instead of the target tokens used in translation. These We stress that these learnable vectors are input-inee couuutddddeedeet.. Second its output not used to select a token from a vocabulary but rather passed to linear layer (resp. a linear layer followed by a softplus non-linearity) to yield the mean parameters (p:)isigNz (resp. the standard deviation parameters (ai) to parameterize the diagonal Gaussian distributions (9g (5) (2(1s))ssiSN> The Transformer Decoder is therefore replaced in Fig b by Transformer Encoder that uses Cross-attention factor information
Inference model:
learnable vectors will go through Cross-Attention without Self-Attention
from the sentence. The distribution of the whole latent vector is simply the product of Gaussians II Nz (4) 9 (zgls).
(s(2) ................................ 2 (s)
Generation model: Our generation model consists of autoregressive decoder (Fig. II, N. po(sy) Sejj ....... =Ng) where sci the series of tokens preceding
e
pe(s)z1....,,2x 2Nz)
Sc. and prior assuming independent standard Gaussian variables, Lee. p(=1, .......,,,............ ((=,,...,,N,))) II, p(:)).
Each latent variable is concatenated with associated learnable vector de (z-Identifier in Fig. (]c) instead of going through positional encoding. From there on, the latent variables are used like source tokens in MT Transformer.
4

---

Under review as a conference paper at ICLR 2022
4.3 OPTIMIZATION OBJECTIVE
We train our ADVAE using the 3-VAB (Higgins et al., 2017) objective, which is the Evidence Lower-Bound (ELBo) with a controllable weight on its Kullback-Leibler (KL) term:
log po 2 ((s)rgo(=lss) [log [=]]]] - (1k)
(I)
our latent vector and the distributions (s(s) po (s|2)) ()(2)))) ( dd and 4 (2)s) are respectively the generation model and the inference model. We use a standard Gaussian distribution as prior p(2) and a diagonal Gaussian distribution as the approximate inference distribution (z(s)) The weight / is used (as in Chen et al. 2018 Xu et al. 2020 Li et al.
In Eq. ! is a sample from our dataset,
2020b) to control disentanglement, but also to find balance between the expressiveness of latent variables and the generation quality.
5 EVALUATION PROTOCOL
In order to quantify disentanglement, we first measure the interaction between latent variables and syntactic roles. To do so, extract core syntactic roles from sentences according to the procedure we describe in $5.1 Subsequently, for the ADVAE decoder, we repeatedly perturb latent variables and measure their influence the realizations of the syntactic roles in generated sentences ($5.2). For the ADVAE encoder, we use attention to determine the syntactic role that participates most in producing the value of each latent variable (5..)
Given these metrics, we measure disentanglement taking inspiration from the Mutual Information Gap (MIG:[Chen
Chen et al. 2018) in $5.4 MIG consists in measuring the difference between the first and second latent variables with the highest mutual information with regard to target factor. is intended quantify the extent which target factor is concentrated in single variable. This metric assumes knowledge of the underlying distribution of the target information in the there no straightforward or agreed-upon way set this distribution for text spans, and therefore to calculate MIG in our case. As workaround, we use the influence metrics defined in $5.2 and $5.3 as a replacement for mutual information to quantify disentanglement.
5.1 SYNTACTIC ROLE EXTRACTION
We use the Spacy dependency parser (Honnibal & Montami 2017 trained on Ontonotes5 (Weischedel al., 2013). For each sentence the realization of verb root of the dependency tree if its POS tag is VERB. Realizations of subj (subject), dobj (direct object), and pobj (prepositional object) are spans of subtrees whose roots are labelled resp. nsubj, dobj, and pobj.
In the rare cases where multiple spans answer the requirement for syntactic role, we take the first one as the subsequent spans are most often part of a subordinate clause. A realization of a syntactic role in R (perb, subj, dobj, pobj} is empty if no node in the dependency tree satisfies its extraction condition
5.2 LATENT VARIABLE INFLUENCE ON DECODER
Intuitively, repeatedly compare the text generated from sampled latent vector to the text generated using the same vector where only one latent variable is resampled. Thus we can isolate the effect of each latent varioble on output text and gather statistics.
More precisely, we sample deee latent vectors (2(1) ssssTde = ( Diststde Then for
each z and for each i we create altered version g(0)) ) where resample only
the ith latent variable (i.e. Vi' i, z((i) (1)
Generating the corresponding sentence 5y with p=(5)2) yields a list of original sentences (s()) /gigTioe and a matrix of sentences displaying the effect modifying each latent variable
https://spacy:
Examples of syntactic role extractions can be found Appendix 'Throughout this work, we use greedy sampling (sampling the maximum maximum-prohability word each step), for all generated sentences.
5

---

Under review as a conference paper at ICLR 2022
(8(1) IsIgTder, N For each syntactic role R, we will denote the realization extracted from sentence with (s))
To measure the influence a variable 2; on the realization of a syntactic role r, denoted T we estimate the probability that change this latent variable incurs change in the span corresponding to the syntactic role. We first discard, for the influence on role sentence pairs (s(O) 3(3) where appears or disappears, because the presence a syntactic role is a property of its parent word, (e.g. the presence or ahsence of dobj is controlled by the fransitivity of the verb) hence not directly connected to the representation of the role r itself. As they are out the scope our work, we report measures of these structural changes (diathesis) in Appendix C and leave their extensive study to future works. We denote the remaining number of samples T
In the following, we use operator [(-, which is equal true and to 0 when it is false. This process yields a matrix pdles of shape (,)), N2) which summarizes interactions the decoder between syntactic roles and latent variables:
when the boolean expression it contains is
Tridac
(pr(s # pr(((()))
ddee rs
Tidec
5.3 ENCODER INFLURNCE ON LATENT VARIABLES
We compute this on a held out of size Tene of sentences 1sigreee.sss ISISN,
Each sentence
(!)) of size Nga generates an attention matrix (a.) DISIENN Attention values are available in the Transformer Encoder with rross-attention computing the inference mode ? and quantify the degree to which each latent variable embedding ene draws information from each token sj to form the value of zi.
For encoder, we consider the influence of a syntactic role on a latent variable to be the probability for the attention values of the latent variable reach their maximum the index of token in that syntactic role's realization. The indices of tokens belonging to a syntactic role in a sentence s(!) are denoted arg, (9). For each syntactic role r and sentence g()), we discard inputs where this syntactic role cannot be found, and denote the remaining number of samples Thene, The resulting measure of influence of syntactic role r on variable 2; is denoted yi The whole process yields matrix I rene of shape (|R), N22 which summarizes interactions in the encoder between syntactic roles and latent variables:
Trene 1 {arg max E (s
pene
j
(3)
Thene
5.4 DISENTANGLEMENT METRICS
For T* (either dee Tenc) each line corresponds syntactic role in the data. The disentanglement metric for role is the following:
AT
T Pmz T
(4)
rm1
s.t. my = arg max TM IsiENz
rit m2
arg max
NiiSNz.imml
We calculate total disentanglement scores for syntactic roles using deee peac as follows:
Daec=
-Ar Denc
Σ> Arene
(5)
In summary, the more each syntactic role's information is concentrated in single variable, the higher the values of Dace and Denc. However, similar to MIG, these metrics do not say whether variables
For simplicity. attention values are averaged over attention heads and transformer layers. This also allows drawing conclusions with regard to the tendency of the whole attention network, and not just particularspeciiaizeed heads as done Clark al. (201)). Nevertheless, display per-layer results in Appendix !

---

Under review as a conference paper at ICLR 2022
capturing our concepts of interest distinct. Therefore, also report the number of distinct variables that capture the most each syntactic role (i.e the number of distinct values of m, in Eq. when looping over r). This is referred to Nrone for the encoder and Ny for the decoder.
6 EXPERIMENTS
Dataset Previous unsupervised disentanglement works (Higgins et al. 2017; Kim & Mnih 2018 Li al. 2020b) tend to use relatively homogeneous and low complexity data. The data has low complexity if it varies along clear factors which correspond to what the model aims to disentangle. Similarly, we use a dataset where samples exhibit low variance in terms of syntactic structure while providing high diversity of realizations for the syntactic roles composing the sentences, which is an adequate test-bed for unsupervised disentanglement is the plain text from the SNLI dataset (Bowman et al. 2015) extracted @ by Schmidt et al. (2020). The SNLI data is a collection of premises (on average 8.92 = 2.66 tokens long) made for Natural Language Inference. We use 90K samples training set, 5K for development, and 5K as test set.
syntactic roles' realizations. This dataset
Setup Our objective is to check whether the architecture of our ADVAE induces better syntactic role disentanglement. We compare it to standard Sequence VAEs (Bowman et al. 2016) and to Transformer- based baseline that doesn't It use cross-attention. Instead of gross-attentie baseline uses mean-pooling over the output of a Transformer encoder for encoding. For decoding, uses the latent variable as a first token in a Transformer decoder, as is done for conditional generation with GPT-2 [Santhanam & Shaikh (2019). These comparisons objectives, and the decoder disentanglement scores as metrics. Training specifics and hyper-paramette settings are detailed Appendix [ For each of the two baselines, the latent variables we vary during the decoder's evaluation are the mono-dimensional components of its latent vector. It is easier to pack information about the realizations of multiple syntactic roles into D. dimensions than into a single dimension. Consequently, the single dimensions we study for the baselines should be at an advantage to separate information into different variables.
this second
performed using the same 8- 3-VAE
Scoring disentanglement on the encoder side will not be possible for the baselines above it requires attention values. To establish that our model effectively tracks syntactic roles, we compare it to a third baseline that locates each syntactic role through its median position across the dataset. This baseline is fairly strong on a language where word order is rigid (i.e configurational language) such as English. We refer to this Position Baseline PB.
The scores are given for different values (Eq. D. Raising 3 lowers the expressiveness of latent variables, but yields better disentanglement (Higgins et al. 2017). Following Xu al. (2020), set 39 to low values to avoid posterior collapse. In our case, we observed that the models do not collapse for 3 < 0.5. Therefore, we display results for 3 {0.3, 0.4). We stop at 0.3 lower values for result in poorer generation quality. For our model we report performance for instances with N2 = 4 (ours-4) and Nz 8 (ours-8).
Results
The global disentanglement metrics are reported in Table
Table 1: Disentanglement quantitative results for the encoder (enc) and the decoder (dec). Nr indicates the number of separated syntactic roles, and ID measures concentration in a single variable. Values are averaged over 5 experiments. The standard deviation between parentheses.
Model
Done
Dageo
Nyww
0.3
0.60X0.09)
2.40(0.55)
Sequence VAE
04
1.28/0.24)
1.40(0.55)
0.3
0.12(0.10)
0.00(0.70)
Transformer VAE
0.4
0.11(0.04)
3.20(0.44)
PB
0.98 (-)
3.00-)
0.3
1,8(0.13)
3.00(0.00)
0.7100.06)
3.00(0.00)
ours-4
0.4
1.43/0.79)
3.00(0.00)
0.72(0.37)
2.80(0.45)
0.3
1,34/0.18
18)
3.80(0.45
0.51(0.14)
2,80(0
ours-8
0.4
1.75(0.47)
2.80/0.45)
0.98(0.27)
2.60(0.89)
hui.. Fiie-ggrinnl scores are given Appendix
wwittub.cco/sseee.ee com com/schmiin http://ff-ggeeratioon//mo
7

---

Under review as a conference paper at ICLR 2022
077
0.0
0.005
0:15
06
0.025
0-10
3.00
0.035
0.25
0.32
0.81
6.1
0.0003
0.3
0.53
0.10
0.21
0.2
0.17
64M
veb
600
posj
ver
pold
Figure 3: Encoder influence heatmap (I' Figure 4: Decoder influence heatmap (I (Idee)..
Table 2: Resampling specific latent variable for sentence. The ID column is an identifier for the example.
1D Original sentence
Resampled subject
Resampled verb
Resampled dobj/pohj
people are sitting on the beach
young man is sitting on the beach
people are playing in the beach
people are sitting on a bench
2
man and woman are sitting on couch
man is sitting on park bench
man and woman are running field
the man and woman beach
grassy
3 a man is playing with his dog
boy is playing in the snow
a man is selling vegeta- bles
a man playing the game with his goal.
On the decoder side, the Sequence VAE exhibits disentanglement scores in the range of those reported for our model for 0.3, and higher for 0.4. However, NNrce shows that struggles to factor the realizations of different syntactic roles in different latent variables, and the higher score shown for 3 0.4 is accompanied by lower tendency separate the information from different syntactic roles. The Transformer VAE baseline assigns different latent variables to the different syntactic role (high Nyuce), but suffers from very low specialization for these latent variables (low Dee.). In contrast, our model is consistently able to separate 3 out of 4 syntactic roles, and while higher raises its Dec. it does not decrease its Nyaae. As ours-8 has more latent variables, this encourages the model to further split the information in each syntactic role between more latent variabled The fact that ADVAEs perform better than both Sequence VARs and classical Transformer VAEs shows due the usage of Cross-Atttention to obtain latent variables,
that its disentanglement capabilities and not only to the usage of Transformers. On the encoding side, our models consistently score above the baseline, showing that our latent variables actively follow the syntactic roles.
In Figures and 4 we display the influence matrices eene and dee for an instance of our ADVAE with N2
heatmaps. The vertical axes correspond to the latent variables. As can be seen, our model successfully associates latent variables to verbs and subjects but chooses not to separate direct objects and prepositional objects into different latent variables. Upon further inspection of the same heatmaps for the VAE baseline, it appears that it most often uses a single latent variable for verb and subj, and another for dobj and pobj.
One can also notice in Figures 3 and 4 that the encoder matrix is sparser than the decoder matrix (which consistent with the higher encoder disentanglement scores Table [T.. This to be expected as the decoder (s(s)2) adapts the realizations of syntactic roles to each other after they are sampled separately from p(z). The reason for this is that the language modeling objective requires some coherence between syntactic roles (conjugating verbs with subjects, changing objects that are semantically inadequate for verb, etc). This co-adaptation, contradicts the independence of our latent variables. It will be further discussed in the following paragraph.
ReRsusss for larger grid of N₂ values reported in Appendix K and show that latent variables still clearly relate syntactic roles, but in groups.
8

---

Under review as a conference paper at ICLR 2022
Table Swapping the value of specific latent variable between two sentences. The SSR (Swapped Syntactic Role) column indicates the syntactic role that has been swapped.
ID
Sentence
Sentence 2
SSR
Swapped Sentence
Swapped Sentence 2
a woman is talking
people are sitting on
people are talking on subj
woman is sitting on the beach
on train
the beach
a train
people are sitting on the beach
woman talking on a train
verb
people are talking on a beach
woman is standing on a train
dobj/
3
a woman is talking on a train
a man is playing
a man is playing the guitar with a goal
woman is perform-
pobj
with his dog
ing a trick
Changing the Realizations of Syntactic Roles Here, we display of few qualitative examples of how the realizations of syntactic roles can be separately changed using an instance of our ADVAE.
As a first example, we generate sentence from a random latent vector, then resample for each syntactic role the corresponding disentangled latent variable to observe the change on the subsequently generated altered sentence. The results of this manipulation are in Table -| As can be seen, some examples exhibit changes that only affect the target syntactic role (example 1). However, the model often produces co-adaptations that go past the target syntactic role either for semantic soundness (example resampled verb adapts the object), simply for lack of generalization from the SNLI data used for training.
A second example we display is a swap of syntactic role realizations between sentences. few examples are given in 3 Similar to Table the model often yields the expected result. Co- adaptation is best seen here, taking a syntactic role a sentence with which it is incompatible results in unexpected changes (example 3).
Further investigations As this a first step in this research direction, we conducted this study on dataset of relatively regular sentences. Running similar experiments on a dataset with more complicated and diverse sentence structures such as in Yelp (Appendix B) results in the same comparative patterns. However, disentanglement scores are much lower. This calls for future iterations to improve upon ADVAE and our evaluation protocol to better model structure in order to scale to User Generated Content (UGC). Our experiments also enabled underlining inherent issue to syntactic role disentanglement: co-adaptation. The independence hetween our latent variables causes the decoder (s))) to correct the incoherence between independently sampled syntactic role realizations. Using structured latent variables to learn relations between syntactic roles seems to be the natural solution to this problem. An investigation of hicrarchical version of the ADVAE (Appendix A showed, however, that a drop-in replacement of the independent prior with a structured prior is not sufficient in order to absorb co-adaptation into the latent variable model. Our future works will, therefore, also include the investigation of training techniques that can achieve improved results with structured latent variables.
CONCLUSION
We introduce framework to study the disentanglement of syntactic roles and show that is possible to learn a representation of sentences that exhibits separation in the realizations of these syntactic functions without supervision. Our framework includes: i) Our model, the ADVAE, which maps syntactic roles to separate latent variables more often than standard Sequence VAEs and with better concentration than standard Transformer VAES, and allows for the use of attention to study the interaction between latent variables and spans, ii) An evaluation protocol to quantify disentanglement between latent variables and spans both in the encoder and in the decoder.
Our study constitutes a first step in promising process towards unsupervised explainable modeling and fine- fine-grained control over the predicate-argume syntactic roles realizations, this architecture well as the evaluation method are generic and could be applied to other tasks. The architecture could be used at the document level (e.g. disentangling discourse relations), while the evaluation protocol could be applied to other spans such as constituents.
structure of sentences. Although we focused on
More Examples available in Appendix H
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pp. 1-15, 2015.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. Generating sentences from disentangled syntactic and semantic spaces, ACL 2019 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp. 6008 6019, 2020. doi: 10. 10.18653/v1 1/119/1621
Melika Behjati and James Henderson. Inducing meaningful units from character sequences with slot attention, 2021.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large
annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 632 632-642, Lishon, Portugal, Septemher 2015, Association for Computational Linguistics. doi: 10. 10.18655//1//15--077. URL https
www. aclweb. org/ wwtp:o/gg///oooooooooo http//nnooooogyyoleee 1075
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. CoNLL 2016 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings, pp. 10-21, 2016. doi: 10.18653/v1/
k00- URLhttp::/// http://axxvv..o / wttp:o/gggaa....eeteeeeeee 06349
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. A multi-task approach for disen- tangling syntax and semantics in sentence representations. NAACL HLT 2019 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings of the Conference, 1:2453 22455-22444, 2019, doi: 10. (0.18653/v1/1199 1254.
URL
wwwpexivv..rr..eeooooo abs/ 904 01173
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In 6th International Conference on Learning Representations, ICLR 2018 Workshop Track Proceedings, 2018.
Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. Improving Disentangled Text Representation Learning with Information- Theoretic Guidance. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 7530-7541, 2020. doi: 10. 18.18653/11/200 1/253///12200aalmmmmam
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decodes for statistical machine translation, 2014.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What Does BERT Look at? An Analysis of BERT's Attention. In BlackBoxNLP@ACL. 2019. doi: 10. 1/w 19-4828. URL httl:http://aaate org hbtp///bs//990 04341
de Marneffe and Christopher D. Manning. The Stanford typed dependencies representation. In Coling 2008: Proceedings of the workshop Cross-Framework and Cross- Domain Parser Evaluation, pp. 1-8, Manchester, UK, August 2008. Coling 2008 Organizing Committee. httttttss//cclaaaee
Marie-Catherine De Marneffe and Christopher D Manning. Stanford typed dependencies manual. Technical report, Technical report, Stanford University, 2008
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL HLT 2019 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies Proceedings of the Conference, 1:4171-4186 2019. ISSN 0140-525X. doi:
181 03600v2.. URL htlp:ttt:11 httt:://axxiii wttpr/iwwwwgg/aaa///////. 04805
10

---

Under review as a conference paper at ICLR 2022
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. In International Conference Learning Representations, 2021. URL
httts::/appenev/ ew. net/ forrm/oormmmmm
Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J O' O'Donnell, Yoshua Bengio, and Yue Zhang. Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
6611-6628, 2020. doi: 10. 10.18665/v1/0220.cco main-mm....991 URL https: /top:://www.o aclweb. org
/anthology/20222 2020 acl main. 591/
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. Recurrent neural network grammars. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 Proceedings of the Conference, doi: 10. 18653/v1/n16- 1024.
pp. 199-209, 2016.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Humun Language Technologies, Volume (Long and Short Papers), pp. 1129-4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/119-149.. URL
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. B-VAE: Learning basic visual concepts with constrained variational framework. 5th International Conference on Learning Representations, ICLR 2017 Conference Track Proceedings, pp. 1-22, 2017.
Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerehner. Darla: Improving zero-shot transfer in reinforcement learning, 2018.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P Levy. A Systematic assessment of syntactic generalization in neural language models. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1725-1744, 2020. doi: 10. 10.18653/w12222...o
1-main. URL
https: /wwww awwwab.ww...... org/ http://ahhhooooggooeeeaae acl ma mwin.........m. 158
James Y. Huang, Kuan-Hao Huang, and Kai-Wei Chang. Disentangling semanties and syntax in sentence embeddings with pre-trained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1372-1379, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 naacl- main. 108. URL ain. 108
naacl-m
Kuan-Hao Huang and Kai-Wei Chang. Generating syntactically controlled paraphrases without using annotated parallel pairs. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1022-1033. Online, April 2021.
Association for Computational Linguistics. URL
ea
cl-mainn --main. 88
Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Ganesh Jawahar Beno"it, it, and Beno"it Sagot. What does BERT learn about the structure of (ACL219), 2019. URL
ria. fr atr/hal-02331
Vincet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for nar-parallel text style transfer. In ACL 2019 57th Annual Meeting of the Associ- ation for Computational Linguistics, Proceedings of the Conference, pp. 434, 9781950737482. doi: 10. 10.186533v1/p11-104
434, 2020. ISBN
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. 35th International Conference on Machine Learning, ICML 2018, 6:4153-4171, 2018

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. ISBN 9781 9781450300728. doi: 10.1 14551133144333/ 8118883088..3000000 1830503. htR.httt::/ httphhtpp///xiiiii org/ab
s/1412 //1412. 6980
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014. Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//axxiv. abs 1312. 6114
Jordan Kodner and Nitish Gupta. Overestimation of syntactic representation in neural lan- guage models. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 1757-1762, 2020. doi: 10.18653/1/22221
main. 160. URL
https://wwed wwwpww/wwwwwbbb....
main. 160
Artur Kulmizev, Vinit Ravishankar, Mostafa Ahdou, and Joakim Nivre. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4077-4091, 2020. doi: 10.18653/1/2020..ee acl-mmallmmiinn
375. URL https: //wp::::/wwwww.// wwww///wwoommmoo wwwwwllww...... http://....aacoogy////aaa
acl wecc-mmiine in. 375
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension, 2019.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Optimus: Organizing sentences via pre-trained modeling of latent space, 2020a.
Juncen Li. Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1865-1874, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v//18- 1169. URL httph::t/////. www. www.wa..www..................... http://wwww.eeb....aaot.ooo.ooea
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progres- sive Learning and Disentanglement of Hierarchical Representations. arXiv, 2 2020b. ISSN 233 23318422. URL v. or..or//aa abs 2002. 1549
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1073-1094, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1011-11..
URL https
7/aclanthology. org/N19-20 1112
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
Minh Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- neural machine translation. Conference Proceedings EMNLP 20/5: Conference on Empirical Methods in Natural Language Processing, pp- 1412-1421, 2015. doi: 10. 18653/v1/d15-1 166.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. pp. 1192-1202, 2020. doi: 10. 1/d1--/////111/ 1151.
Joakim Nivre, Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. Universal Dependencies v1: A multilingual treebank collection. In Proceedings
de Marneffe, Filip Ginter, Yoav Goldberg. Jan Hajië, Christopher D.
12

---

Under review as a conference paper at ICLR 2022
of the Tenth International Conference Language Resources and Evaluation (LREC' 16), pp. 1659-16666 1666, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL httppsss//aaaaaatollaaaaao
org. 1262
Martha Palmer, Daniel Gildea, and Paul Kingsbury. The proposition bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71-106, 2005.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. cooommationtthorretii probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609 4622, Online, July 2020. Association for Computational Linguistics. doi: (0.18653/v1/2220.all main.420. URL
https://acl www.aaccnnnlloooooooooooooo www.oogg...oooooooooo.. 2020.acl0ccc.......
420
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What we know about how BERT works. arXiv, 2020.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June: 1tw.:::::::::....... 1398-124077 2019, ISSN 10636919, doi: 10.1109/CVPR.200 01269.
Sashank Santhanam and Samira Shaikh. Emotional neural language generation grounded in situational contexts. In Proceedings of the 4th Workshop Computational Creativity in Language Generation, pp. 22-27, Tokyo, Japan, 29 October-3 November 2019. Association for Computational Linguistics. URL https:/ www. a c web. org.anthologgttooo /219
2019 ccnlg-1.3
Florian Schmidt, Stephan Mandt, and Thomas Hofmann. Autoregressive text generation beyond feedback loops. In EMNLP-IJCNEL 2019 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference Natural Language Processing, Proceedings of the Conference, number 2003, pp. 3400-3406, 2020. ISBN 9781950737901. doi: 10.18653/v1/dI 1/11//////1--133...
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. In 7th International Conference on Learning Representations, ICLR 2019, pp. 1-14, 2019.
Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. Evaluating the cross-lingual effectiveness of massively multilin- gual neural machine translation, 2019.
Milan Straka. UDPipe 2.0 prototype at CONLL 2018 UD shared task. Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pp. 197-207, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18655/v1/K18-2000
www a c. web. www.:........ta.ee.. ogy/ 2020
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. ACL 2019 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp. 4593 4601, 2020. doi: 10. (0.18653/v1/pll 1tpt//////11////// 1452. URL http:/://xxxvvoo///// abs 0055/995.055 05950
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. NewrlPS, number Nips, 2017. ISBN 1469-8714. doi: 10. 10.1017/5095252388 1003833300000. URL wtt.:ttt:tt/////..t http::://axi...rg//// / 03762
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. OntoNotes Release 5.0, 2013. URL https:: / //hll handle net
711272 121272.22 AR2 JJ2R
13

---

Under review as a conference paper at ICLR 2022
Chen Wu, Prince Zizhwang Wang, and William Yang Wang. On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond. 2020. URL /.R/tttp:///xxaw.oo/////// 2004
09189
Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. On Variational Learning of Controllable Representations for Text without Supervision. The 37th International Conference on Machine Learning (ICML 2020). 2020. URtp:tt: //rtpi wwg. abs /10./.9955... 11975
Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, and Lawrence Carin. Syntax-Infun Variational Autoencoder for Text Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2069-2078, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. ISBN 9781950737482. doi: 10.18653/v1/ P19-1199. URL Utt.:tt...//aaaaaccc////////....
org
/anthoology/PII9-113
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gen- erative models. 34th International Conference on Machine Learning, ICML 2017, 8:6195-6200
2017.
14

---

Under review as a conference paper at ICLR 2022
A A HIERARCHICAL VERSION OF OUR ADVAE
As stated, our ADVAE aims factor sentences into independent latent variables. However, given the dependency structure of sentences, realizations of syntactic roles are known to be interdependent to some degree in general. Therefore may think that structured latent variable model would be better suited to model the realizations of syntactic roles. In fact, such a model could absorb the language modeling co-adaptation between syntactic roles. For instance, instead of sampling object and a verb from p(z) that are imadequate, then co-adapting them through p=(s(2), a structured p=(z) could produce an adequate object for the verb. For this experiment, rather than using an independent prior p(z), we use a structured prior pe (z) = p(=") II Po (=(="|2'') |=l=1) where p(=9) is standard Gaussian, and all subsequent L 1 hierarchy levels are parameterized by learned conditional diagonal Gaussians. The model used for each po (= (2111 is shown in Figure helow:
pi p2
Mivz
o oz
Transformor Decuder
MLP
Self Attention + Cross- attention to align source information with latent variables
Transformer Encoder
Z- identifier
..
(b)
Figure 5: The conditional inference module linking each of the hierarchy levels in our prior with the next level (=(2"| (= This module treats latent variables from previous layers as they are treated in our original decoder, and generates parameters for latent variables in subsequent hierarchy levels as done in our encoder.
We display the results for I, and Table For both models, we set Nz to
Table 4:
Disentanglement results for structured latent variable models on SNLI.
Depth
Besr
Ny...
Nri::
L
0.79(0.36)
3.600.55)
0.51(0.22)
2.60(0.55)
0.4
0.42(0.23)
2.800.45)
0.12/0.20 20)
2.20:0.45)
0.3
0.90(0.25)
3.14(0.6))
0.52/0.20)
2.43(0.53)
0.4
0.32(0.38)
2.75(0.50)
0.25(0.42)
2.25(0.50)
The results show lower mean disentanglement scores, and high standard deviations compared to the standard version of our ADVAE. By inspecting individual training instances of this hierarchical model, we found that some instances achieve disentanglement with close scores to those of the standard ADVAE, while others completely fail (which results in the high variances observed in Table 4). Unfortunately, hierarchical latent variable models are notoriously difficult to train (Zhao et al. 2017). Our independent latent variable model is therefore preferable to the structured one due to these empirical results. More advanced hierarchical latent variable training techniques (such as Progressive Learning and Disentanglement (Li al. 2020b)) may, however, provide better results.
15

---

Under review as a conference paper at ICLR 2022
B EXPERIMENTING WITH THE YELP DATASET
We investigated the behavior of our ADVAE of on the user-generated reviews from the Yclp dataset used in Li al. 2018) using the same procedure we used for SNLI. The length of sentences from this dataset (8. 88 3.64) is similar to the length of sentences from the SNLI dataset. Similar to the experiments in the main body of the paper, display the disentanglement scores in Table 5 and the influence metrics of one of the instances our model as heatmaps in Figures and
Table Disentanglement results for the Yelp dataset
Nyve
Sequenee VAE
0.440.099
2.25(0.50)
2,20:0455
0,33
2,00
2.50
3,00/0.00)
0.2300055)
2.40/0.55)
oun--8
0.3
0.4
0.4400.143
0.57:0.26)
3.40/0.55)
3,80104555
0.15(0.10)
2.8000044
2.40/0.89)
0.28
0.85
0.15
6.2
0.36
0.21
0:24
0.015
6.16
0.095
0.0012
0.0025
021
0.07
0:31
0.13
0.0007
0.017
0.12
0.16
0.3
0.029
0.017
0.03
0.033
0.1
0.044
0.2
0.1
0.013
0.038
0.21
0.29
0.05
0.025
0.056
0.061
0.1
0.05
0.78
0.097
0.0088
0:32
0.0
0.058
0.078
0.057
0.0
subj
velb
daiy
pob)
sub)
vefh
daty
pot)
Figure 6: Encoder influence heatmap for Yelp(Ienc).
Figure 7: Decoder influence heatmap for Yelp(Tdec))
Although the results show similar trends, they are weaker than what we obtained for SNLI. Given the difference between SNLI and Yelp (displayed in Appendix D) there are two clear reasons for this decrease. The first is that Yelp a dataset where it is harder locate the syntactic roles. This is illustrated by the fact that the PB baseline obtains much lower score. The second is that our syntactic role extraction heuristics are tailored for regular sentences with verbal roots, which subjects the evaluation metrics on Yelp to considerable amount of noise. Nevertheless, the comparisons between a VAE, an ADVAE, and PB retain the same conclusions, but with lower margins and some overlapping standard deviations.
Through manual inspection of examples, we observed that the various structural characteristics (enumerations, sentences with nominal roots, presence of coordinating conjunctions, etc) were captured by different variables. This indicates that future iterations of our model need to provide ways to separate structural information from content-related information.
C MEASURING THE EFFECT OF LATENT VARIABLES THE STRUCTURE OF SENTENCES
In Figure & for each latent variable and each syntactic role, we report the probability that resampling the latent variable causes the appearanceddisappaaance of the syntactic role. The instance we use here is the same as the one we use for the heatmaps in the main body of the paper. According to the heatmaps in Figures 3 and latent variable 3 is the one associated with the verb. As can be seen in the present heatmap in Figure g this same variable is the one that has the most influence on the appearance/disappeant to be consistent across our different runs. This constitutes empirical justification for our choice of discarding these cases from our decoder influence metrics.
of direct and prepositional objects, and this is a pattern that proved
16

---

Under review as a conference paper at ICLR 2022
0.022
0.032
0.080
0.13
0.8
02
0.1
-0.
0.1
0.43
0.36
-00
dobi
Figure 8: The influence of latent variables on the appearance or disappearance syntactic roles.
D EXAMPLE SENTENCES FROM YELP AND SNLI AND THEIR CORRESPONDING SYNTACTIC EXTRACTIONS
Table shows some samples from SNLI and Yelp reviews. Samples from Yelp Reviews exhibit clearly higher structural diversity. On the other hand, most SNLI samples highly similar in structure.
Our syntactic role extraction heuristies tailored for sentences with verbal roots. As result, it can be seen that they struggle with sentences with nominal roots as well as other forms of irregular utterances present in Yelp. For SNLI, extractions mostly yield the expected results, allowing for reliable global assessment of our models.
Table 6: Example syntactic role extractions from both SNLI and Yelp
Source
Sentence
subj it told
verb
dobj
pobj
Yclp
was originally told would take _umm_ mins slow, over priced, i 'lll go else- where next time
num
mins
Yelp
Yelp
we will back terrible..
we
Yelp
Yelp
this point they were open and would another hour.
they
this point
SNLI
people are outside playing base- ball
people
baseball
SNLI
dogs pull opposite ends rope
dogs
pull
opposite ends a rope
rope
SNLI
lady lays beach people are running through the streets while people watch.
lady
lays
heach
SNLI
people
running
streets
SNLI
someone prepares food into
someone
prepares
food
bowls
bowls
E TRAINING DETAILS AND
SETTINGS
Our ADVAE's hyper- parameess Our model has been set to be large enough to reach a low re- construction error during the initial reconstruction phase of the training. We use 2-layer Transformers with 4 attention heads and a hidden size 192. Contrary to Vanilla VAEs, our model seems to perform better with high values of Nz. Therefore, set our latent vector to size of 768, and divide it into 96-dimensional variables for our N₂ = 8 model and to 192-dimensional latent variables for Nz 4 model. No automated hyper-paramettr selection has been done afterward.
17

---

Under review as a conference paper at ICLR 2022
Sequence VAE hyper-parameters As is usually done for this baseline (Xu et al. 2020 we set both the encoder and the decoder to be 2-layer LSTMs.
We run this model for hidden LSTM sizes in [256. 512], and latent vector sizes in [16, 32). The results for the model scoring the highest dee are then reported. Even though selection has been done according dec: checked the remaining instances of our baselines and they also yielded low Nra values.
Transformer VAE hyper-parameters We set the hidden sizes and number of layers for this baseline similarly to ADVAE, since it is also Transformer. We run this model for latent vector sizes in [16, 32] and display the highest scoring model, as is done for the Sequence VAE.
Training phases All our models are trained using Kingmaa & Ba 2013) with a batch size of 128 and learning rate of 2e-4 for 20 epochs. The dropout is set to 0.3. To avoid posterior collapse, we train all our models for 3000 steps with 3 = (reconstruction phase), then we linearly increase to its final value for the subsequent 3000 steps. Following Bowmman et al. (2016), we also use word-dropout. We set its probability to O.1.
Evaluation
For the evaluation, Tdee is set to 2000, and Tene is equal to the size of the test set.
F DISENTANGLEMENT SCORES FOR EACH SYNTACTIC ROLE
The full disentanglement scores are reported in Table
7 for the decoder, and in Table for the encoder.
Table 7: Complete decoder disentanglement scores for SNLI
Model
8
Decc
Nyace
AT dec, dec, verb
Ai dec
Al' dec
Ar dec.
0.3
0.68().22)
2.80(0.45)
(.19(0.04)
0.35(0.188 18)
0.06(0.03)
0.07(0.03)
ours-4
0.4
(0.05)
3.00(0.00)
(..14)
0.47(0.03)
0.06(0.02)
0.07(0.02)
0.3
0.60(0.10)
3.00(0.00)
0.17(0.04)
0.31(0.08)
0.05().04)
(.07()
ours-8
0.4
0.63(0.35)
2.80(0.45)
0.17(0.10)
0.32(0.18)
0.05(0.04)
0.08(0.05)
0.3
0.60(0.09)
2.40(0.55)
0.24(0.06)
0.03(0.04)
0.03(0.02)
0.31(0.03)
Sequence VAE
0.4
1.28(0.24)
1.40(0.55)
0.45(0.12)
(.23(0.02)
0.02(0.02)
0.57(0.11)
0.3
0.12(0.10)
3,00(0.70)
(.01(0.01)
0.07(0.06)
(..1()
0.03(0.03)
Transformer VAE
0.4
0.11(0.04)
3,20(0.44)
(.03))
0.04(0.04)
(.01().01)
0.02(0.01)
Table 8: Complete encoder disentanglement scores for SNLI
Model
B
Benc
Nyow
AT enc.cerb
enc.sub;
AI dobi
encc.poj
0.3
1.30(0.09)
3.00(0.00)
0.28(0.05)
0.65(0.02)
0.08(0.03)
(.29(0.03)
ours-4
0.4
1.46(0.33)
3.00(0.00)
(.38(..12)
0.64(0.10)
0.14(0.04)
0.30 10)
0.3
1.36(0.13)
3,40(0.89)
0.44(0.12)
0.60(0.18)
0.21(0.08)
0.11(0.06)
ours-8
0.4
(.44()
3.40(0.55)
0.42(0.23)
(..34)
0.17(0.10)
0.23(0.16)
Average Position
0.98 (-)
3.000))
0.12()
0.700-)
0.12G)
0.04C)
G DISENTANGLEMENT HEATMAPS OVER THE ENTIRE RANGE OF SYNTACTIC ROLES AND POS TAGS
We report decoder and encoder heatmaps for all the syntactic roles following the Stanford Depen- dencies (SD; De Marneffe & Manning 2008) annotation scheme of Ontonotes, which was used to train Spacy2 parser, in Figures 9 and TO For the sake of extensiveness and to make did not draw results from some parser biases, we also report the same heatmaps but using UDPipe 2.0 (Straka 2018), which uses UD type annotation in Figures and 14 Finally, we also report
A widely adopted annotation scheme derived from Stanford Dependencies.
18

---

Under review as a conference paper at ICLR 2022
MOW COMERIOS
0012
CAR
KOW
0:11
2a1e arD vas SETIOOE
DAN SUY 2017
0200
DATI CON 2002
251 CaT CCOS
232 613
011
Figure 9: Decoder influence heatmap for all SD syntactic roles.
690 KM
015
194
KM
K00
0P4
K01
0000
Figure 10: Encoder influence heatmap for all SD syntactic roles.
8.77
oce
0.06
<da
AGP
A6x
ooóu
DCT
whz
Moun
min
mode
PUlcT
S00tL
viloo
Figure 11: Decoder influence heatmap for all PoS Tags.
2.50
E000
11.20
2802
1270
2014
2.33
027
2039
10th
0800
xbe
cobu
MAY
môw
miwe
60/
Figure 12: Encoder influence heatmap for all PoS Tags.
19

---

Under review as a conference paper at ICLR 2022
COMN
XXX
2019 00.
0279
600
1089
0251 C28
COR
Figure 13: Decoder influence heatmap for all UD syntactic Roles.
1-00
1438 CATI SSOT
CAN 560
33X COET
C30M
COR 634
CC.8 6414 080
9:31
KC5 000
X11 687 XTV 671
KM
LOX
APN
527
8006
K7/ 020
Figure 14: Encoder influence heatmap for all UD syntactic Roles.
heatmaps for interaction with PoS Tags extracted with Spacy2 in Figures I and 12
As was done in the main body of the paper, the span corresponding to each syntactic role (in both annotation schemes) was taken to be the series words included its corresponding subtree. In contrast, the span corresponding each PoS tag was just taken be the tagged word. Results from UD parsing extraction lead to the same conclusions as from our initial SD results.
The instance of ADVAE for which display the above heatmaps is the same for which display the heatmaps in Bigures[aadd] in the main body of the paper. As shown in those Figures, mostly uses variable for verbs, variables for subjects, and variable for objects. The remaining variable (0) also seems to capture some interaction with objects. The heatmaps show that our ADVAE tends group syntactic roles into latent variables in a way that aligns with the predicative structure of sentences. In fact, variable 2 displays the highest influence on the PoS tag VERB as well as its surroundings as predicate argument such as adverbs and adverbial phrases. Similarly, latent variable 2 displays a high influence on subjects (nominal or clausal), numeral modifiers, adjectival modifiers, and auxiliaries (for conjugation). Moreover, Variable highly influences the direct and prepositional objects, which study in the main body of the paper, but also diverse clausal modifiers and obliques which often play similar roles to direct and prepositional objects in a predicate structure.
H ADDITIONAL EXAMPLES OF RESAMPLED REALIZATIONS FOR EACH SYNTACTIC ROLE
Table contains wide array of examples where the latent variable corresponding to each syntactic role is resampled.
20

---

Under review as a conference paper at ICLR 2022
Table 9: More examples where we resample specific latent variable for a sentence.
Original sentence
Resampled subject
Resampled verb
Resampled dobj/pobj
the woman riding
riding in large city
the woman is wet the police are at an arid
the woman riding bus
large brown dog
the police are running in
a man is looking at a date
the police are running in
strategy
wooded area
a man is holding a ball
man holding a ball
man and woman
man sitting on cell- phone outside
are talking on a road everyone is a man
everyone is watching the game
some individuals are watching tv
everyone is watching the game in the air man in a red shirt
is man the air group of friends are standing beach
man is sitting the air
there is no women wear- ing swim trunks group of people are standing beach
an elderly father and child standing the beach
group of friends are looking the beach
the women are in store a man is playing a game little girl is playing with a
man is playing a game
two women are on a break
two women are sitting on bench
man clean
man is sitting on lake to an old country man is playing a game
man is playing a game some dogs are playing in the pool
man is preparing to chase himself
the memorial woman happy
a dog happy
memorial happy memorial workers are in a room man is dancing for the camera
man is wearing green boy sitting in green device
the man is wearing hat a man is playing with a guitar
jacket and ship a man is playing a game
a man is playing a game
two men are tripod
is wearing brown sweater green shirt
karate dog is swim- ming a chair
is bought brown cat an airplane
is wearing dress talks to woman
the woman is about to visitors
three people are working babies
the woman is wearing a sewer a man wearing clee- tronics
the woman is about to sell
a man sitting in the snowy field two people are playing the snow a man is standing next to another man is his bike a man is sitting in front of tree taking a pic- ture
a man sitting in the smowy field the motorcycle woman on the floor a bay is standing next to another man man is his bike a man is sitting in front of a tree
a man is sitting on park bench
two people play soccer the snow a man is standing
two people are playing concert
a man is standing next to man is his bike a man is sitting on a sub- urban own
dog showing water a man is holding a red shirt and climbing a tree
man sitting with a dog
children are sitting with the dog
man playing with a dog
man sitting with an umbrella
the man is holding a ball
boy is playing with a ball
the man is on bike
the man is waiting for a counts jump for the first base
a man is holding a game
five people buying a skateboard from easter the kids play the snow game
a man is on a bicycle
a man is very large
men playing
men are playing
men are playing field
field
a man is wearing hat
a woman is wearing a hat
the man is they oil
a man is wearing a black
bathing suit near build- ings
the man wearing a blue shirt a man is on the truck the people are on the truck
a man is playing guitar
a man is sitting on a bench
a man wearing a hat playing guitar
woman playing
man is playing game
woman playing game
playing game
game
a man is holding a truck
a man is on the grass
man playing with the cut
small boy is playing on the cut
man is warming up the cut
man playing a game
a group of people are at park
the man is wearing a blue shirt
group of people are at park
group of people are at park
21

---

Under review as a conference paper at ICLR 2022
I RECONSTRUCTION AND KULLBACK-LEIBLER VALUES ACROSS EXPERIMENTS
Table 10: Reconstruction loss and Kullback-Leibler values.
Model
-[il-gsella [log pota
Perpiexity Upper Bound
0.3
31.3)(0.12)
2.8X0.25)
22.02(0..3))
Sequence VAE
19(0..3)
1.22(0.04)
21.08(0.22)
0.3
24.35(0.14)
13.38(0.19)
25.07(0.27)
Transformer VAF
0.4
20.57(0.27)
8.36(0.32)
20.68(0.16)
0.3
(0.75(0.94))
42.63((.16)
68,49(5.96)
ours-4
(6.01(0.64)
27.93(1.52)
36.16(2.20)
0.3
8.83(1.66)
46.99(2.99)
77.26(9002
ours-8
0.4
(6.84(8.50)
27.34(14.99)
39.23(11.27
The values for the reconstruction loss, the DKL, divergence, and the upper bound on perplexity concerning the experiments in the main body of the paper are reported in Table TO Since our models are VAE-based, one can only obtain the upper bound on the perplexity and not its exact value. These upper bound values are obtained using an importance sampling- estimate of the negative log-likelihood, was done in Wu et al. (2020). We set the number of importance samples to 10.
can seen that the behavior of ADVAES is very different from classical Sequence VAEs and Transformer VAEs. On the plus side, they are capable of sustaining much more information in their latent variables as shown by their higher DKL, and they do better at reconstruction. The upper bound estimate of their perplexity is however higher. A high DKL makes more difficult for the importance sampling-bais based perplexity estimate to reach the true value of the model's perplexity. This may be the reason behind the higher values observed for ADVAEs.
J LAYER-WILD ENCODER ATTENTION
In the main body of the paper, attention values that averaged throughout the network. We hereby display the encoder heatmaps obtained by using attention values from the first layer (Fig. [5) the second layer (Fig. 16), average both layers (Fig. 17 for comparison.
0.M
031
617
60m
0:15
0:53
0.2
6:3
0.4
6.5
651
034
3
Figure 15: Encoder influence heatmap (Tenc) when only using the first layer.
Figure 16: Encoder influence Figure 17: Encoder influence heatmap (Tenc) when only using the second layer.
heatmap (Tenc) when averaging over both layers.
As can seen, first layer alone provides the most sparse heatmap, and thus, the clearest correspondence between syntactic roles and latent variables. This corroborates the claims of Tenney ct al. (2020) about syntax being most prominently processed in the early layers of Transformers.
K ADVAE RESULTS FOR A LARGER GRID OF N VALUES
We display in Table 11 the quantitative results of ADVAE on SNLI for N₂ in 4, 8). For ours-2, it is normal that it only separates syntactic role realizations into maximum of 2 latent variables, as seen from the values of Nre.e and Nrous since is its total number latent variables.
As observed in the main body of the paper, the increase of the number of latent variables used in ADVAE leads to dispatching the influence on the realization of a single syntactic role to multiple latent variables. This in lead to for Dene and D that observe. In Figures T8 and 19
22

---

Under review as a conference paper at ICLR 2022
respectively display the encoder and decoder heatmaps of ADVAE with 16 latent variables. As be seen in these figures, each latent variable still highly specializes a specific syntactic role. This is seen more clearly on the encoder heatmap due to co-adaptation harming the clarity of the decoder heatmap. This specialization seems to be shared among groups (e.g. variables and 8 specialize in the subject, indicated by the green squares the figure). This causes the difference of influence between the most influential variable and the second most influential one to be low, and thus decreases the values of ene and dec.
Table 11: Disentanglement quantitative results on SNLI for larger grid of N₂ values.
Model
3
D
Npvav
Dec
N pdec
0.3
(.007)
2.00(0.00)
0.92(0.00)
2.00(0.00)
ours-2
0.4
0.33(0.15)
1.60(0.55)
0.13(0.09)
1.20(0.45)
0.3
1.30(0.09)
3.00(0.00)
0.68(..22)
2.80(0.45)
ours-4
0.4
1.46(0.33)
3.00(0.00)
0.81(0.05)
3.00(0.00)
0.3
1.36(0.13)
3,40(0.89)
0.60(0.10)
3.00(0.00)
ours-8
0.4
1.44(0.79)
3.40(0.55)
0.63(..35)
2.80(0.45)
0.3
0.60(0.31)
3.60(0.55)
0.33(0.30)
2.60(
ours-16
0.4
0.65(0.16)
3.40(0.55)
0.56(0.28)
2. 2.60().55)
0,0034
2800
0,00032
0 1.99
0.0098
0.011
0.0046
0.0052
0.015
0.01
0.01
0.01
0.015
0.025
0.035
0.7
0.0031
0,0051
82000
0.011
0.014
0.8
0.005
0.0051
0.88
0000
0.0014
0,0012
0.73
0.18
0.09
0.29
0.25
0.08
0.081
0.13
0:29
0.48
0.16
0.58
0,015
0.011
0.041
0.038
0.11
0.27
0.4
0.98
0.0026
0,00047
0.00031
0.41
0.084
0.034
0.21
0.0031
0
0.019
0.014
0.4
0.015
0.015
0.0051
0.035
0.3
10
0.0044
0.0013
0,088
0.11
0.01
0.01
0.015
0.045
0.2
0.055
0.1
0.0047
0.0031
13
0.01
0.01
0.01
0.02
13
0.0052
14
0.0026
0.008
0.01
0.015
0.015
0
0.035
15
0.0031
0.02
0.015
0.01
0.005
0.035
subj
verb
dobj
pobj
subj
verb
dobj
pobj
Figure 18: Encoder influence heatmap for ADVAE with latent variables on SNLI (Tene).
Figure 19: Decoder influence heatmap for ADVAE with 16 latent variables on SNLI (Teec). Squares with similar colors highlight groups of latent variables that relate to the same syntactic role.
Squares with similar colors highlight groups of latent variables that relate to the same syntactic role.
23


Reviews:
[{"review_id": "yqQ4iEiV2lk", "reviewer": "Reviewer_WuPD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The insight that cross-attention disentangles content automatically in Transformers is interesting, and I feel that this insight could be exploited for a variety of controllable text generation tasks. \n\nCons:\n- The baseline VAE is not a proper comparison, given that an LSTM is used (versus a Transformer in the proposed model). Please either make the proposed model LSTM-based or the baseline a Transformer.\n- I feel that the ADVAE, while explained relatively clearly in the figure, could benefit from a step-by-step explanation in the text. I found it slightly confusing to follow section 3, which could benefit from a re-organization: for example, 1st introduce the operations happening in Fig 2b, describe the latent variable generation procedure, and then what happens in Fig 2c (reconstruction). An example of a paper that I feel does this well is Zhang et al., 2016.\n- I also feel that the impact of increasing/decreasing N_z is not discussed enough (e.g. if you increase the N_z to become the maximum sequence length, what happens, etc…).\n- Overall, I feel it would be pertinent to test how well different aspects (not just syntactic roles) are disentangled. Rather than the method per se, I feel that this insight into unsupervised disentanglement is interesting, however, constraining the method to syntactic roles can be limiting. For example, for the “low complexity” datasets (e.g. sentiment analysis), it would be interesting to see when running your method, is there a vector that emerges that can control sentiment, or is it able to both disentangle syntactic roles and sentiment, etc….? (As far as I know, John et al. 2019, did this in a supervised manner).\n\nMinor comments/missing references:\n- Machine Translation/MT Transformer -> Transformer encoder-decoder models (as this architecture is widely used, even in non-MT tasks)\n- Sentence representations have been extracted from sequence-to-sequence Transformer models before (Lewis et al., 2020, Raffel et al., 2020, Siddhant et al., 2020)\n- The [CLS] representation is generally the default sentence representation in BERT (not [SEP])\n- p_r (the dependency parser?) is not defined.\n\nQuestions/Other comments:\n- It would be interesting to compare the variational model to a denoising AE (seq2seq masked language model; Lewis et al., 2020), in which the input is corrupted and the prior distribution is removed). This could make the method more generally applicable, even to large pre-trained models.\n- Furthermore, I am sceptical of the use of self-attention when computing the mean and standard deviation vectors (Fig 1b). I would be curious what would happen if only cross-attention would be used. If that is the case, then the argument could be made that the original e^{enc} vectors correspond to a syntactic role (or other factors) themselves.\n- Also, it would be curious if this emerges in machine translation (given the motivation stated as such), where the source and target represent the same sentence in different languages.\n---\n\nLewis, Mike et al. “BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020): n. pag. Crossref. Web.\n\nSiddhant, Aditya et al. “Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation.” Proceedings of the AAAI Conference on Artificial Intelligence 34.05 (2020): 8854–8861. Crossref. Web.\n\nJohn, Vineet et al. “Disentangled Representation Learning for Non-Parallel Text Style Transfer.” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019): n. pag. Crossref. Web.\n\nRaffel, Colin et al. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.” ArXiv abs/1910.10683 (2020): n. pag.\n\nZhang, Biao et al. “Variational Neural Machine Translation.” Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)", "summary_of_the_paper": "This paper proposes a new model ADVAE, which uses a sequence of latent variables which are constructed using cross-attention, which are then used to condition the inference model. The findings show that certain latent variables correlated with different syntactic roles (measured using a dependency parser).  The work claims that the latent variables, in this case, are able to disentangle content effectively, which I am inclined to agree with, however, I feel that the experimental setup is lacking to solidly support this hypothesis (which I detail in the “cons” and “questions” section). ", "summary_of_the_review": "Overall, although I like intuition and motivation, however, I feel the empirical support is lacking. I would be willing to raise my score if experiments on other aspects or in other languages were done. I believe this would further reinforce the claims made in the paper regarding unsupervised disentanglement.\n\n---\nPost rebuttal:\n\nI appreciate the additional experiments the authors provided and their time and effort in writing the rebuttal. This has clarified some of my concerns and added extra evidence to the paper. For this reason, I will raise my score to a 5. Despite this, I still feel that a more general approach, able to disentangle different factors, or in different languages would add more substance that I feel this paper is lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qEwAVPc4j7", "reviewer": "Reviewer_7uFL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths (Reasons to accept)\n- The paper is generally well-formed.\n- It proposes a new protocol that can be utilized for estimating the extent to which each latent variable is mapped to a specific syntactic role.\n\n## Weaknesses (Reasons to reject)\n\n- I'm not sure this work is the first that attempts to combine Transformers into the VAE framework. For instance, Li et al. (https://arxiv.org/pdf/2004.04092.pdf) already proposed a model consisting of pre-trained Transformer encoders and decoders.\nConsidering that the main novelty of this paper comes from the fact that the authors suggest exchanging the previous RNN to Transformer, it is doubtful this work has a meaningful novelty.\n- It is encouraged for the paper to self-contain the specification of the exact Transformer architecture (one with co-attention, Lu et al (2019)) utilized in this work.\n- There is no (theoretical) guarantee that the proposed method is specialized for assigning specific syntactic roles to latent variables, except that the method (by relying on the existing $\\beta$-VAE) encourages the latent variables to \"generally\" represent different aspects of sentences. Please explain why the proposed framework should be decent for disentangling \"syntactic roles\" rather than other linguistic or semantic properties.\n\n", "summary_of_the_paper": "## Summary\n\n- This paper proposes a probabilistic model called Attention-Driven Variational Autoencoder (ADVAE). This model is another instance of $\\beta$-VAE whose encoder and encoders are composed of Transformers rather than previous neural architectures such as RNN.\n- The authors aim to disentangle the semantics of latent variables according to some syntactic roles (e.g., nouns and verbs) defined by syntax. To achieve this goal, they suggest employing the combination of Transformer and the existing $\\beta$-VAE framework which is known to be effective for disentangling the role of each latent variable in VAE.\n- Moreover, this work presents a new way of quantifying syntactic disentanglement between latent variables, relying on the information obtained from the attention matrices of the Transformer architecture.\n- The experiments show that the proposed method is quantitatively better than the normal VAE, and that swapping the value of a specific latent variable can impact the generation of the target word (decided by the syntactic role of the latent variable we choose).", "summary_of_the_review": "To summarize, even though this paper demonstrates that Transformers with $\\beta$-VAE can be an attractive option for generating latent variables representing some syntactic roles, its novelty is generally limited to the fact that it proposed a new evaluation protocol aimed at estimating the extent to which each latent variable is mapped to a specific syntactic role.\nTherefore, my suggestion is a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l0u_Koq6he", "reviewer": "Reviewer_wDkZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Learning the syntactic disentanglement in an unsupervised way.\n- Providing a series of evaluation protocol aimed at measuring the disentanglement of syntactic roles, which can be used by other work focus on syntactic disentanglement for sentence representations.\n\nWeakness:\n- The paper is hard to follow.\n- The evaluation protocol only contains intrinsic evaluation, but how these learned latent variables can help the downstream tasks is not clear. For example, whether this can help unsupervised dependency parsing?\n\nOverall, I think the paper provides an interesting perspective for learning syntactic disentanglement. However, I really suggest the authors to reorganize the paper. I cannot understand the main problem settings until I read the paper twice. To help the reader get the background knowledge, the authors can provide some related work in learning disentangled representations at the very beginning of the introduction (basically just remove some part in Section 6). Another issue is Figure 2, the caption does not indicate what is the input and output of the framework, which is illustrated in the following section instead, and the reader need to refer to the next part of the paper to understand the figure. There are some essential information in the appendix that really need to be moved in to the main body, e.g., some analysis of the latent variables and syntactic roles.\n\nSome questions:\n- How to determine the value of N_z? Is this related to the number of syntactic roles?\n- The syntactic roles used in evaluation protocol are from a dependency parsing instead of gold standard one.  Have you considered testing on texts that have annotated dependency structures?\n- What does the predicative structure in Figure 1 do with the syntactic roles discussed in the paper?", "summary_of_the_paper": "This paper propose a framework to obtain the disentanglement of syntactic roles as latent variables for sentence representations. The model is an attention-driven VAE which maps syntactic roles to separate latent variables using an encoder-decoder framework. In the second part of the paper, the authors introduce an evaluation protocol to quantify disentanglement between latent variables and spans both in the encoder and in the decoder, which includes syntactic role extraction, latent variable influence on decoder, encoder influence on latent variables and disentanglement metrics.", "summary_of_the_review": "As a final comment, this work does some contribution for learning and evaluating syntactic disentanglement for sentence representations, and will be helpful to the community. However, the writing of the paper should be improved. For detailed comments please refer to the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0tiHMEr6SmX", "reviewer": "Reviewer_9pDc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The paper studies a research question that is at the heart of representation learning, which could potentially be very impactful\n* The model is based on an intuitive and simple idea\n* The results are convincing and well presented\n\nWeaknesses:\n\n* No ablation studies. In my opinion, the results are not actually conclusive about what model components are responsible for the improvements. For example, the VAE baseline is based on LSTM encoder-decoder models, whereas the proposed model employs Transformers. Moreover, the VAE baseline seems to be significantly smaller than the Tranformer. Are these two models really comparable enough to conclude that the improvements are due to the product of Gaussians formulation? I think a fairer comparison with minimal changes between the baseline and proposed model could be made as follows: Pool the outputs of 2b) before estimating a single Gaussian distribution, which can then be used as a standard VAE.\n\n* The concrete architecture is not well motivated: It is not clear to me why the inference network architecture has to look the way it does in 2b). Does the model actually use co-attention (i.e., sentence representations and latent variables serve as queries for each other)? Figure 2b) looks like a vanilla Transformer encoder-decoder architecture to me, which uses cross-attention, not co-attention. Moreover, it is not clear that the model actually needs self-attention to refine the latent variables. Wouldn't the model still work if the learnable $e_{z_i}^{enc}$ functioned merely as queries to a single additional attention function on top of the Transformer encoder (followed by a linear layer and softplus)?\n\n* Some related work is not discussed. Specifically, Behjati and Henderson (2021) propose a very similar architecture for learning meaningful units in text: They train an autoencoder with a fixed number of latent vectors via slot attention, which are supposed to capture morphemes. Could their model be applied to disentangle semantic roles, too? In general, how does your work compare to slot attention (Locatello et al. (2020))\n\nMinor:\n* Table captions should appear above the table according to the style guide\n\nLocatello et al. (2020): https://arxiv.org/abs/2006.15055\n\nBehjati and Henderson (2021): https://arxiv.org/pdf/2102.01223.pdf", "summary_of_the_paper": "The paper proposes a method for unsupervised disentanglement of text components and shows its ability to identify semantic roles.\nTo this end, a neural network is trained to compress the input into a fixed number of independent latent variables which are regularized to be standard Gaussians via the VAE framework. The inference network consists of a Transformer encoder-decoder network, where the decoder inputs correspond to the latent variables, which cross-attend to the outputs of the Transformer encoder, i.e., the encoded sentence. The idea behind this architecture is that attention-based seq2seq architectures align source and target sequences with each other.\n\nThe model is evaluated on disentanglement of semantic roles. To this end, they investigate how resampling of individual latent variables impacts the semantic roles in the text generated by the decoder, and how syntactic roles are aggregated into latent vectors via attention. They find that their proposed architecture is more successful at disentangling semantic roles into the latent variables than standard VAEs.", "summary_of_the_review": "While the research presented in this study is very exciting, it is not quite convincing enough yet for me to trust that the results are actually due to the modeling decisions presented. Moreover, a similar work (Behjati and Henderson (2020)) is not discussed, which casts doubt on the novelty of the approach. I therefore recommend to reject the paper in its current state, but I am happy to change my scores upwards if my concerns above can be resolved.\n\n* UPDATE:  I raised my score slightly as a consequence of the rebuttal phase, where authors conducted an ablation study that now more clearly shows that the improvements over standard VAEs are due to multi-vector nature of the representation rather than just because of using Transformers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes improving human interpretability and manipulability of neural representations by obtaining syntactic roles (here, subject, object, prepositional object, and main verb) without supervision by means of them becoming linked to latent variables in a novel proposed attention-driven VAE (ADVAE) model, which provides cross attention between a language transformer and latent variables. The paper argues that syntactic roles are quite central to meaning interpretation and that the ADVAE recovers them better than LSTM or Transformer (with mean pooling) VAEs.

This is a quite interesting direction and paper. There was active discussion with the reviewers, one of whom (9pDc) moved their rating from reject to quite strong support, while the other reviewers either sat on the fence or raised from reject to borderline. Nevertheless, I overall tend to agree that the paper is still lacking in empirical support, a view clearly shared by reviewers WuPD and 7uFL. The SNLI data is very simple descriptive sentences, nearly all in the form of S V O or S V PP. Would this work on more complex data, in other languages, or with more word order variation? There isn't very much investigation, but the new results added during reviewing based on Yelp data seem to offer more concerns than confidence. These are also very short sentences but with more varied structure and some complementation. It seems like D_{dec} is now very low (much lower than for the sequence VAE), the ability to distinguish out grammatical roles seems limited to {subj} vs. {dobj, pobj} in the encoder and none at all in the decoder (Figure 6/7). And then for the examples in Appendix D, the disentanglement abilities barely seem stronger than being able to pick out subjects, though when there are sentences with subordinate clauses, it is perhaps random which subject you get. The resampled realizations in appendix H also seem to show limited disentanglement: resampling the subject usually seems to change the object as well, often markedly. No convincing downstream applications are shown. As such, while I agree that disentanglement is at the heart of representation learning, I can't get on board with reviewer 9pDc feeling that this paper now has convincing results. Reviewer 7uFL also emphasizes that there is no strong reason that the latent variables have to align with syntactic roles. In particular, the motivation in NMT whereby constituents clump and reorder together does not exist here. It may only work for the very simple and regular sentences of SNLI.

Hence, overall, I feel that this method needs more extensive validation on harder, more varied data sets before it becomes a convincing contribution, and so I propose rejecting the paper at this point in time. Nevertheless, I do think the topic is interesting and this approach has the potential to be good.
Decision: Reject