Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

1 INTRODUCTION . A disentangled representation of data describes information as a combination of separate understandable factors . This separation provides better transparency , but also better transfer performance ( Higgins et al. , 2018 ; Dittadi et al. , 2021 ) . When it comes to disentanglement , Variational Autoencoders ( VAEs ; Kingma & Welling , 2014 ) were extensively proven effective ( Higgins et al. , 2017 ; Chen et al. , 2018 ; Rolinek et al. , 2019 ) . and were used throughout several recent works ( Chen et al. , 2019 ; Li et al. , 2020b ; John et al. , 2020 ) . In NLP , disentanglement has been mostly performed to separate the semantics ( or content ) in a sentence from characteristics such as style and structure in order to generate paraphrases ( Chen et al. , 2019 ; John et al. , 2020 ; Bao et al. , 2020 ; Huang & Chang , 2021 ; Huang et al. , 2021 ) . We show in our work that the information in the content itself can be separated with a VAE-based model . In contrast to the aforementioned works , we use neither supervision nor input syntactic information for this separation . We demonstrate this ability by controlling the lexical realization of core syntactic roles . For example , the subject in a sentence can be encoded separately and controlled to generate the same sentence with another subject . Our framework includes a model and an evaluation protocol aimed at measuring the disentanglement of syntactic roles . The model we introduce is an Attention-Driven VAE ( ADVAE ) , which we train on the SNLI raw text dataset ( Schmidt et al. , 2020 ) . It draws its inspiration from attention-based machine translation models ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) . Such models translate sentences between languages with 1URL to be disclosed upon publication . Our code is provided as supplemental material during the submission process . different underlying structures and can be inspected to show a coherent alignment between spans from both languages . Our ADVAE uses Transformers ( Vaswani et al. , 2017 ) , an attention-based architecture , to map sentences from a language to independent latent variables , then map these variables back to the same sentences . Although ADVAE could be used to study other attributes , we motivate it ( §4.1 ) and therefore study it for the alignment of syntactic roles with latent variables . Evaluating disentanglement with regard to spans is challenging . After training the model and only for evaluation , we use linguistic information ( from an off-the-shelf dependency parser ) to first extract syntactic roles from sentences , and then study their relation to latent variables . To study this relation on the ADVAE decoder , we repeatedly i ) generate a sentence from a sampled latent vector ii ) perturb this latent vector at a specific location iii ) generate a sentence from this new vector and observe the difference . On the encoder side , we study the attention values to see whether each latent variable is focused on a particular syntactic role in input sentences . The latter procedure is only possible through the way our ADVAE uses attention to produce latent variables . To the best of our knowledge , we are the first to use this transparency mechanism to obtain quantitative results for a latent variable model . We first justify our focus on syntactic roles in §3 , then we go over our contribution , which is threefold : i ) We introduce the ADVAE , a model that is designed for unsupervised disentanglement of syntactic roles , and that enables analyzing the interaction between latent variables and observations through the values of attention ( §4 ) , ii ) We design an experimental protocol for the challenging assessment of disentanglement over realizations of syntactic roles , based on perturbations on the decoder side and attention on the encoder side ( §5 ) , iii ) Our empirical results show that our architecture disentangles syntactic roles better than standard sequence VAEs and Transformer VAEs and that it is capable of controlling realizations of syntactic roles separately during generation ( §6 ) . 2 RELATED WORKS . Linguistic information in neural models Accounting for linguistic information provided better inductive bias in the design of neural NLP systems during recent years . For instance , successful attempts at capturing linguistic information with neural models helped improve grammar induction ( RNNG ; Dyer et al. , 2016 ) , constituency parsing and language modeling ( ON-LSTM ; Shen et al. , 2019 , ONLSTM-SYD ; Du et al. , 2020 ) , as well as controlled generation ( SIVAE ; Zhang et al. , 2019 ) . Many ensuing works have also dived into the linguistic capabilities of the resulting models , the types of linguistic annotations that emerge best in them , and syntactic error analyses ( Hu et al. , 2020 ; Kodner & Gupta , 2020 ; Marvin & Linzen , 2020 ; Kulmizev et al. , 2020 ) . Based on the Transformer architecture , the self-supervised model BERT ( Devlin et al. , 2019 ) has also been subject to studies showing that the linguistic information it captures is organized among its layers in a way remarkably close to the way a classical NLP pipeline works ( Tenney et al. , 2020 ) . Furthermore , ( Clark et al. , 2019 ) , showed that many attention heads in BERT specialize in dependency parsing . We refer the reader to ( Rogers et al. , 2020 ) for an extensive review of Bert-related studies . However , such studies most often rely on structural probes ( Jawahar et al. , 2019 ; Liu et al. , 2019 ; Hewitt & Manning , 2019 ) to explain representations , probes which are not without issues , as shown by Pimentel et al . ( 2020 ) . In that regard , the generative capabilities and the attention mechanism of our model offer an alternative to probing : analysis is performed directly on sentences generated by the model and on internal attention values . Disentanglement in NLP The main line of work in this area revolves around using multitask learning to separate concepts in neural representations ( e.g . style vs content ( John et al. , 2020 ) , syntax vs semantics ( Chen et al. , 2019 ; Bao et al. , 2020 ) ) . Alternatively , Huang & Chang ( 2021 ) and Huang et al . ( 2021 ) use syntactic trees as inputs to separate syntax from semantics , and generate paraphrases without a paraphrase corpus . Towards less supervision , Cheng et al . ( 2020 ) only uses style information to separate style from content in representations . Literature on unsupervised disentanglement in NLP remains sparse . Examples are the work of Xu et al . ( 2020 ) on categorical labels ( sentiment and topic ) , and that of Behjati & Henderson ( 2021 ) on representing morphemes using character-level Seq2Seq models . The work of Behjati & Henderson ( 2021 ) is closest to ours as it uses Slot Attention ( Locatello et al. , 2020 ) , which , like ADVAE , is a cross-attention-based representation technique . Our contribution depart from previous work since i ) syntactic parses are not used as learning signals but as a way to interpret our model , and ii ) cross-attention enables our model to link a fixed number of latent variables to text spans . 3 SYNTACTIC ROLES AND DEPENDENCY PARSING . We present in Figure 1 an example sentence with its dependency parse2 , its Part-of-Speech ( PoS ) tags , and a flat predicative structure with PropBank-like semantic roles ( Palmer et al. , 2005 ) . Dependency parsing yields a tree , where edges are labeled with syntactic roles ( or relations or functions ) such as nominal subject ( nsubj ) . The lexical realizations of these syntactic functions are textual spans and correspond to syntactic constituents . For instance , the lexical realization of the direct object ( dobj ) of the verb holds in this sentence is the span his nice guitar , with guitar as head . In short , the spans corresponding to subtrees consist of tokens that are more dependent of each other than of the rest of the sentence . As a consequence , and because a disentanglement model seeks independent substructures in the data , we expect such a model to converge to representations that display separation in realizations of frequent syntactic roles . In our work , we focus3 within the same framework . on nominal subjects , verbal roots of sentences , and direct or prepositional objects . These are core ( as opposed to oblique ; see Nivre et al . ( 2016 ) for details on the distinction ) syntactic roles , since they directly relate to the predicative structure . In fact in most cases , as illustrated in Figure 1 , the verbal root of a sentence is its main predicate , the nominal subject its agent ( ARG0 ) and the direct or prepositional object its patient ( ARG1 ) . 4 MODEL DESCRIPTION . The usual method to obtain sentence representations from Transformer models uses only a Transformer encoder either by taking an average of the token representations or by using the representation of a special token ( e.g [ CLS ] in BERT ( Devlin et al. , 2019 ) ) . Recently , the usage of both Transformer encoders and decoders has also been explored in order to obtain representations whether by designing classical Autoencoders ( Lewis et al. , 2019 ; Siddhant et al. , 2019 ; Raffel et al. , 2020 ) , or VAEs ( Li et al. , 2020a ) . Our model , the ADVAE , differs from these models in that it uses targets in Machine Translation ( MT ) Transformers ( i.e an encoder and a decoder ) to produce sentence representations . Producing representations with Cross-Attention has been introduced by Locatello et al . ( 2020 ) as part of the Slot Attention modules in the context of unsupervised object discovery . However , in contrast to Locatello et al . ( 2020 ) , we simply use Cross-Attention as it is found in Vaswani et al . ( 2017 ) , i.e . without normalizing attention weights over the query axis , or using GRUs ( Cho et al. , 2014 ) to update representations . As will be shown through our experiments , this is sufficient to disentangle syntactic roles . We explain the observation that motivates our work in §4.1 , we then describe in §4.2 the minimal changes we apply to MT Transformers , and finally , we present the objective we use in §4.3 . The parallel between our model and MT Transformers is illustrated in Figure 2 . 4.1 THE INTUITION BEHIND OUR MODEL . Consider s = ( sj ) 1≤j≤Ns and t = ( tj ) 1≤j≤Nt , two series of tokens forming respectively a sentence in a source language and a sentence in a target language . Given s , attention-based translation models 2Following the ClearNLP constituent to dependency conversion , close to Stanford Dependencies de Marneffe & Manning ( 2008 ) . See https : //github.com/clir/clearnlp-guidelines/blob/master/ md/components/dependency_conversion.md . 3Future research that takes interest in the finer-grained disentanglement of content may simply study a larger array of syntactic roles . Using our current system we display results including all syntactic roles in Appendix G. are capable of yielding t while also providing information about the alignment between the groups of tokens ( of different sizes ) in both sentences ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) ) . This evidence suggests that attention-based architectures are capable of factoring information from groups of words according to a source structure , and redistributing it according to a target structure . The aim of our design is to use , as a target , a set of NZ independent latent variables that will act as fixed placeholders for the information in sentences . We stress that NZ is fixed and independent of the input sentence size Ns . Combining Transformers , an attention-based MT model , and the VAE framework for disentanglement , our ADVAE is intended to factor information from independent groups of words into separate latent variables . In the following sections , we will refer to this set of independent latent variables as the latent vector z = ( zi ) 1≤i≤NZ and to each zi as a latent variable .

Summary: ## Summary  - This paper proposes a probabilistic model called Attention-Driven Variational Autoencoder (ADVAE). This model is another instance of $\beta$-VAE whose encoder and encoders are composed of Transformers rather than previous neural architectures such as RNN. - The authors aim to disentangle the semantics of latent variables according to some syntactic roles (e.g., nouns and verbs) defined by syntax. To achieve this goal, they suggest employing the combination of Transformer and the existing $\beta$-VAE framework which is known to be effective for disentangling the role of each latent variable in VAE. - Moreover, this work presents a new way of quantifying syntactic disentanglement between latent variables, relying on the information obtained from the attention matrices of the Transformer architecture. - The experiments show that the proposed method is quantitatively better than the normal VAE, and that swapping the value of a specific latent variable can impact the generation of the target word (decided by the syntactic role of the latent variable we choose).