Unsupervised Distillation of Syntactic Information from Contextualized Word Representations

Contextualized word representations , such as ELMo and BERT , were shown to perform well on various semantic and structural ( syntactic ) task . In this work , we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations : we aim to learn a transformation of the contextualized vectors , that discards the lexical semantics , but keeps the structural information . To this end , we automatically generate groups of sentences which are structurally similar but semantically different , and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors . We demonstrate that our transformation clusters vectors in space by structural properties , rather than by lexical semantics . Finally , we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in a few-shot parsing setting . 1 INTRODUCTION . Human language1 is a complex system , involving an intricate interplay between meaning ( semantics ) and structural rules between words and phrases ( syntax ) . Self-supervised neural sequence models for text trained with a language modeling objective , such as ELMo ( Peters et al. , 2018 ) , BERT ( Devlin et al. , 2019 ) , and RoBERTA ( Liu et al. , 2019b ) , were shown to produce representations that excel in recovering both structure-related information ( Gulordava et al. , 2018 ; van Schijndel & Linzen ; Wilcox et al. , 2018 ; Goldberg , 2019 ) as well as in semantic information ( Yang et al. , 2019 ; Joshi et al. , 2019 ) . In this work , we study the problem of disentangling structure from semantics in neural language representations : we aim to extract representations that capture the structural function of words and sentences , but which are not sensitive to their content . For example , consider the sentences : 1 . Neural networks are interesting . 3 . I study neural networks . 2 . Maple syrup is delicious . 4 . John loves maple syrup . While ( 1 ) and ( 2 ) are different in content , they share a similar structure , the corresponding words in them , while unrelated in meaning,2 serve the same function . Similarly for sentences ( 3 ) and ( 4 ) . In contrast , sentence ( 1 ) shares the phrase neural networks with sentence ( 3 ) and maple syrup is shared between ( 2 ) and ( 4 ) .3 While the two occurrences of each phrase share the meaning , they are used in different structural ( syntactic ) configurations , serving different roles within the sentence ( appearing in subject vs object position ) .4 We seek a representation that will expose the similarity between “ networks ” in ( 1 ) and “ syrup ” in ( 2 ) while ignoring the similarity between “ syrup ” in ( 2 ) and “ syrup ” in ( 4 ) . 1In this work we focus on English . 2We focus on lexical semantics . 3There is a syntactic distinction between the two , with “ maple ” being part of a noun compound and “ neural ” being an adjective . However , we focus in their similarity as noun modifiers in both phrases . 4These differences in syntactic position are also of relevance to language modeling , as different positions may pose different restrictions on the words that can appear in them . We aim to learn a function from contextualized word representations to a space that exposes these similarities . Crucially , we aim to do this in an unsupervised manner : we do not want to inform the process of the kind of structural information we want to obtain . We do this by learning a transformation that attempts to remove the lexical-semantic information in a sentence , while trying to preserve structural properties . Disentangling syntax from lexical semantics in word representations is a desired property for several reasons . From a purely scientific perspective , once disentanglement is achieved , one can better control for confounding factors and analyze the knowledge the model acquires , e.g . attributing the predictions of the model to one factor of variation while controlling for the other . In addition to explaining model predictions , such disentanglement can be useful for the comparison of the representations the model acquires to linguistic knowledge . From a more practical perspective , disentanglement can be a first step toward controlled generation/paraphrasing that considers only aspects of the structure , akin to the style-transfer works in computer vision , i.e. , rewriting a sentence while preserving its structural properties while ignoring its meaning , or vice-versa . It can also inform search-based application in which one can search for “ similar ” texts while controlling various aspects of the desired similarity . To achieve this goal , we begin with the intuition that the structural component in the representation ( capturing the form ) should remain the same regardless of the lexical semantics of the sentence ( the meaning ) . Rather than beginning with a parsed corpus , we automatically generate a large number of structurally-similar sentences , without presupposing their formal structure ( §3.1 ) . This allows us to pose the disentanglement problem as a metric-learning problem : we aim to learn a transformation of the contextualized representation , which is invariant to changes in the lexical semantics within each group of structurally-similar sentences ( §3.3 ) . We demonstrate the structural properties captured by the resulting representations in several experiments ( §4 ) , among them automatic identification of structurally-similar words and few-shot parsing . 2 RELATED WORK . The problem of disentangling different sources of variation has long been studied in computer vision , and was recently applied to neural models ( Bengio et al. , 2013 ; Mathieu et al. , 2016 ; Hadad et al. , 2018 ) . Such disentanglement can assist in learning representations that are invariant to specific factors , such as pose-invariant face-recognition ( Peng et al. , 2017 ) or style-invariant digit recognition ( Narayanaswamy et al. , 2017 ) . From a generative point of view , disentanglement can be used to modify one aspect of the input ( e.g. , “ style ” ) , while keeping the other factors ( e.g. , “ content ” ) intact , as done in neural image style-transfer ( Gatys , 2017 ) . In the field of NLP , disentanglement is much less researched . In controlled natural language generation and style transfer , several works attempted to disentangle factors of variation such as sentiment or age of the writer , with the intention to control for those factors and generate new sentences with specific properties , or transfer existing sentences to similar sentences that differ only in the those properties . Several works ( Sohn et al. , 2015 ; Ficler & Goldberg , 2017 ) have trained conditional generative models by explicitly conditioning a decoder network with a vector of attributes . On training , the attributes derive from the training sentence , while in testing the conditioning vector can be set to generate a text with the desired attributes . Other works Fu et al . ( 2018 ) ; Hu et al . ( 2017 ) aim to achieve style transfer ( as opposed to generation ) by explicitly training representations that are invariant to the controlled attributes ( e.g . by co-training of a generator and attribute discriminator ) . A decoder generates the transferred text from the disentangled representation and from an explicit attribute representation . Lample et al . ( 2018 ) use a conditioned back-translation approach to achieve a similar goal . While these works try to disentangle sentence-level attributes , in this study we focus on disentangling between two components in the representations of individual words . Several works examine the way semantic and syntactic information is distributed across the layers of neural models of text ( Blevins et al. , 2018 ; Tenney et al. , 2019 ) . They use diagnostic classifiers ( Adi et al. , 2016 ; Hupkes et al. , 2018 ) to predict syntactic properties and demonstrate that different parts of the model encode information in different levels of abstraction ( e.g . POS information , dependency label and semantic role ) . Liu et al . ( 2019a ) used diagnostic classifiers trained to predict various syntactic and semantic properties from state-of-the-art LMs representations , and demonstrated that many syntactic and semantic distinctions are encoded in the probed representations . Clark et al . ( 2019 ) probed the attention patterns of BERT , and showed that individual attention-heads focus in syntactically-meaningful relations in the input . Beyond the descriptive level , recent works have focused on supervised extraction of syntax-related representations from neural representations . Artetxe et al . ( 2018 ) used a linear transformation , inspired from the notion of ” similarity order ” in classic word representation learning , to tailor uncontextualized word representations to syntactic vs. semantic tasks . Hewitt & Manning ( 2019 ) demonstrated that it is possible to train a linear transformation , under which squared euclidean distance between transformed contextualized word vectors correspond to the distances between the respective words in the syntax tree that represents the hierarchical structure of the sentence . Concurrent to this work , Li & Eisner ( 2019 ) have used a variational estimation method ( Alemi et al. , 2016 ) of the information-bottleneck principle ( Tishby et al. , 1999 ) to extract word embeddings that are useful to the end task of parsing . While impressive , those works presuppose a specific formal syntactic structure ( e.g . annotated parse trees following a specific linguistic annotation schema ) and use this syntactic signal to learn structural information in a supervised manner . In other words , these works assume a given structure , and use supervision to make the structural information more salient , mapping ( or forcing ) the neural representations to known linguistic properties . In contrast , we aim to expose the structural information encoded in the network in an unsupervised manner , without pre-supposing an existing syntactic annotation scheme . 3 METHOD . Our goal is to learn a function f : Rn 7→ Rm , which operates on contextualized word representations x and extracts vectors f ( x ) which make the structural information encoded in x more salient , while discarding as much lexical information as possible . In the sentences “ Maple syrup is delicious ” and “ Neural networks are interesting ” , we want to learn a f such that f ( v2syrup ) ≈ f ( v1networks ) , where viword is the contextualized vector representation of the word in sentence i . We also want f ( v 4 syrup ) ≈ f ( v3networks ) , while keeping f ( v 1 networks ) 6≈ f ( v3networks ) . Moreover , we would like the relation between the words “ maple ” and “ delicious ” in the second sentence , to be similar to the relation between “ neural ” and “ interesting ” in the first sentence : pair ( v2maple , v 2 delicious ) ≈ pair ( v1neural , v1interesting ) . Operativly , we represent pairs of words ( x , y ) by the difference between their transformation f ( x ) − f ( y ) , and aim to learn f that preserves : f ( v2maple ) − f ( v2delicious ) ≈ f ( v1neural ) − f ( v1interesting ) . The choice to represent pairs this way was inspired by several works that demonstrated that nontrivial semantic and syntactic relations between uncontextualized word representations can be approximated by simple vector arithmetic ( Mikolov et al. , 2013a ; b ; Levy & Goldberg , 2014 ) . To learn f , we start with groups of sentences that the sentences within each group are known to share their structure but differ in their lexical semantics . We call the sentences in each group structurally equivalent . Figure 1 shows an example of two structurally equivalent sets . Acquiring such sets is challenging , especially if we do not assume a known syntactic formalism and can not mine for sentences based on their observed tree structures . To this end , we automatically generate the sets starting with known sentences and sampling variants from a language model ( §3.1 ) . Our sentenceset generation procedure ensures that words from the same set that share an index also share their structural function . We call such words corresponding . We now proceed to learn a function f to map contextualized vectors of corresponding words ( and the relations between them , as described above ) to neighbouring points in the space . We train f such that the representation assigned to positive pairs — pairs that share indices and come from the same equivalent set — is distinguished from the representations of negative pairs — challenging pairs that come from different sentences , and thus do not share the structure of the original pair , but can , potentially , share their lexical meaning . We do so using Triplet loss , which pushes the representations of pairs coming from the same group closer together ( §3.3 ) . Figure 2 sketches the network .

Summary: The authors state a clear hypothesis: it is possible to extract syntactic information from contextualized word vectors in an unsupervised manner. The method of creating syntactically equivalent (but semantically different) sentences is indeed interesting on its own. Experiments do support the main hypothesis -- the distilled embeddings are stronger in syntactic tasks than the default contextualized vectors. The authors provide the code for ease of reproducibility which is nice.