Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder

1 INTRODUCTION . A dependency tree is a lightweight syntactic structure exposing ( possibly labeled ) bi-lexical relations between words ( Tesnière , 1959 ; Kaplan & Bresnan , 1982 ) , see Figure 1 . This representation has been widely studied by the NLP community leading to very efficient state-of-the-art parsers ( Kiperwasser & Goldberg , 2016 ; Dozat & Manning , 2017 ; Ma & Hovy , 2017 ) , motivated by the fact that dependency trees are useful in downstream tasks such as semantic parsing ( Reddy et al. , 2016 ; Marcheggiani & Titov , 2017 ) , machine translation ( Ding & Palmer , 2005 ; Bastings et al. , 2017 ) , information extraction ( Culotta & Sorensen , 2004 ; Liu et al. , 2015 ) , question answering ( Cui et al. , 2005 ) and even as a filtering method for constituency parsing ( Kong et al. , 2015 ) , among others . Unfortunately , syntactic annotation is a tedious and expensive task , requiring highly-skilled human annotators . Consequently , even though syntactic annotation is now available for many languages , the datasets are often small . For example , 31 languages in the Universal Dependency Treebank,1 the largest dependency annotation resource , have fewer than 5,000 sentences , including such major languages as Vietnamese and Telugu . This makes the idea of using unlabeled texts as an additional source of supervision especially attractive . In previous work , before the rise of deep learning , the semi-supervised parsing setting has been mainly tackled with two-step algorithms . On the one hand , feature extraction methods first learn an intermediate representation using an unlabeled dataset which is then used as input to train a supervised parser ( Koo et al. , 2008 ; Yu et al. , 2008 ; Chen et al. , 2009 ; Suzuki et al. , 2011 ) . On the other hand , the self-training and co-training methods start by learning a supervised parser that is then used to label extra data . Then , the parser is retrained with this additional annotation ( Sagae & Tsujii , 2007 ; Kawahara & Uchimoto , 2008 ; McClosky et al. , 2006 ) . Nowadays , unsupervised feature extraction is achieved in neural parsers by the means of word embeddings ( Mikolov et al. , 2013 ; Peters et al. , 2018 ) . The natural question to ask is whether one can exploit unlabeled data in neural parsers beyond only inducing generalizable word representations . 1http : //universaldependencies.org/ Our method can be regarded as semi-supervised Variational Auto-Encoder ( VAE , Kingma et al. , 2014 ) . Specifically , we introduce a probabilistic model ( Section 3 ) parametrized with a neural network ( Section 4 ) . The model assumes that a sentence is generated conditioned on a latent dependency tree . Dependency parsing corresponds to approximating the posterior distribution over the latent trees within this model , achieved by the encoder component of VAE , see Figure 2a . The parameters of the generative model and the parser ( i.e . the encoder ) are estimated by maximizing the likelihood of unlabeled sentences . In order to ensure that the latent representation is consistent with treebank annotation , we combine the above objective with maximizing the likelihood of gold parse trees in the labeled data . Training a VAE via backpropagation requires marginalization over the latent variables , which is intractable for dependency trees . In this case , previous work proposed approximate training methods , mainly differentiable Monte-Carlo estimation ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) and score function estimation , e.g . REINFORCE ( Williams , 1992 ) . However , REINFORCE is known to suffer from high variance ( Mnih & Gregor , 2014 ) . Therefore , we propose an approximate differentiable Monte-Carlo approach that we call Differentiable Perturb-and-Parse ( Section 5 ) . The key idea is that we can obtain a differentiable relaxation of an approximate sample by ( 1 ) perturbing weights of candidate dependencies and ( 2 ) performing structured argmax inference with differentiable dynamic programming , relying on the perturbed scores . In this way we bring together ideas of perturb-and-map inference ( Papandreou & Yuille , 2011 ; Maddison et al. , 2017 ) and continuous relaxation for dynamic programming ( Mensch & Blondel , 2018 ) . Our model differs from previous works on latent structured models which compute marginal probabilities of individual edges Kim et al . ( 2017 ) ; Liu & Lapata ( 2018 ) . Instead , we sample a single tree from the distribution that is represented with a soft selection of arcs . Therefore , we preserve higher-order statistics , which can then inform the decoder . Computing marginals would correspond to making strong independence assumptions . We evaluate our semi-supervised parser on English , French and Swedish and show improvement over a comparable supervised baseline ( Section 6 ) . Our main contributions can be summarized as follows : ( 1 ) we introduce a variational autoencoder for semi-supervised dependency parsing ; ( 2 ) we propose the Differentiable Perturb-and-Parse method for its estimation ; ( 3 ) we demonstrate the effectiveness of the approach on three different languages . In short , we introduce a novel generative model for learning latent syntactic structures . 2 DEPENDENCY PARSING . A dependency is a bi-lexical relation between a head word ( the source ) and a modifier word ( the target ) , see Figure 1 . The set of dependencies of a sentence defines a tree-shaped structure.2 In the parsing problem , we aim to compute the dependency tree of a given sentence . Formally , we define a sentence as a sequence of tokens ( words ) from vocabulary W. We assume a one-to-one mapping between W and integers 1 . . . |W| . Therefore , we write a sentence of length n as a vector of integers s of size n + 1 with 1 ≤ si ≤ |W| and where s0 is a special root symbol . A dependency tree of sentence s is a matrix of booleans T ∈ { 0 , 1 } ( n+1 ) × ( n+1 ) with Th , m = 1 meaning that word sh is the head of word sm in the dependency tree . 2 Semantic dependencies can have a more complex structure , e.g . words with several heads . However , we focus on syntactic dependencies only . More specifically , a dependency tree T is the adjacency matrix of a directed graph with n + 1 vertices v0 . . . vn . A matrix T is a valid dependency tree if and only if this graph is a v0-rooted spanning arborescence,3 i.e . the graph is connected , each vertex has at most one incoming arc and the only vertex without incoming arc is v0 . A dependency tree is projective if and only if , for each arc vh → vm , if h < m ( resp . m < h ) then there exists a path with arcs T from vh to each vertex vk such that h < k < m ( resp . m < k < h ) . From a linguistic point of view , projective dependency trees combine contiguous phrases ( sequence of words ) only . Intuitively , this means that we can draw the dependency tree above the sentence without crossing arcs . Given a sentence s , an arc-factored dependency parser computes the dependency tree T which maximizes a weighting function f ( T ; W ) = ∑ h , m Th , mWh , m , where W is a matrix of dependency ( arc ) weights . This problem can be solved with a O ( n2 ) time complexity ( Tarjan , 1977 ; McDonald et al. , 2005 ) . If we restrict T to be a projective dependency tree , then the optimal solution can be computed with a O ( n3 ) time complexity using dynamic programming ( Eisner , 1996 ) . Restricting the search space to projective trees is appealing for treebanks exhibiting this property ( either exactly or approximately ) : they enforce a structural constraint that can be beneficial for accuracy , especially in a low-resource scenario . Moreover , using a more restricted search space of potential trees may be especially beneficial in a semi-supervised scenario : with a more restricted space a model is less likely to diverge from a treebank grammar and capture non-syntactic phenomena . Finally , Eisner ’ s algorithm ( Eisner , 1996 ) can be described as a deduction system ( Pereira & Warren , 1983 ) , a framework that unifies many parsing algorithms . As such , our methodology could be applied to other grammar formalisms . For all these reasons , in this paper , we focus on projective dependency trees only . 3 GENERATIVE MODEL . We now turn to the learning problem , i.e . estimation of the matrix W . We assume that we have access to a set of i.i.d . labeled sentences L = { 〈s , T 〉 , . . . } and a set of i.i.d . unlabeled sentences U = { s , . . . } . In order to incorporate unlabeled data in the learning process , we introduce a generative model where the dependency tree is latent ( Subsection 3.1 ) . As such , we can maximize the 3 Tree refers to the linguistic structure whereas arborescence refers to the graph structure . likelihood of observed sentences even if the ground-truth dependency tree is unknown . We learn the parameters of this model using a variational Bayes approximation ( Subsection 3.2 ) augmented with a discriminative objective on labeled data ( Subsection 3.3 ) . 3.1 GENERATIVE STORY . Under our probabilistic model , a sentence s is generated from a continuous sentence embedding z and with respect to a syntactic structure T . We formally define the generative process of a sentence of length n as : T ∼ p ( T |n ) z ∼ p ( z|n ) s ∼ p ( s|T , z , n ) This Bayesian network is shown in Figure 2a . In order to simplify notation , we omit conditioning on n in the following . T and z are latent variables and p ( s|T , z ) is the conditional likelihood of observations . We assume that the priors p ( T ) and p ( z ) are the uniform distribution over projective trees and the multivariate standard normal distribution , respectively . The true distribution underlying the observed data is unknown , so we have to learn a model pθ ( s|T , z ) parametrized by θ that best fits the given samples : θ = arg max θ ∑ s log pθ ( s ) ( 1 ) Then , the posterior distribution of latent variables pθ ( T , z|s ) models the probability of underlying representations ( including dependency trees ) with respect to a sentence . This conditional distribution can be written as : pθ ( T , z|s ) = pθ ( s|T , z ) p ( T ) p ( z ) pθ ( s ) ( 2 ) In the next subsection , we explain how these two quantities can be estimated from data . 3.2 VARIATIONAL AUTO-ENCODERS . Computations in Equation 1 and Equation 2 require marginalization over the latent variables : pθ ( s ) = ∑ T ∫ pθ ( s , T , z ) dz which is intractable in general . We rely on the Variational Auto-Encoder ( VAE ) framework to tackle this challenge ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . We introduce a variational distribution qφ ( T , z|s ) which is intended to be similar to pθ ( T , z|s ) . More formally , we want KL [ qφ ( T , z|s ) ‖pθ ( T , z|s ) ] to be as small as possible , where KL is the Kulback-Leibler ( KL ) divergence . Then , the following equality holds : log pθ ( s ) = Eqφ ( T , z|s ) [ log pθ ( s|T , z ) ] − KL [ qφ ( T , z|s ) |p ( T , z ) ] + KL [ qφ ( T , z|s ) ‖pθ ( T , z|s ) ] where log pθ ( s ) is called the evidence . The KL divergence is always positive , therefore by removing the last term we have : log pθ ( s ) ≥ Eqφ ( T , z|s ) [ log pθ ( s|T , z ) ] − KL [ qφ ( T , z|s ) |p ( T , z ) ] = Ẽθ , φ ( s ) ( 3 ) where the right-hand side is called the Evidence Lower Bound ( ELBO ) . By maximizing the ELBO term , the divergence KL [ qφ ( T , z|s ) ‖pθ ( T , z|s ) ] is implicitly minimized . Therefore , we define a surrogate objective , replacing the objective in Equation 1 : θ = arg max θ ∑ s max φ Ẽθ , φ ( s ) ( 4 ) The ELBO in Equation 4 has two components . First , the KL divergence with the prior , which usually has a closed form solution . For the distribution over dependency trees , it can be computed with the semiring algorithm of Li & Eisner ( 2009 ) . Second , the non-trivial term Eqφ ( T , z|s ) [ log pθ ( s|T , z ) ] . During training , Monte-Carlo method provides a tractable and unbiased estimation of the expectation . Note that a single sample from qφ ( T , z|s ) can be understood as encoding the observation into the latent space , whereas regenerating a sentence from the latent space can be understood as decoding . However , training a VAE requires the sampling process to be differentiable . In the case of the sentence embedding , we follow the usual setting and define qφ ( z|s ) as a diagonal Gaussian : backpropagation through the the sampling process z ∼ qφ ( z|s ) can be achieved thanks to the reparametrization trick ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . Unfortunately , this approach can not be applied to dependency tree sampling T ∼ qφ ( T |s ) . We tackle this issue in Section 5 .

Summary: This paper proposed a variational autoencoder-based method for semi-supervised dependency parsing. Given an input sentence s, an LSTM-based encoder generates a sentence embedding z, and a NN of Kiperwasser & Goldberg (2016) generates a dependency structure T. Gradients over the tree encoder are approximated by (1) adding a perturbation matrix over the weight matrix and (2) relax dynamic programming-based parsing algorithm to a differentiable format. The decoder combines standard LSTM and Graph Convolutional Network to generate the input sentence from z and T. The authors evaluated the proposed method on three languages, using 10% of the original training data as labeled and the rest as unlabeled data.