Venue: NAACL 2019

# Bayesian Learning for Neural Dependency Parsing

Ehsan Shareghi,\({}^{\spadesuit}\) Yingzhen Li,\({}^{\clubsuit}\) Yi Zhu,\({}^{\clubsuit}\) Roi Reichart,\({}^{\diamond}\) Anna Korhonen\({}^{\clubsuit}\)

\({}^{\spadesuit}\) Language Technology Lab, DTAL, University of Cambridge

\({}^{\spadesuit}\)Microsoft Research Cambridge, \({}^{\diamond}\)Technion, IIT

\({}^{\clubsuit}\)(es776, yz568, alk23)@cam.ac.uk

\({}^{\spadesuit}\)Yingzhen.Li@microsoft.com, \({}^{\diamond}\)roiri@technion.ac.il

###### Abstract

While neural dependency parsers provide state-of-the-art accuracy for several languages, they still rely on large amounts of costly labeled training data. We demonstrate that in the small data regime, where uncertainty around parameter estimation and model prediction matters the most, Bayesian neural modeling is very effective. In order to overcome the computational and statistical costs of the approximate inference step in this framework, we utilize an efficient sampling procedure via stochastic gradient Langevin dynamics to generate samples from the approximated posterior. Moreover, we show that our Bayesian neural parser can be further improved when integrated into a multi-task parsing and POS tagging framework, designed to minimize task interference via an adversarial procedure. When trained and tested on 6 languages with less than \(5k\) training instances, our parser consistently outperforms the strong Bilstm baseline Kiperwasser and Goldberg (2016). Compared with the biaffine parser Dozat et al. (2017) our model achieves an improvement of up to 3% for Vietnamese and Irish, while our multi-task model achieves an improvement of up to 9% across five languages: Farsi, Russian, Turkish, Vietnamese, and Irish.

## 1 Introduction

Dependency parsing is essential for many Natural Language Processing (NLP) tasks Levy and Goldberg (2014); Angeli et al. (2015); Toutanova et al. (2016); Hadiwinoto and Ng (2017); Marcheggiani et al. (2017). While earlier work on dependency parsing required careful feature engineering McDonald et al. (2005b); Koo et al. (2008), this has become less of a concern in recent years with the emergence of deep neural networks Kiperwasser and Goldberg (2016); Dozat et al. (2017). Nonetheless, an accurate parser still requires a large amount of labeled data for training, which is costly to obtain, while the lack of data often causes overfitting and poor generalization.

Several approaches for parsing in the small data regime have been proposed. These include augmenting input data with pretrained embedding Dozat et al. (2017); Che et al. (2018), leveraging unannotated data via semi-supervised learning Corro and Titov (2018), predicting based on a pool of high probability trees Niculae et al. (2018); Keith et al. (2018), and transferring annotation or model across languages Agic et al. (2016); Lacroix et al. (2016); Rasooli and Collins (2017). Despite the empirical success of these approaches, an inherent problem still holds: The maximum likelihood parameter estimation (mle) in deep neural networks (DNNs) introduces statistical challenges at both estimation (training), due to the risk of overfitting, and at test time as the model ignores the uncertainty around the estimated parameters. When training data is small these challenges are more pronounced.

The Bayesian paradigm provides a statistical framework which addresses both challenges by (i) including prior knowledge to guide the learning in the absence of sufficient data, and (ii) predicting under the full posterior distribution of model parameters which offers the desired degree of uncertainty by exploring the posterior space during inference. However, this solution comes with a high computational cost, specifically in DNNs, and is often replaced by regularization techniques such as dropout Srivastava et al. (2014) as well as ensemble learning and prediction averaging Liu et al. (2018); Che et al. (2018).

Bayesian neural networks (BNNs) have attracted some attention Welling and Teh (2011); Hernandez-Lobato and Adams (2015); Li et al. (2016); Gong et al. (2018). Yet, its current application to NLP is limited to language modeling Gan et al. (2017), and BNNs have not been developed for structured

prediction tasks such as dependency parsing.

In this paper we aim to close this gap and propose the first BNN for dependency parsing (BNNP). To address the costs of inference step, we apply an efficient sampling procedure via stochastic gradient Langevin dynamics (sgld) [25]. At training, samples from the posterior distribution of the parser parameters are generated via controlled noise injection to the maximum a posteriori (map) gradient update. The generated samples are then used during the inference step to create multiple viable parses based on which the final dependency parse is generated.

Another means of directing a model towards more accurate predictions in the small data regime is via multi-task learning [17]. Such a framework allows models for multiple tasks to reinforce each other towards more accurate joint solutions, which is particularly useful when training data is scarce. We hence present a multi-task framework where our BNNP is integrated with a POS tagger through an adversarial procedure designed to guide the two models towards improved joint solutions.

Our experiments with monolingual and delexicalized cross-lingual parsing using the Universal Dependency treebank [30] demonstrate the effectiveness of our approach. Particularly, our BNNP consistently outperforms the single task Bilstm baseline [46], while outperforming the biaffine parser [8] by up to \(3\%\) on Vietnamese and Irish. Additionally, our multi-task model achieves an improvement of up to 9% over the biaffine parser for five low-resource languages: Farsi, Russian, Turkish, Vietnamese, and Irish.

## 2 Bilstm Dependency Parsing

Our parser extends the graph-based bilstm parser of kiperwasser2016. We briefly review their work and its notable extensions, and then discuss our extension of their architecture and the limitations of mle training.

ModelGiven an input word sequence \({w_{1}}_{:n}\),1 and its corresponding POS tag sequence \(p_{1}{:}\), each input word is encoded as a \(d\)-dimensional representation,

Footnote 1: We overload the notation \(w_{v}\), and it refers to both a word and its \(d\)-dimensional representation.

\[w_{i}=e(w_{i})\circ e(p_{i})\circ\hat{e}(w_{i}),\]

where \(e(.)\) denotes trainable embedding, \(\hat{e}\) denotes pretrained external embedding, and \(\circ\) denotes concatenation. The context dependent representation of each word is generated through a bilstm,

\[v_{i}=\textsc{lstm}_{\zeta}(w_{1}{:}_{j})\circ\textsc{dlstm}_{\phi}(w_{n}{:} _{i}).\]

Here, \(\zeta\) and \(\phi\) denote the lstems' parameters. The resulting sequence of \(l\)-dimensional vectors, \(v_{1}{:}_{n}\), is then used for computing the arc-score matrix (\(n\times n\)) where cell arc-score\((i,j)\) is computed as,

\[V\times\tanh\left(\underbrace{W^{(arc-head)}v_{i}+\underbrace{W^{(arc- mod)}v_{j}}_{\text{med~{}deculiar~{}specialized~{}$w_{j}$}}+b}_{\text{modular~{} specialized~{}$w_{j}$}}+b}_{\text{modifier~{}specialized~{}$w_{j}$}}+b}_{\text{modifier~{}specialized~{}$ w_{j}$}}+b\right),\]

indicating the likelihood of having the arc from \(w_{j}\) to \(w_{j}\). Here, \(V\) is \((1\times h)\), \(b\) is \((h\times 1)\), and \(W^{(.)}\) is a \((h\times l)\) matrix used for creating specialized versions of words in \(w_{1}{:}_{n}\). The \(h\)-dimensional specialized representation of \(w_{i}\) as a head or a modifier is generated via multiplication with \(W^{(arc-head)}\) and \(W^{(arc-mod)}\), respectively. Following a first-order dependence assumption between the arcs, the parsing is done via a dynamic programming solution that finds the dependency parse \(T^{*}\) such that,

\[T^{*}=\arg\max_{T}\left(\textsc{score}(T)=\sum_{(i,j)\in T}\textsc{arc- score}(i,j)\right).\]

Next, given \(T^{*}\), for each arc \((i,j)\in T^{*}\) the label-score\((i,j,r)\) is computed as:

\[\left(V^{\prime}\times\operatorname{tanh}(\underbrace{W^{\prime(arc-head)} v_{i}}_{\text{head specialized~{}$w_{i}$}}+\underbrace{W^{\prime(arc-mod)}v_{j}}_{\text{modifier~{}specialized~{}$w_{ j}$}}+b^{\prime})\right)[r],\]

where \(r\) is a dependency relation type, \(r\in\{r_{z}\}_{z=1}^{r}\), \(V^{\prime}\) is \((\mathbf{r}\times h)\), \(b^{\prime}\) is \((\mathbf{r}\times 1)\), and \(W^{\prime(.)}\) is \((h\times l)\). The label for each dependency arc \((i,j)\) is then chosen by a max operation.

Dozat et al. Dozat et al. (2017) proposed an extension of the bilstm parser by replacing the non-linear transformation of arc-score and label-score with a linear transformation (biaffine). This was further extended by che2018 who utilized contextualized word embeddings [10]. Both extensions showed success in dependency parsing shared tasks [30, 26].

#### Our Neural Parser Architecture

Our network architecture extends the bilstm model with an additional bilstm layer and input signals. While our 

architecture is not the core contribution of this paper, we aim to implement our BNNP on a strong architecture. In SS5.3 we demonstrate the contribution of these additions to our final results.

In our BNNP, each word is represented as,

\[w_{i}=e(w_{i})\mathtt{o}e(p_{j})\mathtt{o}\hat{e}(w_{ij})\mathtt{o}e(c_{i}),\]

where, similar to the Biaffine parser, \(e(c_{i})\) is a character-level representation of the word \(w_{i}\), generated by a Bilstm:

\[e(c_{i})=\textsc{lstm}_{a}(c_{1:|w_{i}|})\mathtt{oLSTM}_{\delta}(c_{|w_{i}|:1}).\]

We consider a maximum of two layers of Bilstms, denoted as \(1^{st}\) and \(2^{nd}\).2 The output of the \(2^{nd}\)Bilstm is denoted as,

Footnote 2: Stacking BiLstms is believed to be helpful. In our case, the addition of a third layer led to overfitting.

\[u_{i}=\textsc{lstm}_{\kappa}(v_{1:i})\mathtt{oLSTM}_{\delta}(v_{n:i}),\]

where \(v_{i}\) is the encoding generated by the \(1^{st}\) layer of the Bilstms.

LearningFor structured prediction, the objective function is to maximize the margin between the gold structure \(\mathbb{T}\) and the other structures. To achieve this, the cost augmented hinge loss is used, where the fixed margin \(m\) is replaced by a cost function, \(\textsc{cost}(\mathbb{T},T)\), sensitive to the local mistakes (here, each local mistake has a constant cost of 1), and computed as:

\[\mathcal{L}_{arc}=\max\!\left(0,\textsc{cost}(\mathbb{T},T)+\textsc{score}( T)-\textsc{score}(\mathbb{T})\right)\]

For label prediction, hinge loss with \(m=1\) is used (denoted by \(\mathcal{L}_{rel}\)). We refer to the parser loss as:

\[\mathcal{L}_{parse}=\mathcal{L}_{arc}+\mathcal{L}_{rel}.\]

Beyond mle TrainingThe point-estimate of DNN parameters is computationally efficient, but ignores the uncertainty around model parameters during learning. This results in an overconfidence over model predictions during the inference phase. A common generic practice to incorporate a degree of uncertainty is to consider an ensemble of models. Indeed, for dependency parsing ensemble learning has shown to improve accuracy Surdeanu and Manning (2010); Kuncoro et al. (2016); Che et al. (2018). However, ensembles are computationally demanding due to the large number of participating models. The de-facto approach to overcome this has been to randomly perturb the structure of the network for each training instance by switching off connections between the nodes, a practice known as dropConnect Wan et al. (2013), or eliminating the nodes entirely, which is known as dropout Srivastava et al. (2014).

Hinton et al. (2012) demonstrated that dropping out a node with probability \(\rho\) from a neural model \(\mathcal{M}\) at training, and scaling the same node's output at test time (in the fully connected \(\mathcal{M}\)), performs very well in practice. While this scaling trick avoids the need for running multiple models at test time, it lacks theoretical guarantees when applied to DNNs. Gal and Ghahramani (2016) showed that dropout can be casted as integrating out the posterior of model parameters, and in this regard is a form of Bayesian approximation. Still, dropout is not as effective when applied to the small data regime Goodfellow et al. (2016). We hence propose a fully Bayesian approach.

## 3 Bayesian Neural Parsing

The Bayesian paradigm is appealing in its ability to capture uncertainty around parameters, avoid overfitting, and incorporate prior knowledge to compensate for the lack of sufficient training data in resource lean settings. Dependency parsing may be a natural candidate for Bayesian modeling since a typical sentence can have multiple viable parses corresponding to different grammatical ambiguities, and considering these possibilities have shown to improve predictive accuracy Niculae et al. (2018); Keith et al. (2018). In the same spirit, Bayesian inference can be interpreted as a statistically grounded means to generate this pool of high quality trees.

In detail, Bayesian inference for parsing computes the _predictive distribution_\(P(T|s,D)\)3 of the dependency parse tree \(T\) for an input sentence \(s\) given the training data \(D\) by _posterior averaging_:

Footnote 3: When \(P(T|s,\theta)\) is replaced by the unnormalized score from the first-order arc-factored model, \(P(T|s,D)\) can be interpreted as the expected score assigned to a dependency tree under the full posterior distribution of model parameters.

\[P(T|s,D)=\int P(T|s,\theta)P(\theta|D)d\theta,\]

where \(P(\theta|D)\propto p(D|\theta)p(\theta)\) denotes the posterior distribution of the BNNP parameters \(\theta\) given observations \(D\). Here the prior distribution \(p(\theta)\) is a standard Gaussian \(\mathcal{N}(0,I)\), and, by assuming i.i.d training datapoints, the likelihood term 

is \(P(D|\theta)=\prod_{x\in D}P(x_{i}|\theta)\) where \(x_{i}=(s_{i},T_{i})\) is a training instance.

The above integral is intractable, however an empirical estimate can be computed using \(\mathcal{K}\) samples from the posterior,

\[P(T|s,D)\approx\frac{1}{\mathcal{K}}\sum_{\{\theta_{k}\sim P(\theta|D)\}^{ \mathcal{K}}_{k=1}}P(T|s,\theta_{k}). \tag{1}\]

Therefore, efficient generation of posterior samples is the key for applying the Bayesian treatment. Since the posterior is intractable in DNNs, additional machineries such as Markov Chain Monte Carlo (mcmc)Robert and Casella (2010) are required for sampling. In this paper we explore an efficient class of Stochastic Gradient Markov Chain Monte Carlo (sg-mcmc) methods, designed for NNs by adjusting stochastic gradient descent with properly scaled Gaussian noise to generate posterior samples.

### Stochastic Gradient Langevin Dynamics

We first consider the Stochastic Gradient Langevin Dynamics (sgld)Welling and Teh (2011) sampler to generate posterior samples. Intuitively, this mechanism guides sgd to explore the posterior distribution rather than finding the maximum a posteriori (map) solution. More concretely, at each time step \(t\) we first compute a stochastic estimate of the gradient for the negative log-posterior distribution on a mini-batch \(b_{t}\):

\[\nabla_{\theta_{t}}\hat{\mathcal{L}}\approx-\bigg{(}\nabla_{\theta_{t}}\log P (\theta_{t})+\frac{|D|}{|b_{t}|}\underset{x_{i}\in b_{t}}{\sum}\nabla_{ \theta_{t}}\log P(x_{i}|\theta_{t})\bigg{)},\]

and then update the BNNP parameters by SGD with noise injection,

\[\theta_{t+1}\leftarrow\theta_{t}-\frac{\eta_{t}}{2}\nabla_{\theta_{t}}\hat{ \mathcal{C}}+\xi_{t},\quad\xi_{t}\sim\mathcal{N}(0,\eta_{t}\mathbf{I}) \tag{2}\]

Theoretically, the distribution of \(\theta_{t}\) converges to the true posterior \(p(\theta|D)\) when \(b_{t}=D\), \(t\rightarrow\infty\) and \(\eta_{t}\to 0\)see Teh et al. (2016) for sgdl convergence rate proofs and analysis).

In practice, the learning rate \(\eta_{t}=a(b+t)^{-7}\), starting high and decaying, plays a crucial role in eqn. 2 dynamics. Intuitively, while the variance of the injected noise is \(\eta_{t}\), the variance of the gradient is of \(\mathcal{O}(\eta_{t}^{2})\). In the earlier stages of the optimization, the \(\frac{\eta_{t}}{2}\nabla_{\theta}\mathcal{L}\) term is dominant as gradients are of greater impact and the learning rate is high, simulating stochastic gradient descent. As the optimization proceeds, the impact of the gradient shrinks and the injected Gaussian noise becomes the dominant term, transitioning from sgd to Langevin Monte CarloNeal (2011).

Since \(\eta_{t}\to 0\) might cause slow mixing in practice, we also set a minimum threshold for the learning rate to allow mixing at later stages. sgdl and sgd have the same time complexity as sampling occurs along with parameter updates with a negligible overhead of drawing \(\xi_{t}\)s.

### Preconditioned sgld

As sgdL relies on a single learning rate along all dimensions of \(\theta\), it has the same potential inefficiencies of sgd for optimizing functions with different curvatures along each dimension. Inspired by more advanced optimization techniques that adjust to the geometry of parameter space, Li et al. (2016) proposed preconditioning of sgdl (similar to RMSpropTieleman and Hinton (2012)) which scales the gradients with respect to the weighted average of the gradients along each dimension, such that a unified learning rate is sufficient. The proposed preconditioner diagonal matrix is computed as follows,

\[G(\theta_{t}) =\text{diag}(1~{}\mathcal{O}~{}(\lambda 1+\sqrt{V(\theta_{t})})),\] \[V(\theta_{t}) =aV(\theta_{t-1})+(1-\alpha)\nabla_{\theta_{t}}\hat{\mathcal{L}} \odot\nabla_{\theta_{t}}\hat{\mathcal{L}},\]

where \(\odot,\mathcal{O}\) are element-wise matrix product and division, and \(\lambda,\alpha\) are hyperparameters controlling the extremes of the curvature, and weighting average of previous and current gradients, respectively. The updated preconditioned eqn. 2 is,

\[\theta_{t+1}\leftarrow\theta_{t}-\frac{\eta_{t}}{2}G(\theta_{t})\nabla_{ \theta_{t}}\hat{\mathcal{L}}+\xi_{t},\quad\xi_{t}\sim\mathcal{N}(0,\eta_{t} G(\theta_{t})).\]

Here, replacing the preconditioner matrix \(G(\theta_{t})\) with an identity matrix will recover the sgdl update formulation. In case of a diagonal preconditioner, it has the same time complexity of sgdl, although the constant factors in the complexity figure can be slightly higher due to additional computations involved in \(G(\theta_{t})\) and \(V(\theta_{t})\).

While sgdl is designed to explore different modes of the posterior distribution, we found (SS5.4) that in practice we still require dropout to facilitate modes exploration via random perturbation of the model structure.

### Prediction

The generated posterior samples can be used to compute the approximate expectation of eqn. 1 by scoring all plausible parses via explicit model averaging 

As this solution is not computationally feasible, we use the sampled parameters and follow a procedure that minimizes the Bayes risk (mbr)Goodman (1996). Given each sampled parameter, first we generate the maximum scoring parse using the arc-factored decomposition McDonald et al. (2005a) and dynamic programming Eisner (1996). This can be done concurrently for all samples, resulting in a running time identical to the non-Bayesian approach. For each labelled edge, we replace its score in the arc-score matrix with its occurrence count in the collection of sampled trees and infer the final tree using counts as scores. The predicted structure is then passed to the label predictor, which assigns labels to the edges (SS2). This decoding approach, while selecting the global structure with the highest probability under the approximate posterior, could potentially allow for additional corrections of the highest scoring tree in the pool of samples Shareghi et al. (2015); Kuncoro et al. (2016).

## 4 Multi-Task Learning

Multi-task learning Caruana (1997) lends itself as a natural choice for low-resource settings as it aims at leveraging the commonality between tasks to improve their performance in the absence of sufficient amount of training data. This framework hence naturally complements Bayesian modeling in dealing with the challenges of the small data regime.

We couple our BNNP with POS tagging due to the strong connection between the two tasks Rush et al. (2010) and the availability of joint training data in several languages Zeman et al. (2017). While multi-task frameworks have shown success in some areas Reichart et al. (2008); Finkel and Manning (2009); Liu et al. (2016); Malca and Reichart (2018), in our case we found that our two tasks interfered with each other and degraded the parser performance (see similar findings for other tasks at Sogaard and Goldberg (2016); Plank and Alonso (2017)).

To minimize task interference, an approach shown effective Ganin and Lempitsky (2015); Kim et al. (2017); Chen et al. (2017); ZareMoodi and Haffari (2018) is to implicitly guide the update signals during training via an adversarial procedure that avoids shared parameters contamination. We adapt this idea to our multi-task learning.

ArchitectureGiven an input word, \(w_{i}\), the score for a POS tag \(p\) is computed as \(\tanh(W^{\prime\prime\prime}v_{i}+b^{\prime\prime})[p]\), where \(p\in\{p_{z}\}_{=1}^{\mathbf{p}}\), \(W^{\prime\prime\prime}\) is \((\mathbf{p}\times l)\), and \(b^{\prime\prime}\) is \((\mathbf{p}\times 1)\). To train the POS tagger, the cross-entropy loss is used (denoted by \(\mathcal{L}_{pos}\)). Taking \(w_{i}\)s as input, the _shared Bilstm_ strictly encodes a task-agnostic \(l\)-dimensional representation of the input (explained in the next subsection),

\[s_{i}=\textsc{ls}\widehat{\textsc{tm}}_{\rho}(w_{1:j})\textsc{ols}\widehat{ \textsc{tm}}_{\chi}(w_{n:i}).\]

The input to each task is then defined as the concatenation of task-agnostic and task-specific representations. We considered a basic architecture where both tasks have their separate parameters (identical number of layers, dimensions, etc.) and they only share the _shared Bilstm_ (denoted as **multi Task** in Table 1). 3

Footnote 3: We also tried layer-wise placements of tasks Saggaard and Goldberg (2016); Hashimoto et al. (2017) and the results were slightly worse. Details are omitted for space reason.

Adversarial TrainingThe _shared_Bilstm output, \(s_{i}\), is meant to encode a task-agnostic representation of the input word \(w_{i}\). In order to enforce this criterion, we apply an adversarial training procedure. The shared representation, \(s_{i}\), is forwarded to a task discriminator (denoted as _Disc._ in Figure 1) through a gradient reversal layer. The task discriminator predicts the task identity for each word in the input via a linear transformation \((W^{\prime\prime\prime}s_{i}+b^{\prime\prime\prime})[\tau]\) followed by a softmax, where \(\tau\in\{\texttt{parser},\texttt{tagger}\}\) and \(W^{\prime\prime\prime}\) and \(b^{\prime\prime\prime}\) are \((2\times l)\) and \((2\times 1)\), respectively. To train the discriminator, a sum of the cross-entropy losses for \(w_{1:n}\) is used (denoted by \(\mathcal{L}_{disc}\)). As the parameters of the discriminators are being updated, the gradient signals to minimize the discriminator's error are backpropagated with an opposite sign to the _shared_ Bilstm layer, which adversarially encourages the _shared_Bilstm to fool the discriminator.

Our training schedule alternates between the two modes, in one mode optimizing the shared and task-specific parameters based on \(\mathcal{L}_{parse}\) and \(\mathcal{L}_{pos}\) (in

Figure 1: The **multi task** architecture flow for a single word in a sequence.

 

random order), while in the other mode optimizing \({\cal L}_{disc}\) which includes the shared and discriminator-specific parameters.

## 5 Experiments and Results

We experiment with mono-lingual and cross-lingual dependency parsing using the treebanks of the CoNLL 2017 shared task on parsing to Universal Dependencies (UD) Zeman et al. (2017).5

Footnote 5: For train and dev sets (1-1983), test set (1-2184), and pretrained embeddings (1-1989) see: [https://lindat.mff.cuni.cz/repository/xmui/handle/11234/](https://lindat.mff.cuni.cz/repository/xmui/handle/11234/)(1-1983,1-2184,1-1989)

### Experimental Setup

We use the UDPipe baseline outputs for segmentation and POS tagging of the raw test data (released along with the raw test data). While segmentation and POS errors substantially impact the quality of the final predicted parse, their exploration is beyond our scope. Our evaluation metric is Labeled Attachment Score (LAS), computed by the shared task evaluation script. Statistical significance, when mentioned, is computed over 20 runs, via the Kolmogorov-Smirnov test Reimers and Gurevych (2017); Dror et al. (2018) with \(p=0.01\).

Mono-Lingual ExperimentsWe experiment with Persian (fa), Korean (ko), Russian (ru), Turkish (tr), Vietnamese (vi) and Irish (ga), all with less than 5\(k\) training sentences (Table 1). For comparison we report the scores published by the top system of the CoNLL 2017 shared task, biaffineDozat et al. (2017), noting the following differences between their input and output and ours. The biaffine parser: (i) uses the UDPipe outputs for segmentation but corrects POS errors before parsing, (ii) includes both language specific and universal POS tags in the input layer while we only include the universal POS tags, and (iii) applies post-process correction for non-projective languages.

Cross-Lingual ExperimentsWe use the English (en), French (fr), Russian (ru), and Persian (fa) datasets of the UD treebanks as our training and test data, with the addition of 3 languages for which we did not have any training data: Kurmanji (kmr), Buriat (bxr), and Northern Sami (sme). We also report the results for each language, where the combination of training datasets for the rest of the languages (marked as +) was used for training. The cross-lingual experiments are done on delexicalized parses after replacing the words with their Universal POS tags.

Models and Baselines - Single-TaskWe consider the following models: base is the Bilstm model of Kiperwasser and Goldberg (2016); base++ extends base by having 2 layers of Bilstms and using 1 layer of character level Bilstm (SS2); +shared includes an additional BiLSTM (dashed box in Figure 1). We included this to provide a fair comparison (in terms of the number of parameters) with the multi-task experiments but we apply a higher dropout rate to resolve overfitting; ensemble denotes a collection of 9 +shared models each randomly initialized Reimers and Gurevych (2017) and trained for mle with MBR (SS3.3) applied for prediction; map denotes the +shared model optimized for map instead of mle; +sgld denotes Bayesian learning and prediction (SS3.1), and **+precond** denotes preconditioned sgld (SS3.2). +sgld and **+precond** are applied to the +shared model.

Models and Baselines - Multi-TaskWe consider the same models as in the single-task setup, with the following changes. basemt is the variant of multitask architecture where the _shared_ Bilstm is removed and all components except for decoders are shared between the two tasks. +shared and +adv denote the results without and with the adversarial training (SS4), respectively.

### Hyperparameters

Training is done for 330 epochs with early stopping, using successive6 mini-batches of 5000 words,7 with drop-out rate 0.33 unless stated otherwise. sgld is only applied for learning the parameters of the parser, with learning rate decaying from 0.01 to 0.0001 (\(\gamma=0.5\)), and the preconditioned matrix hyperparameters are \(\lambda=10^{-5},\alpha=0.99\). The rest of the parameters are updated using Adam Kingma and Ba (2014) with default DyNet parameters Neubig et al. (2017). The size and selection of the samples used in Bayesian inference are tuned on Irish (see SS6.1 of the main paper). In all experiments, training stops based on loss convergence of a single model on the dev set. We also tried grus and our results were substantially worse than lstMs.

Footnote 6: We found that random batches perform slightly worse.

Footnote 7: When the 5000 words limit lands in the middle of a sentence, the entire sentence is included.

We consider the following sizes within the models: base is 1 layer of 

units, 100-D word embeddings, 25-D POS embeddings, and dropout rate of 0.33; base++ extends base by having 2 layers of Bilstms with 200-D hidden units, 100-D external word embeddings, and using 1 layer of character level Bilstm with 200-D hidden units to generate 100-D character embeddings (SS2); +shared includes an additional Bilstm with 200-D hidden units in base++ to provide a fair comparison (in terms of the number of parameters) with the multi-task experiments but we apply a higher dropout rate of 0.66 to resolve overfitting.

### Results

Mono-Lingual ParsingTable 1 summarizes the training data statistics and the LAS results of the various models, with bold-font marking cases where a model outperforms the biaffine parser.

**1.****single task Comparison between base++ and +shared reveals that the additional BiLSTM in the +shared does not improve the results. As expected, model averaging in ensemble improved the results over the +shared model. Additionally, the map results show slight improvements compared with +shared. Both of these findings suggest that the quality of predictions is likely to improve if some notion of uncertainty (via averaging, or prior insertion) is included.**

For Bayesian solutions, both +sgld and +precond consistently (and in a statistically significant manner) outperform the ensemble models and the strong bilsm baselines on all languages, while precond outperforms biaffine by up to 3% on two languages (vi, and ga). The consistency of improvements as we incorporate richer means of capturing the uncertainty suggests that these gains are independent of our specific choice of neural architecture and that they might also hold for future parsing architectures.

**2.****multi task As explained earlier (SS4), comparing the parser performances under +shared models in single and multi task settings indicates that parser quality was degraded with the inclusion of a POS tagging task, possibly due to interference between the tasks. This issue was alleviated with the inclusion of task discriminator and the adversarial training. The +adv model consistently improves the results (in a statistically significant manner), while outperforming the single task (+shared) by 2.2% for Persian (fa), and 5.2% for Irish (ga). This is an indicator that low-resource languages could potentially benefit more from multi-task learning. Other variants of +adv are also reported in Table 1 and similar patterns to the single task setting are observed.**

Next, we test the effectiveness of Bayesian learning and inference using \({\cal K}=9\) thinned samples (SS5.4). For all languages in the multi-task setting the gain from including +sgld in +adv is statistically significant. However, the gain decays as the amount of training data increases: from 3.8% on Irish (ga) to 1.1% on Persian (fa). This verifies our expectation that Bayesian learning helps parameter estimation and prediction in the small data regime. Compared to +sgld, +precond provides further improvements (all are statistically significant) of 2.2% on Irish and 1% on Persian, showing a similar generic negative correlation with data size.

To summarize our mono-lingual results, the Bayesian framework shows its merit in improving predictive quality, and multi-task learning introduces new and informative signal via a related task which allows for better parameter estimation. The integration of a Bayesian parser into a multi-task learning framework outperforms the biaffine parser on up to 5 languages, while we observe decreasing improvements as the data size grows. Also, since our Bayesian approach builds on the base model, it is subject to the shortcomings of this model: For instance, on Korean (ko) the difference between base and biaffine is too large to be closed without further adjustment of model design,

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & fa & ko & ru & tr & vi & ga \\ train(sen.) & 4798 & 4400 & 3850 & 3685 & 1400 & 566 \\ \hline \multicolumn{6}{c}{**Labeled Attachment Score (LAS)**} \\ biaffine & 86.31 & **82.49** & 83.65 & 62.79 & 42.13 & 70.06 \\ base & 80.97 & 64.76 & 75.45 & 52.64 & 39.36 & 62.50 \\ \hline \multicolumn{6}{c}{**SINGLE Task**} \\ base++ & 83.15 & 76.70 & 79.44 & 58.92 & 41.03 & 66.58 \\ +shared & 83.11 & 76.32 & 79.62 & 58.72 & 41.10 & 66.42 \\ ensemble & 84.12 & 77.28 & 80.17 & 59.36 & 41.89 & 68.12 \\ map & 83.59 & 76.61 & 79.78 & 59.13 & 41.33 & 66.95 \\ +sgld & 84.98 & 78.91 & 80.86 & 60.53 & **43.12** & 69.51 \\ +precond & 85.76 & 79.83 & 81.9 & 61.71 & **45.25** & **70.91** \\ \multicolumn{6}{c}{**MITask**} \\ basemt & 81.54 & 75.78 & 78.61 & 57.12 & 40.08 & 60.51 \\ +shared & 81.03 & 75.08 & 78.64 & 57.09 & 40.04 & 65.24 \\ adv & 84.93 & 78.12 & 81.23 & 60.66 & **43.11** & 69.89 \\ ensemble & 85.01 & 78.31 & 81.56 & 60.92 & **43.3** & 70.00 \\ map & 85.02 & 78.26 & 81.59 & 60.72 & **43.25** & **70.36** \\ +sgld & 85.89 & 79.50 & 83.06 & 62.13 & **44.57** & **72.52** \\ +precond & **86.75** & 80.97 & **84.51** & **63.24** & **45.96** & **74.12** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Mono-Lingual results. Models that outperform the biaffine parser are highlighted in bold.

 

or its inputs and outputs.

Cross-Lingual Delexicalized ParsingWe tested our most successful mono-lingual model, **multi task(+precond)**, in the cross-lingual setup against an ensemble of randomly initialized **single task** (base++) models. Table 2 summarizes the training data size (rounded up) and the LAS results of both models.

As expected, the ensemble models perform best when the training and test data come from the same language or language family (as in Buryat to Russian and Kurmanji to Persian). Showing a similar pattern to mono-lingual parsing, in all cases the performance gain of the Bayesian **multi task** (+precond) setting, reported as under-text in the table, consistently outperforming the ensemble (in a statistically significant manner) and the impact decays with the training set size.

### Ablation Analysis

We aim to answer two questions: (i) how many of the generated samples are used during inference? and (ii) how much the success of sgld depends on other sources of noise during training?

1. Utilizing SamplesOnce the training stops, different strategies could be applied to utilize the collection of posterior samples to recompute arc-score matrix (see SS3.3). An extreme option is including all samples (denoted by _All_), another option is to only use the last sample (denoted by _Last_). An intermediate alternative is to choose samples with interval \(\lfloor\frac{M}{K}\rfloor\), an approach known as _Thinning_[6].

Comparing the results in Figure 1(a), _Thinning_ consistently outperforms the other strategies. Interestingly, when we compared _Thinning_ with a variant (denoted by \(\Delta\) in Figure 1(a)) that just uses the last 5 samples , it appears that _Thinning_ which includes earlier samples is still superior. This is likely to be an indicator that a larger (and more diverse) set of trees can potentially improve mbr decoding. In practice, we use _Thinned_ samples.

2. Dropout, sgld, and Mini-BatchWe would next like to better understand the interdependence between sgld and two existing sources of noise: dropout and mini-batch size. Results are reported in Figure 1(b). In all four stacked bars, the top segment demonstrates the gain when including +sgld. The right-most stacked bar illustrates the performance of **multi task (+ADV)** without (bottom part of the bar) and with +sgld. This is our reference and in the following three experiments, we make a single change to this model while keeping everything else fixed.

In the left-most stacked bar, we exclude dropout. This significantly hurts the performance, an indication that the random perturbation of model by dropout can provide a positive complimentary effect for better exploration of the posterior modes via sgld. Comparing the top parts of all stacked bars (representing the gain with +sgld) reveals that this gain is at its maximum when dropout is switched off. An interpretation can be that in the absence of dropout, sgld becomes the main component for countering overfitting.

In the second stacked bar from the right, we remove the noise caused by mini-batching and use the entire dataset as one batch. This slightly improves the results, although this improvement is not statistically significant. Based on eqn. 2, full training data gradient updates cause \(\nabla_{g}\mathcal{L}\to 0\) in fewer steps. Consequently, the learning rate (which controls the variance of the noise) is at a higher

\begin{table}
\begin{tabular}{r 

value when the gradient is close to zero, giving a better chance for the sampler to mix via \(\xi_{t}\).

In the second stacked bar from left, parameters are updated for each training sentence. In this setting the learning rate, \(\eta_{t}\), is consumed much faster and can potentially reach zero on the map solution, discarding the effect of the injected noise and +sgld. Hence, the gain for including sgld is at its minimum. We speculate that adjusting the decaying speed of the learning rate (which we left untouched) could allow for this extreme case to perform better.

## 6 Conclusion

We proposed a Bayesian framework for neural dependency parsing (BNNP). We employ efficient SG-MCMC approximate inference mechanisms through stochastic gradient Langevin dynamics to generate posterior samples during optimization. The collected samples are then used via a minimum Bayes risk parsing algorithm to generate the final parse tree. In mono-lingual and cross-lingual experiments in the small data regime, where Bayesian learning in expected to be most effective, our BNNP consistently outperformed the strong Bilst baselines. Moreover, when integrating the BNNP into a multi-task learning framework, utilized to prevent task interference, we outperformed the biaffine parser (best system of the CoNLL17 shared task) on 5 low-resource languages by up to 9% LAS.

In future work, we intend to investigate other types of priors over the network parameters (e.g., sparse priors Lobacheva et al. (2017)). We would also like to explicitly quantify the uncertainty captured in our framework under different sampling strategies or mcmc-sg methods (e.g., similar to McClure and Kriegeskorte (2016); Teye et al. (2018)).

#### Acknowledgments

This work is supported by the ERC Consolidator Grant LEXICAL (648909). The first author would like to thank Costanza Conforti, Victor Prokhorov, and Gamal Crichton for their comments on the presentation of this work. The authors would like to thank the anonymous reviewers for their helpful suggestions.

## References

* Agic et al. (2016) Zeljko Agic, Anders Johannsen, Barbara Plank, Hector Martinez Alonso, Natalie Schluter, and Anders Sogaard. 2016. Multilingual projection for parsing truly low-resource languages. _TACL_.
* Angeli et al. (2015) Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In _Proceedings of ACL_.
* Caruana (1997) Rich Caruana. 1997. Multitask learning. _Machine learning_.
* Che et al. (2018) Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In _Proceedings of CoNLL_.
* Chen et al. (2017) Xinchi Chen, Zhan Shi, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-criteria learning for chinese word segmentation. In _Proceedings of ACL_. Corro and Titov (2018) Caio Corro and Ivan Titov. 2018. Differentiable perturb-and-parse: Semi-supervised parsing with a structured variational autoencoder. _arXiv preprint arXiv:1807.09875_.
* Dozat et al. (2017) Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford's graph-based neural dependency parser at the conll 2017 shared task. In _Proceedings of CoNLL_.
* Dror et al. (2018) Roten Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In _Proceedings of ACL_.
* Eisner (1996) Jason M Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In _Proceedings of COLING_.
* Finkel and Manning (2009) Jenny Rose Finkel and Christopher D Manning. 2009. Joint parsing and named entity recognition. In _Proceedings of NAACL_.
* Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In _Proceedings of ICML_.
* Gan et al. (2017) Zhe Gan, Chunyuan Li, Changyou Chen, Yunchen Pu, Qinliang Su, and Lawrence Carin. 2017. Scalable bayesian learning of recurrent neural networks for language modeling. In _Proceedings of ACL_.
* Ganin and Lempitsky (2015) Yaroslav Ganin and Victor S. Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In _Proceedings of ICML_.
* Gong et al. (2018) Wenbo Gong, Yingzhen Li, and Jose Miguel Hernandez-Lobato. 2018. Meta-learning for stochastic gradient MCMC. _CoRR_.
* Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. _Deep Learning_. MIT Press.
* Goodman (1996) Joshua Goodman. 1996. Parsing algorithms and metrics. In _Proceedings of ACL_.
* Hadiwinito and Ng (2017) Christian Hadiwinito and Hwee Tou Ng. 2017. A dependency-based neural reordering model for statistical machine translation. In _Proceedings of AAAI_ 

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In _Proceedings of EMNLP_.
* Hernandez-Lobato and Adams (2015) Jose Miguel Hernandez-Lobato and Ryan P. Adams. 2015. Probabilistic backpropagation for scalable learning of bayesian neural networks. In _Proceedings of ICML_.
* Hinton et al. (2012) Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. _CoRR_.
* Keith et al. (2018) Katherine A. Keith, Su Lin Blodgett, and Brendan T. O'Connor. 2018. Monte carlo syntax marginals for exploring and using dependency parses. In _Proceedings of NAACL_.
* Kim et al. (2017) Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017. Cross-lingual transfer learning for POS tagging without cross-lingual resources. In _Proceedings of EMNLP_.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _CoRR_.
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _TACL_.
* Koo et al. (2008) Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In _Proceedings of ACL_.
* Kulcoro et al. (2016) Adhiguna Kulcoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one MST parser. In _Proceedings of EMNLP_.
* Lacroix et al. (2016) Ophelie Lacroix, Lauriane Aufrant, Guillaume Wisniewski, and Francois Yvon. 2016. Frustratingly easy cross-lingual transfer for transition-based dependency parsing. In _Proceedings of NAACL_.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In _Proceedings of ACL_.
* Li et al. (2016) Chunyuan Li, Changyou Chen, David E. Carlson, and Lawrence Carin. 2016. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In _Proceedings of AAAI_.
* Liu et al. (2016) Pengfei Liu, Kipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. In _Proceedings of IJCAI_.
* Liu et al. (2018) Yijia Liu, Yi Zhu, Wanxiang Che, Bing Qin, Nathan Schneider, and Noah A. Smith. 2018. Parsing tweets into universal dependencies. In _Proceedings of NAACL_.
* Lobacheva et al. (2017) Ekaterina Lobacheva, Nadezhda Chirkova, and Dmitry P. Vetrov. 2017. Bayesian sparsification of recurrent neural networks. _CoRR_.
* Malca and Reichart (2018) Rivka Malca and Roi Reichart. 2018. Neural transition based parsing of web queries: An entity based approach. In _Proceedings of EMNLP_.
* Marcheggiani et al. (2017) Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In _Proceedings of CoNLL_.
* McClure and Kriegeskorte (2016) Patrick McClure and Nikolaus Kriegeskorte. 2016. Representation of uncertainty in deep neural networks through sampling. _CoRR_.
* McDonald et al. (2005a) Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005a. Online large-margin training of dependency parsers. In _Proceedings of ACL_.
* McDonald et al. (2005b) Ryan T. McDonald, Koby Crammer, and Fernando C. N. Pereira. 2005b. Online large-margin training of dependency parsers. In _Proceedings of ACL_.
* Neal (2011) Radford M Neal. 2011. MCMC using hamiltonian dynamics. _Handbook of Markov Chain Monte Carlo_.
* Neubig et al. (2017) Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. _arXiv preprint arXiv:1701.03980_.
* Niculae et al. (2018) Vlad Niculae, Andre F. T. Martins, Mathieu Blondel, and Claire Cardie. 2018. Sparsemap: Differentiable sparse structured inference. In _Proceedings of ICML_.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of NAACL_.
* Plank and Alonso (2017) Barbara Plank and Hector Martinez Alonso. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In _Proceedings of EACL_.
* Rasooli and Collins (2017) Mohammad Sadegh Rasooli and Michael Collins. 2017. Cross-lingual syntactic transfer with limited resources. _TACL_.
* Reichart et al. (2008) Roi Reichart, Katrin Tomanek, Udo Hahn, and Ari Rappoport. 2008. Multi-task active learning for linguistic annotations. In _Proceedings of ACL_.
* Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In _Proceedings of EMNLP_.
* Robert and Casella (2010) Christian P. Robert and George Casella. 2010. _Monte Carlo Statistical Methods_.
* Rush et al. (2010) Alexander M. Rush, David Sontag, Michael Collins, and Tommi S. Jaakkola. 2010. On dual decomposition and linear programming relaxations for natural language processing. In _Proceedings of EMNLP_ 

Ehsan Shareghi, Gholamreza Haffari, Trevor Cohn, and Ann Nicholson. 2015. Structured prediction of sequences and trees using infinite contexts. In _Proceedings of ECML_.
* Sogaard and Goldberg (2016) Anders Sogaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In _Proceedings of ACL_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _JMLR_.
* Surdeanu and Manning (2010) Mihai Surdeanu and Christopher D. Manning. 2010. Ensemble models for dependency parsing: Cheap and good? In _Proceedings of NAACL_.
* Teh et al. (2016) Yee Whye Teh, Alexandre H. Thiery, and Sebastian J. Vollmer. 2016. Consistency and fluctuations for stochastic gradient langevin dynamics. _JMLR_.
* Teye et al. (2018) Mattias Teye, Hossein Azizpour, and Kevin Smith. 2018. Bayesian uncertainty estimation for batch normalized deep networks. In _Proceedings of ICML_.
* Tieleman and Hinton (2012) Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. _COURSERA: Neural networks for machine learning_.
* Toutanova et al. (2016) Kristina Toutanova, Victoria Lin, Wen-tau Yih, Hoifung Poon, and Chris Quirk. 2016. Compositional learning of embeddings for relation paths in knowledge base and text. In _Proceedings of ACL_.
* Wan et al. (2013) Li Wan, Matthew D. Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In _Proceedings of ICML_.
* Welling and Teh (2011) Max Welling and Yee W Teh. 2011. Bayesian learning via stochastic gradient langevin dynamics. In _Proceedings of ICML_.
* ZareMoodi and Haffari (2018) Poorya ZareMoodi and Gholamreza Haffari. 2018. Neural machine translation for bilingually scarce scenarios: a deep multi-task learning approach. In _Proceedings of NAACL_.
* Zeman et al. (2018) Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. Conll 2018 shared task: Multilingual parsing from raw text to universal dependencies. In _Proceedings of the CoNLL_.
* Zeman et al. (2017) Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, et al. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In _Proceedings of CoNLL_.