SlowGuess/ACL_2020_OCR huggingface dataset:
Zixia Jia\*, Youmi Ma†, Jiong Cai\*, Kewei Tu\*
$^{\diamond}$ School of Information Science and Technology, ShanghaiTech University  
Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences  
University of Chinese Academy of Sciences
Shanghai Engineering Research Center of Intelligent Vision and Imaging
$\dagger$ Tokyo Institute of Technology
{jiazx, caijiong, tukw}@shanghaiitech.edu.cn
youmi.ma@nlp.c.titech.ac.jp
Semantic dependency parsing, which aims to find rich bi-lexical relationships, allows words to have multiple dependency heads, resulting in graph-structured representations. We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework. Our encoder is a discriminative neural semantic dependency parser that predicts the latent parse graph of the input sentence. Our decoder is a generative neural model that reconstructs the input sentence conditioned on the latent parse graph. Our model is arc-factored and therefore parsing and learning are both tractable. Experiments show our model achieves significant and consistent improvement over the supervised baseline.
Semantic dependency parsing (SDP) is a task aiming at discovering sentence-internal linguistic information. The focus of SDP is the identification of predicate-argument relationships for all content words inside a sentence (Oepen et al., 2014, 2015). Compared with syntactic dependencies, semantic dependencies are more general, allowing a word to be either unattached or the argument of multiple predicates. The set of semantic dependencies within a sentence form a directed acyclic graph (DAG), distinguishing SDP from syntactic dependency parsing tasks, where dependencies are usually tree-structured. Extraction of such high-level structured semantic information potentially benefits downstream NLP tasks (Reddy et al., 2017; Schuster et al., 2017).
Several supervised SDP models are proposed in the recent years by modifying syntactic dependency parsers. Their parsing mechanisms are either transition-based (Kanerva et al., 2015; Wang et al.,
2018) or graph-based (Martins and Almeida, 2014; Peng et al., 2017; Dozat and Manning, 2018; Wang et al., 2019).
One limitation of supervised SDP is that labeled SDP data resources are limited in scale and diversity. Due to the rich relationships in SDP, the annotation of semantic dependency graphs is expensive and difficult, calling for professional linguists to design rules and highly skilled annotators to annotate sentences. This limitation becomes more severe with the rise of deep learning, because neural approaches are more data-hungry and susceptible to over-fitting when lacking training data. To alleviate this limitation, we investigate semi-supervised SDP capable of learning from both labeled and unlabeled data.
While a lot of work has been done on supervised SDP, the research of unsupervised and semi-supervised SDP is still lacking. Since parsing results of semantic dependencies are DAGs without the tree-shape restriction, most existing successful unsupervised (Klein and Manning, 2004; I. Spitkovsky et al., 2010; Jiang et al., 2016; Cai et al., 2017) and semi-supervised (Koo et al., 2008; Druck et al., 2009; Suzuki et al., 2009; Corro and Titov, 2019) learning models for syntactic dependency parsing cannot be applied to SDP directly and it would be non-trivial to extend these models for SDP. There also exist several unsupervised (Poon and Domingos, 2009; Titov and Klementiev, 2011) and semi-supervised (Das and Smith, 2011; Kočisky et al., 2016; Yin et al., 2018) methods for semantic parsing, but these models are designed for semantic representations different from dependency graphs, making their adaptation to SDP difficult.
In this work, we propose an end-to-end neural semi-supervised model leveraging both labeled and unlabeled data to learn a dependency graph parser. Our model employs the framework of Conditional
6795
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6795-6805 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Random Field Autoencoder (Ammar et al., 2014), modeling the conditional reconstruction probability given the input sentence with its dependency graph as the latent variable. Our encoder is the supervised model of Dozat and Manning (2018), formulating an SDP task as labeling each arc in a directed graph with a simple neural network. Analogous to a CRF model (Sutton et al., 2012), our encoder is capable of computing the probability of a dependency graph conditioned on the input sentence. The decoder is a generative model based on recurrent neural network language model (Mikolov et al., 2010), which formulates the probability of generating the input sentence, but we take into account the information given by the dependency parse graphs when generating the input.
Our model is arc-factored, i.e., the encoding, decoding and reconstructing probabilities can all be factorized into the product of arc-specific quantities, making both learning and parsing tractable. A unified learning objective is defined that takes advantage of both labeled and unlabeled data. Besides, compared with previous semi-supervised approaches based on Variational Autoencoder (Kingma and Welling, 2013), our learning process does not involve sampling, promising better stability.
We evaluate our model on SemEval 2015 Task 18 Dataset (English) (Oepen et al., 2015) and find that our model consistently outperforms the supervised baseline. We also conduct detailed analysis showing the benefits of different amounts of unlabeled data.
Our model is based on the CRF autoencoder framework (Ammar et al., 2014) which provides a unified fashion for structured predictors to leverage both labeled and unlabeled data. A CRF autoencoder aims to produce a reconstruction of the input  $\hat{\mathbf{X}}$  from the original input  $\mathrm{X}$  with an intermediate latent structure  $\mathrm{Y}$ . It is trained to maximize the conditional reconstruction probability  $P(\hat{\mathrm{X}} = \mathrm{X}|\mathrm{X})$  with the latent variable  $\mathrm{Y}$  marginalized. Ideally, successful reconstruction implies that the latent structure captures important information of the input.
We adopt the following notations when describing our model. We represent a vector in lowercase bold, e.g., s, and use a superscript for indexing, e.g.,  $\mathbf{s}^i$  for the  $i$ -th vector. We represent a scalar in lowercase italics, e.g., s, and use a subscript for
indexing, e.g.,  $s_i$  for the  $i$ -th element of vector  $\mathbf{s}$ . An uppercase italic letter such as  $Y$  denotes a matrix. A lower case letter with a subscript pair such as  $y_{i,j}$  refers to the element of matrix  $Y$  at row  $i$  and column  $j$ . An uppercase bold letter, e.g.,  $\mathbf{U}$ , stands for a tensor. We maintain this convention when indexing, e.g.,  $\mathbf{y}_i$  is the  $i$ -th row of matrix  $Y$ .
In our model, the input is a natural language sentence consisting of a sequence of words. A sentence with  $m$  words is represented by  $\mathbf{s} = (s_0, s_1, s_2, \ldots, s_m)$ , where  $s_0$  is a special token TOP. The latent variable produced by our encoder is a dependency parse graph of the input sentence, represented as a matrix of booleans  $Y \in \{0, 1\}^{(m+1) \times (m+1)}$ , where  $y_{i,j} = 1$  indicates that there exists an dependency arc pointing from word  $s_i$  to word  $s_j$ . The reconstructed output generated by our decoder is a word sequence  $\hat{\mathbf{s}} = (\hat{s}_1, \hat{s}_2, \dots, \hat{s}_m)$ .
Our encoder with parameters  $\Theta$  computes  $P_{\Theta}(Y|\mathbf{s})$ , the probability of generating a dependency parse graph  $Y$  given a sentence  $\mathbf{s}$ . Our decoder with parameters  $\Lambda$  computes  $P_{\Lambda}(\hat{\mathbf{s}} | Y)$ , the probability of reconstructing sentence  $\hat{\mathbf{s}}$  conditioned on the parse graph  $Y$ . The encoder and decoder in combination specify the following conditional distribution.
$$
P _ {\Theta , \Lambda} (\hat {\mathbf {s}}, Y | \mathbf {s}) = P _ {\Theta} (Y | \mathbf {s}) P _ {\Lambda} (\hat {\mathbf {s}} | Y)
$$
To compute the conditional probability  $P(\hat{\mathbf{s}}|\mathbf{s})$ , we sum out the latent variable  $Y$ .
$$
P _ {\Theta , \Lambda} (\hat {\mathbf {s}} | \mathbf {s}) = \sum_ {Y \in \mathcal {Y}} P _ {\Theta , \Lambda} (\hat {\mathbf {s}}, Y | \mathbf {s})
$$
where  $\mathcal{V}$  is the set of all possible dependency parse graphs of s. During training, we set  $\hat{\mathbf{s}} = \mathbf{s}$  and maximize the conditional reconstruction probability  $P(\hat{\mathbf{s}}|\mathbf{s})$ .
Note that throughout our model, we only consider dependency arc predictions (i.e., whether an arc exists between each word pair). Arc-labels will be learned separately as described in Section 3. We leave the incorporation of arc-label prediction in our model for future work.
Our encoder can be any arc-factored discriminative SDP model. Here we adopt the model of Dozat and Manning (2018), one of the best-performing SDP models, which formulates the semantic dependency parsing task as independently labeling each arc in
6796
Figure 1: Illustration of the encoder, following the design of Dozat and Manning (2018).
a directed complete graph. To predict whether or not a directed arc  $(s_i, s_j)$  exists, the model computes contextualized representations of  $s_i$  and  $s_j$  and feeds them into a binary classifier.
The architecture of our encoder is shown in Figure 1. Word, part-of-speech tag (for short, POS tag), and lemma embeddings<sup>1</sup> of each word in the input sentence are concatenated and fed into a multi-layer bi-directional LSTM to get a contextualized representation of the word.
$$
\begin{array}{l} \mathbf {x} _ {i} = \mathbf {e} _ {i} ^ {\left(\text {w o r d}\right)} \oplus \mathbf {e} _ {i} ^ {\left(\text {t a g}\right)} \oplus \mathbf {e} _ {i} ^ {\left(\text {l e m m a}\right)} \tag {1} \\ R = \operatorname {B i L S T M} (X) \\ \end{array}
$$
where  $\mathbf{e}_i^{(\mathrm{word})},\mathbf{e}_i^{(\mathrm{tag})}$  and  $\mathbf{e}_i^{(\mathrm{lemma})}$  are notations for the word, POS tag and lemma embedding respectively, concatenated  $(\oplus)$  to form an embedding  $\mathbf{x}_i$  for word  $s_i$ . Stacking  $\mathbf{x}_i$  for  $i = 0,1,\ldots ,m$  forms matrix  $X$ .
The contextualized word representation is then fed into two single-layer feedforward neural networks (FNN) with different parameters to produce two vectors: one for the representation of the word as a dependency head and the other for the representation of the word as a dependent. They are denoted as  $\mathbf{h}_i^{(\mathrm{head})}$  and  $\mathbf{h}_i^{(\mathrm{dep})}$  respectively.
$$
\begin{array}{l} \mathbf {h} _ {i} ^ {\mathrm {(h e a d)}} = \mathbf {F N N} ^ {\mathrm {(e n c - h e a d)}} (\mathbf {r} _ {i}) \\ \mathbf {h} _ {i} ^ {\mathrm {(d e p)}} = \mathbf {F N N} ^ {\mathrm {(e n c - d e p)}} (\mathbf {r} _ {i}) \\ \end{array}
$$
Finally, a biaffine function is applied to every arc between word pairs  $(s_i,s_j)$  to obtain an arcexistence score  $\psi_{i,j}$
$$
\psi_ {i, j} = \mathbf {h} _ {i} ^ {\mathrm {(h e a d) \top}} W \mathbf {h} _ {j} ^ {\mathrm {(d e p)}} + b
$$
where  $W$  is a square matrix of size  $d \times d$  ( $d$  is the size of vector  $\mathbf{h}_i^{(\mathrm{head})}$  and  $\mathbf{h}_j^{(\mathrm{dep})}$ ), and  $b$  is a scalar.
The likelihood of every arc's presence given a sentence,  $P(y_{i,j} = 1|\mathbf{s})$ , can be computed by applying a sigmoid function on score  $\psi_{i,j}$ . The arc-absence probability  $P(y_{i,j} = 0|\mathbf{s})$  is evidently  $1 - P(y_{i,j} = 1|\mathbf{s})$ .
To conclude, the probability of producing a dependency parse graph  $Y$  from the encoder given an input sentence  $s$  can be computed as below.
$$
P (Y | \mathbf {s}) = \prod_ {i, j} P (y _ {i, j} | \mathbf {s})
$$
Our generative decoder is based on recurrent neural network language models (Mikolov et al., 2010), but we take dependency relationships into account during reconstruction. Our inspiration sources from the decoder with a Graph Convolutional Network (GCN) used by Corro and Titov (2019) to incorporate tree-structured syntactic dependencies when generating sentences, but our decoder differs significantly from theirs in that ours handles parse graphs and is arc-factored.
As mentioned above, semantic dependency parsing allows a word to have multiple dependency heads. If we generate a word conditioned on multiple heads, then it becomes difficult if not impossible to make the decoder arc-factored and hence we may have to enumerate all parse graphs during parsing and learning, which is intractable. Instead, we propose to generate a word for multiple times, each time conditioned on a different head, which leads to a fully arc-factored generative decoder and hence tractable parsing and learning. Specifically, we split dependency graph  $Y$  of a sentence  $\mathbf{s} = (s_0, s_1, \ldots, s_m)$  with  $m$  words and a TOP token into  $m + 1$  parts:
$$
Y = \left[ \mathbf {y} _ {0}; \mathbf {y} _ {1}; \mathbf {y} _ {2}; \dots ; \mathbf {y} _ {m} \right]
$$
Each  $\mathbf{y}_i$  is the  $i$ -th row of  $Y$ , representing a subgraph where arcs are rooted at the  $i$ -th word of the sentence  $\mathbf{s}$ . Mathematically, we have  $\mathbf{y}_i = \{y_{i,j} | j \in (1, 2, \dots, m)\}$ .
We then generate  $m + 1$  sentences  $(\hat{\mathbf{s}}^0,\hat{\mathbf{s}}^1,\hat{\mathbf{s}}^2,\dots ,\hat{\mathbf{s}}^m)$  using  $m + 1$  neural generators. The generation of sentence  $\hat{\mathbf{s}}^i$  is guided by the  $i$ -th sub-graph  $\mathbf{y}_i$ . Each generator is a left-to-right LSTM language model and computes  $P_{\Lambda}(\hat{s}_i^k |\hat{\mathbf{s}}_{0:i - 1}^k,y_{k,i})$ , the probability of generating
<sup>1</sup> The latest experimental results in Dozat and Manning (2018) show that using lemma embedding improves performance even further while including character-level word embedding produces little effect. Thus unless stated otherwise, our model makes use of lemma embeddings by default.
6797
Figure 2: Illustration of the decoder generating  $\hat{\mathbf{s}}^k$  from the  $k$ -th neural generator, guided by sub-graph  $\mathbf{y}_k$ . Dashed arcs at the bottom represent dependency arcs. A cross over arc  $(s_k, s_4)$  indicates the absence of this arc.
each word conditioned on its preceding words and whether  $\mathbf{y}_k$  contains a dependency arc to the word. We share parameters among all the  $m + 1$  generators.
Figure 2 shows an example for computing the generative probability of  $\hat{\mathbf{s}}^k$  by the  $k$ -th generator  $(k \in \{0,1,\dots,m\})$  that incorporates the information of the  $k$ -th sub-graph  $\mathbf{y}_k$ . Recall that  $\mathbf{y}_k$  contains only dependencies rooted at  $s_k$ . Below we describe how to compute the generative probability of each word  $\hat{s}_i^k$  with and without the dependency arc  $(s_k,s_i)$  respectively.
Generative probability with a dependency Suppose there is a dependency arc from  $s_k$  to  $s_i$ , we need to compute the generative probability  $P_{\Lambda}(\hat{s}_i^k |\hat{s}_{0:i - 1}^k,y_{k,i} = 1)$ . The LSTM in the  $k$ -th generator takes the embedding of the previous word  $s_{i - 1}$  computed through Eq.1 as its input and outputs the hidden state  $\mathbf{g}_{i - 1}$ , which is fed into an FNN to produce a representation  $\mathbf{m}_{i - 1}^{(\mathrm{pre})}$ . Meanwhile, the embedding of the  $k$ -th word (also computed through Eq.1) is fed into another FNN to get its representation  $\mathbf{m}_k^{(\mathrm{head})}$  as a dependency head.
$$
G = \operatorname {L S T M} (X)
$$
$$
\mathbf {m} _ {i - 1} ^ {\left(\text {p r e}\right)} = \mathbf {F N N} ^ {\left(\text {d e c - p r e}\right)} \left(\mathbf {g} _ {i - 1}\right) \tag {2}
$$
$$
\mathbf {m} _ {k} ^ {\text {(h e a d)}} = \mathbf {F N N} ^ {\text {(d e c - h e a d)}} (\mathbf {x} _ {k}) \tag {3}
$$
$\mathbf{m}_k^{(\mathrm{head})}$  and  $\mathbf{m}_{i - 1}^{(\mathrm{pre})}$  are fed into a bilinear function to obtain a vocabulary-size score vector  $\phi_i^k$
$$
\phi_ {i} ^ {k} = \mathbf {m} _ {k} ^ {(\text {h e a d}) \top} \mathbf {U m} _ {i - 1} ^ {(\text {p r e})} \tag {4}
$$
Here,  $\mathbf{U}$  is a tensor of size  $d\times V\times d$ , where  $V$  is the vocabulary size and  $d$  is the size of vector  $\mathbf{m}_k^{(\mathrm{head})}$  and  $\mathbf{m}_{i - 1}^{\mathrm{(pre)}}$ . To conserve parameters, the tensor  $\mathbf{U}$  is diagonal (i.e.,  $u_{i,k,j} = 0$  wherever  $i\neq j$ ). A softmax function can then be applied to  $\phi_i^k$ , from which we pick the generative probability of  $\hat{s}_i^k$ .
Generative probability without a dependency Suppose there is no dependency arc from  $s_k$  to  $s_i$ . In this case, reconstruction of  $\hat{s}_i^k$  resembles a normal recurrent neural network language model. The representation  $\mathbf{m}_{i-1}^{(\mathrm{pre})}$  from Eq.2 is fed into a fully connected layer to get  $\bar{\phi}_i^k$ , a vector of vocabulary size containing generative scores of all the words.
$$
\bar {\phi} _ {i} ^ {k} = \mathbf {F C} \left(\mathbf {m} _ {i - 1} ^ {(\mathrm {p r e})}\right) \tag {5}
$$
The generative probability  $P_{\Lambda}(\hat{s}_i^k |\hat{\mathbf{s}}_{0:i - 1}^k,y_{k,i} = 0)$  can then be computed by applying a softmax function on  $\bar{\phi}_i^k$  and selecting the corresponding probability of  $\hat{s}_i^k$ . Since we simply reconstruct word  $s_i$  without considering the dependency arc information, this probability is exactly the same in the  $m + 1$  generators and only needs to be computed once.
To conclude the overall design of our decoder, it is worth noting that in  $m + 1$  generation processes, parameters among all LSTMs are shared, as well as those among all  $\mathrm{FNNs}^2$  and FCs. Still, embeddings in Eq.1 are shared among both encoder and decoder.
With  $P(\hat{s}_i^k |\hat{\mathbf{s}}_{0:i - 1}^k,y_{k,i})$  computed for  $i = 1,\dots ,m,k = 0,1,\dots ,m$ , the probability of generating  $\hat{\mathbf{s}}^0,\hat{\mathbf{s}}^1,\hat{\mathbf{s}}^2,\dots ,\hat{\mathbf{s}}^m$  from dependency graph  $Y$  can be computed through:
$$
\begin{array}{l} P _ {\Lambda} \left(\hat {\mathbf {s}} ^ {0}, \dots , \hat {\mathbf {s}} ^ {m} | Y\right) = \prod_ {k = 0} ^ {m} P _ {\Lambda} \left(\hat {\mathbf {s}} ^ {k} | \mathbf {y} _ {k}\right) \\ = \prod_ {k = 0} ^ {m} \prod_ {i = 1} ^ {m} P _ {\Lambda} \left(\hat {s} _ {i} ^ {k} | \hat {\mathbf {s}} _ {0: i - 1} ^ {k}, y _ {k, i}\right) \\ \end{array}
$$
In our model, we are only interested in the case where all the  $m + 1$  sentences are the same. In addition, to balance the influence of the encoder and the decoder, we take the geometric mean of the  $m + 1$  probabilities. The final decoding probability is defined as follows.
$$
P _ {\Lambda} (\hat {\mathbf {s}} | Y) := \prod_ {i = 1} ^ {m} \prod_ {k = 0} ^ {m} \sqrt [ m + 1 ]{P _ {\Lambda} \left(\hat {s} _ {i} ^ {k} \mid \hat {\mathbf {s}} _ {0 : i - 1} ^ {k} , y _ {k , i}\right)}
$$
$^{2}\mathbf{FNN}^{(\mathrm{dec - pre})}$  and  $\mathbf{FNN}^{(\mathrm{dec - head})}$  never share parameters between each other, since their usages are different.
6798
Note that this is not a properly normalized probability distribution, but in practice we find it sufficient for semi-supervised SDP.
2.3 Parsing
Given parameters  $\{\Theta, \Lambda\}$  of our encoder and decoder, we can parse a sentence  $s$  by finding a  $Y \in \mathcal{Y}(s)$  which maximizes probability  $P(\hat{s} = s, Y|s)$ , where  $\mathcal{Y}(s)$  is the set of all parse graphs of sentence  $s$ .
$$
\begin{array}{l} Y ^ {*} = \underset {Y \in \mathcal {Y} (\mathbf {s})} {\arg \max } \log P _ {\Theta , \Lambda} (\hat {\mathbf {s}}, Y | \mathbf {s}) \tag {6} \\ = \operatorname *{arg  max}_{Y\in \mathcal{Y}(\mathbf{s})}\log P_{\Theta}(Y|\mathbf{s})P_{\Lambda}(\hat{\mathbf{s}} |Y) \\ = \underset {Y \in \mathcal {Y} (\mathbf {s})} {\arg \max } \sum_ {i, j} \left(\log P _ {\Theta} \left(y _ {i, j} \mid \mathbf {s}\right) \right. \\ \left. + \frac {1}{m + 1} \log P _ {\Lambda} \left(\hat {s} _ {j} \mid \hat {\mathbf {s}} _ {0: j - 1}, y _ {i, j}\right)\right) \\ \end{array}
$$
Since the probability is arc-factored, we can determine the existence of each dependency arc independently by picking the value of  $y_{i,j}$  that maximizes the corresponding term. The time complexity of our parsing algorithm is  $O(m^2)$  for a sentence with  $m$  words.
Since we want to train our model in a semi-supervised manner, we design loss functions for labeled and unlabeled data respectively. For each training sentence  $\mathbf{s}$ , the overall loss function is defined as a combination of supervised loss  $\mathcal{L}_l$  and unsupervised loss  $\mathcal{L}_u$ .
$$
\mathcal {L} (\mathbf {s}) = \iota (\mathbf {s}) * \mathcal {L} _ {l} (\mathbf {s}) + (1 - \iota (\mathbf {s})) * \rho \mathcal {L} _ {u} (\mathbf {s}) \tag {7}
$$
where an indicator  $\iota(\mathbf{s}) \in \{0,1\}$  specifies whether training sentence  $\mathbf{s}$  is labeled or not and a tunable constant  $\rho$  balances the two losses.
Supervised Loss For any labeled sentence  $(\mathbf{s},Y^{*})$  , where s stands for a sentence and  $Y^{*}$  stands for a gold parse graph, we can compute the discriminative loss.
$$
\mathcal {L} _ {l} (\mathbf {s}) = - \log P _ {\Theta , \Lambda} (\hat {\mathbf {s}} = \mathbf {s}, Y ^ {*} | \mathbf {s}) \tag {8}
$$
Following the derivation of Eq.6, we have:
$$
\begin{array}{l} \log P _ {\Theta , \Lambda} (\hat {\mathbf {s}}, Y ^ {*} | \mathbf {s}) = \sum_ {i, j} \Big (\log P _ {\Theta} (y _ {i, j} ^ {*} | \mathbf {s}) \\ \left. + \frac {1}{m + 1} \log P _ {\Lambda} \left(\hat {s} _ {j} \mid \hat {\mathbf {s}} _ {0: j - 1}, y _ {i, j} ^ {*}\right)\right) \\ \end{array}
$$
Usage	Source	Sentences	Tokens
train	WSJ Sec.00-20	35,656	802,717
test (id)	WSJ Sec.21	1,410	31,948
test (ood)	Brown	1,849	31,583
Table 1: The sources and scale of the SDP 2014 & 2015 (English) dataset. We extract WSJ Section 20 (1,692 sentences) from the train set for development purpose. id stands for in-domain testing, while ood stands for out-of-domain testing.
Gold parses also provide a label for each dependency. We follow Dozat and Manning (2018) and model dependency labels with a purely supervised module on top of the BiLSTM layer of the encoder. Its parameters are learned by optimizing a cross-entropy loss function.
Unsupervised Loss For any unlabeled sentence  $s$ , we maximize the conditional reconstruction probability  $P(\hat{\mathbf{s}} = \mathbf{s}|\mathbf{s})$ . The unsupervised loss is:
$$
\begin{array}{l} \mathcal {L} _ {u} (\mathbf {s}) = - \log P _ {\Theta , \Lambda} (\hat {\mathbf {s}} | \mathbf {s}) \tag {9} \\ = - \log \sum_ {Y \in \mathcal {Y} (\mathbf {s})} P _ {\Theta , \Lambda} (Y, \hat {\mathbf {s}} | \mathbf {s}) \\ = - \log \sum_ {Y \in \mathcal {Y} (\mathbf {s})} P _ {\Theta} (Y | \mathbf {s}) P _ {\Lambda} (\hat {\mathbf {s}} | Y) \\ = - \sum_ {i, j} \log \sum_ {y _ {i, j} \in \{0, 1 \}} \left(P _ {\Theta} \left(y _ {i, j} | \mathbf {s}\right) \right. \\ \times \sqrt [ m + 1 ]{P _ {\Lambda} (\hat {s} _ {j} | \hat {\mathbf {s}} _ {0 : j - 1} , y _ {i , j})}) \\ \end{array}
$$
Derivations of Eq.9 are provided in the Appendix A.
Given a dataset containing both labeled and unlabeled sentences, our model can be trained end-to-end by optimizing the loss function Eq.7 over the combined dataset using any gradient based method.
Dataset We examine the performance of our model on the English corpus of the SDP 2014 & 2015: Broad Coverage Semantic Dependency Parsing dataset (Oepen et al., 2015). The corpus is composed of three distinct and parallel semantic dependency annotations (DM, PAS, PSD) of Sections 00-21 of the WSJ Corpus, as well as a balanced sample of twenty files from the Brown Corpus. The scale of this dataset is shown in Table 1.
6799