SlowGuess/ACL_2020_OCR huggingface dataset:
Hidden Layer	Hidden Sizes
Word/GloVe/POS/Lemma/Char	100
GloVe Linear	125
Encoder BiLSTM	3*600
Encoder FNN(head)	1*600
Encoder FNN(dep)	1*600
Decoder UniLSTM	1*600
Decoder FNN(head)	1*400
Decoder FNN(pre)	1*400
Dropouts	Dropout Prob.
Word/GloVe/POS/Lemma	20%
Encoder/Decoder FNN	25 %
BiLSTM (FF/recur)	45%/25%
Optimizer & Loss	Value
Adam β1	0
Adam β2	0.95
Learning rate	1e-3
L2 regularization	3e-9
Table 2: Summary of hyper-parameters.
We evaluate the performance of models through two metrics: Unlabeled F1 score (UF1) and Labeled F1 score (LF1). UF1 measures the accuracy of the binary classification of arc existence, while LF1 measures the correctness of each arc-label as well. Unless stated otherwise, we report scores averaged over three runs.
Network Configuration For our encoder, we adopt the hyper-parameters of Dozat and Manning (2018). Following Dozat and Manning (2018), we concatenate pre-trained 100-dimensional GloVe embeddings (Pennington et al., 2014) linearly transformed to 125-dimension into our input word embeddings. Words or lemmas whose occurrences are less than 7 times within the training set are treated as UKN.
For our decoder, we set the number of layer(s) of uni-directional LSTM to 1, whose recurrent hidden size is 600. For  $\mathbf{FNN}^{(\mathrm{dec - head})}$  and  $\mathbf{FNN}^{(\mathrm{dec - pre})}$ , the output sizes are both 400, activated by a tanh(·) function.
Learning Our loss function (Eq.7) is optimized by the Adam+AMSGrad optimizer (Reddi et al., 2018), with hyper-parameters  $\beta_{1},\beta_{2}$  kept the same as those of Dozat and Manning (2018). The interpolation constant  $\rho$  is tuned with the size of unlabeled data. A detailed table of hyper-parameters is provided in Table 2. The training time for one batch with our autoencoder is 2-3 times of that of Dozat and Manning (2018) because of the extra decoder.
(a) UF1 and LF1 for in-domain tests on DM.
Figure 3: Results with fixed amount of labeled data and varying amount of unlabeled data. +10U represents using  $10\%$  unlabeled data, and so on. Numbers on Y-axes represent F1 scores. Dashed horizontal lines are results of the supervised baseline (Dozat and Manning, 2018) trained on labeled data only. Solid lines are our results.
(b) UF1 and LF1 for out-of-domain tests on DM.
4.2 Varying Size of Unlabeled Data
In our first experiment (with the DM annotations only), we fix the amount of labeled data and continuously incorporate more unlabeled data into the training set.
Specifically, we randomly sample  $10\%$  of the whole dataset as labeled data. Unlabeled data are then sampled from the remaining part (with their gold parses removed), with a proportion increasing from  $0\%$  to  $90\%$  of the complete dataset.
For unlabeled data, we find that long sentences do not help in improving F1 scores and therefore in this and all the subsequent experiments we remove unlabeled sentences longer than 20 to reduce the running time and memory usage.
Experimental results are visualized in Figure 3. It is observed that in the purely supervised setting (i.e.,  $+0\%$  unlabeled data), our model already outperforms the baseline (Dozat and Manning, 2018). Since our encoder is exactly the baseline model, this shows the benefit of adding the decoder for joint learning and parsing even in the supervised setting. With an increasing size of unlabeled data, we can see the increase in performance of our model, especially when evaluated on out-of-domain data, suggesting the benefit of semi-supervised learning with our model.
4.3 Varying Proportion of Unlabeled Data
In our second experiment (again with the DM annotations), we use the full training set and vary the proportion of labeled and unlabeled data.
Experimental results are shown in Table 3. Our semi-supervised model shows the largest advantage over the supervised models with the 0.1:9.9 proportion (which contains only 339 labeled sentences),
Models	Labeled:Unlabeled
0.1:9.9	1:9	3:7	5:5	10:0
UF1	LF1	UF1	LF1	UF1	LF1	UF1	LF1	UF1	LF1
id	D&M	75.21	70.70	88.32	86.60	91.65	90.52	92.81	91.90	94.11	93.38
Ours-Sup	75.52	70.59	88.58	86.74	91.88	90.73	92.99	92.05	94.30	93.55
Ours-Semi	76.73	72.16	88.98	87.11	92.04	90.92	93.02	92.07	-	-
ood	D&M	70.51	65.63	83.15	80.87	86.91	85.17	88.35	86.93	90.01	88.87
Ours-Sup	70.53	65.48	83.33	80.92	87.16	85.45	88.63	87.24	90.22	89.05
Ours-Semi	72.18	67.30	83.93	81.48	87.43	85.70	88.67	87.28	-	-
Table 3: Experimental results with varying proportions of labeled and unlabeled data. D&M stands for the supervised model of Dozat and Manning (2018) trained on labeled data only. Ours-Sup stands for our model trained on labeled data only. Ours-Semi stands for our model trained on both labeled and unlabeled data. All scores in this table are averaged over 10 runs. We do paired permutation test  $(p < 0.05)$ . Two different scores being both boldfaced means that they are not significantly different.
indicating the strength of our mode in low resource setting.
With the increased proportion of labeled data, the performance of all the models goes up, but the advantage of our semi-supervised model diminishes. This is consistent with the tendency of many semi-supervised approaches to work well when given small labeled data but have diminishing effectiveness when adding more labeled data.
Another worth-noting observation is that the superiority of our semi-supervised model is much stronger on the out-of-domain tests, which suggests good generalizability of our semi-supervised model.
4.4 On All Representations
In the previous two experiments, we evaluate our model on the DM representation. Here we evaluate our model on all the three representations: DM, PAS and PSD. We slightly tune the hyperparameters based on the optimal values from the previous experiments of the DM representation. We use  $10\%$  of the sentences as labeled data and the rest  $90\%$  of the sentences as unlabeled data. For the completeness of our experiment, we follow Dozat and Manning (2018) and examine four different word representations: basic (i.e., using only word and POS tag embeddings), +Lemma (i.e., using word, POS tag and lemma embeddings), +Char (i.e., using word, POS tag and character embeddings) and +Lemma+Char (i.e. using word, POS tag, lemma and character embeddings).
Table 4 shows the experimental results of  $+\mathbf{Lemma}$ , the default word representation. The results of the other word representations show very
similar trends (see the Table 7 in Appendix B). We observe significant improvement of our semi-supervised model over the two supervised baselines on both DM and PSD representations. However, it is surprising to find that on the PAS representation, our semi-supervised model exhibits little advantage over its supervised counterpart. One possible explanation, as Dozat and Manning (2018) also noted, is that PAS is the easiest of the three representations (as can be seen by comparing the scores of the three representations in Table 4) and our supervised model may already reach the performance ceiling.
4.5 Analysis of Our Decoder
We empirically study alternative structures of our decoder. In the first variant, we remove the LSTM layer of our decoder, so each word  $s_i$  is generated without access to the generation history before  $s_{i-1}$ . In the second variant, we replace the bilinear function in Eq.4 with a fully connected layer that takes as input either the concatenation or the summation of  $\mathbf{m}_k^{\mathrm{(head)}}$  and  $\mathbf{m}_{i-1}^{\mathrm{(pre)}}$ . All the other settings are the same as in Section 4.4 on the DM annotation. Experimental results are shown in Table 5. We can see that these alternatives lead to worse scores, which verifies the effectiveness of our decoder design.
4.6 Stability of Our Model
To test the stability of our model, we repeat the experiment of Section 4.4 on the DM annotation for three times (without tuning hyper-parameters), each time with respect to different labeled data sampled from the training dataset. Table 6 shows the results. We observe consistent advantage of our
6801
Models	DM	PAS	PSD	Avg
UF1	LF1	UF1	LF1	UF1	LF1	UF1	LF1
id	D&M	88.32	86.60	91.89	90.57	88.17	73.42	89.46	83.53
Ours-Sup	88.58	86.74	92.14	90.91	88.49	73.34	89.74	83.66
Ours-Semi	88.98	87.11	92.07	90.84	88.62	73.68	89.89	83.88
ood	D&M	83.15	80.87	88.34	86.32	85.10	71.30	85.53	79.50
Ours-Sup	83.33	80.92	88.57	86.68	85.09	71.11	85.66	79.57
Ours-Semi	83.93	81.48	88.61	86.68	85.30	71.46	85.95	79.87
Table 4: Experimental results on all the three representations. All scores in this table are averaged over 10 runs. We do paired permutation test  $(p < 0.05)$ . Two different scores being both boldfaced means that they are not significantly different.
ID	OOD
UF1	LF1	UF1	LF1
Default	88.95	87.07	83.94	81.55
-LSTM	88.72	86.88	83.58	81.20
Concat	88.75	86.77	83.72	81.26
Sum	86.13	83.90	79.08	76.34
Table 5: Experimental results on different structures of our decoder. Default is our default semi-supervised model, -LSTM means removing the LSTM layer of our decoder, Concat stands for the concatenation setting, and Sum stands for the summation setting, as stated in Section 4.5.
model over the baseline on all the three datasets.
Work on unsupervised or semi-supervised dependency parsing, to the best of our knowledge, is dominated by tree-structured parsing (Koo et al., 2008; Druck et al., 2009; Suzuki et al., 2009). Recently, Corro and Titov (2019) introduced an approximate inference method with a Variational Autoencoder (Kingma et al., 2014) for semi-supervised syntactic dependency parsing. Our decoder is inspired by their work, but differs from theirs in that our decoder handles parse graphs and is arc-factored. Cai et al. (2017) used the framework of CRF Autoencoder (Ammar et al., 2014) to perform unsupervised syntactic dependency parsing. The same framework has been used by Zhang et al. (2017) for semi-supervised sequence labeling. Our work also adopts the CRF Autoencoder framework, but with both the encoder and the decoder redesigned for semantic dependency parsing.
Existing unsupervised and semi-supervised approaches to semantic parsing focused on semantic representations different from dependency graphs,
e.g., general-purpose logic forms (Sondheimer and Nebel, 1986) and formal meaning representations (Bordes et al., 2012). Poon and Domingos (2009) presented the first unsupervised semantic parser to transform dependency trees into quasi-logical forms with Markov logic. Following this work, Titov and Klementiev (2011) proposed a non-parametric Bayesian model for unsupervised semantic parsing using hierarchical PitmanYor process (Teh, 2006). Das and Smith (2011) described a semi-supervised approach to framesemantic parsing. Kočisky et al. (2016) proposed a semi-supervised semantic parsing approach making use of unpaired logical forms with sentence being unobserved. Recently, Yin et al. (2018) proposed a variational autoencoding model for semi-supervised semantic parsing of tree-structured semantic representations. Take Yin et al. (2018) for example. To extend their approach for SDP, one needs to design a different transition system for their encoder for graph parsing and design a graph linearization method for their sequence-to-sequence decoder. In addition, SDP-specific constraints (e.g., the graph structure contains exactly the same set of words as the sentence) shall be incorporated into their model. Therefore, previous semi-supervised semantic parsing models cannot be applied to SDP directly and modifying them for SDP is non-trivial. We leave for future work such modification and extension of previous semi-supervised semantic parsing approaches to SDP.
In this work, we proposed a semi-supervised learning model for semantic dependency parsing using CRF Autoencoders. Our model is composed of a discriminative neural encoder producing a dependency graph conditioned on an input sentence, and
6802
Models	Data1	Data2	Data3	Avg
UF1	LF1	UF1	LF1	UF1	LF1	UF1	LF1
id	D&M	88.25	86.55	88.70	87.09	88.49	86.85	88.48	86.83
Ours-Sup	88.68	86.84	88.79	86.96	88.71	86.97	88.73	86.92
Ours-Semi	88.95	87.07	89.24	87.45	88.97	87.19	89.05	87.24
ood	D&M	83.12	80.89	83.30	81.01	83.62	81.26	83.35	81.05
Ours-Sup	83.36	80.98	83.46	81.05	84.00	81.62	83.60	81.22
Ours-Semi	83.94	81.55	83.87	81.51	84.11	81.68	83.97	81.58
Table 6: Experimental results on three randomly sampled datasets.
a generative neural decoder for input reconstruction based on the dependency graph. The model works in an arc-factored fashion, promising end-to-end learning and efficient parsing.
We evaluated our model under both full-supervision settings and semi-supervision settings. Our model outperforms the baseline on multiple target representations. By adding unlabeled data, our model exhibits further performance improvements. In particular, our semi-supervised model performs well in the low resource setting and on the out-of-domain test set. This points to future directions of applying our model to low-resource languages and cross-domain settings. Our code is publicly available at https://github.com/JZXXX/Semi-SDP.
This work was supported by the National Natural Science Foundation of China (61976139).
Waleed Ammar, Chris Dyer, and Noah A. Smith. 2014. Conditional Random Field Autoencoders for Unsupervised Structured Prediction. Advances in Neural Information Processing Systems, 4:3311-3319.
Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2012. Joint Learning of Words and Meaning Representations for Open-text Semantic Parsing. In Artificial Intelligence and Statistics, pages 127-135.
Jiong Cai, Yong Jiang, and Kewei Tu. 2017. Crf autoencoder for unsupervised dependency parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Caio Corro and Ivan Titov. 2019. Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder. In International Conference on Learning Representations.
Dipanjan Das and Noah A Smith. 2011. Semi-Supervised Frame-Semantic Parsing for Unknown
Predicates. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1435-1444. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2018. Simpler but More Accurate Semantic Dependency Parsing. In ACL.
Gregory Druck, Gideon S. Mann, and Andrew McCallum. 2009. Semi-Supervised Learning of Dependency Parsers using Generalized Expectation Criteria. In IJCNLP-ACL.
Valentin I. Spitkovsky, Hiyan Alshawi, Daniel Jurafsky, and Christoper Manning. 2010. Viterbi Training Improves Unsupervised Dependency Parsing. CoNLL 2010 - Fourteenth Conference on Computational Natural Language Learning, Proceedings of the Conference, pages 9-17.
Yong Jiang, Wenjuan Han, and Kewei Tu. 2016. Unsupervised Neural Dependency Parsing. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 763-771.
Jenna Kanerva, Juhani Luotolahti, and Filip Ginter. 2015. Turku: Semantic Dependency Parsing as A Sequence Classification. In Proceedings of the 9th International Workshop on Semantic Evaluation (Se-mEval 2015), pages 965-969.
Diederik Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. ICLR.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-Supervised Learning with Deep Generative Models. In Advances in neural information processing systems, pages 3581-3589.
Dan Klein and Christopher D Manning. 2004. Corpus-based Induction of Syntactic Structure: Models of Dependency and Constituency. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 478. Association for Computational Linguistics.
Tomáš Kocisky, Gábor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and
6803
Karl Moritz Hermann. 2016. Semantic Parsing with Semi-Supervised Sequential Autoencoders. In EMNLP.
Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple Semi-Supervised Dependency Parsing. In In Proceedings of ACL-08.
André FT Martins and Mariana SC Almeida. 2014. Priberam: A Turbo Semantic Parser with Second Order Features. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 471-476.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent Neural Network Based Language Model. In Eleventh annual conference of the international speech communication association.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinkov, Dan Flickinger, Jan Haji, and Zdeka Ureov. 2015. SemEval 2015 Task 18: Broad-Coverage Semantic Dependency Parsing. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 915-926, Denver, CO, USA.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Dan Flickinger, Jan Hajic, Angelina Ivanova, and Yi Zhang. 2014. Semeval 2014 task 8: Broad-Coverage Semantic Dependency Parsing. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 63-72.
Hao Peng, Sam Thomson, and Noah A Smith. 2017. Deep Multitask Learning for Semantic Dependency Parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Hoifung Poon and Pedro Domingos. 2009. Unsupervised Semantic Parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 1-10. Association for Computational Linguistics.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the Convergence of Adam and Beyond. In International Conference on Learning Representations.
Siva Reddy, Oscar Tackstrom, Slav Petrov, Mark Steedman, and Mirella Lapata. 2017. Universal Semantic Parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Sebastian Schuster, Éric Villemonte de La Clergerie, Marie Candito, Benoit Sagot, Christopher Manning, and Djamé Seddah. 2017. Paris and Stanford at EPE
2017: Downstream Evaluation of Graph-based Dependency Representations. In EPE 2017-The First Shared Task on Extrinsic Parser Evaluation, pages 47-59.
Norman K Sondheimer and Bernhard Nebel. 1986. A Logical-Form and Knowledge-Base Design for Natural Language Generation. In Proceedings of the workshop on Strategic computing natural language, pages 231-241. Association for Computational Linguistics.
Charles Sutton, Andrew McCallum, et al. 2012. An Introduction to Conditional Random Fields. Foundations and Trends in Machine Learning, 4(4):267-373.
Jun Suzuki, Hideki Isozaki, Xavier Carreras, and Michael Collins. 2009. An Empirical Study of Semi-Supervised Structured Conditional Models for Dependency Parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pages 551-560. Association for Computational Linguistics.
Yee Whye Teh. 2006. A Hierarchical Bayesian Language Model Based on Pitman-Yor Processes. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 985-992. Association for Computational Linguistics.
Ivan Titov and Alexandre Klementiev. 2011. A Bayesian Model for Unsupervised Semantic Parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1445-1455. Association for Computational Linguistics.
Xinyu Wang, Jingxian Huang, and Kewei Tu. 2019. Second-order semantic dependency parsing with end-to-end neural networks. ACL.
Yuxuan Wang, Wanxiang Che, Jiang Guo, and Ting Liu. 2018. A Neural Transition-Based Approach for Semantic Dependency Graph Parsing. In Thirty-Second AAAI Conference on Artificial Intelligence.
Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. 2018. StructVAE: Tree-Structured Latent Variable Models for Semi-Supervised Semantic Parsing. In ACL.
Xiao Zhang, Yong Jiang, Hao Peng, Kewei Tu, and Dan Goldwasser. 2017. Semi-Supervised Structured Prediction with Neural CRF Autoencoder. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1701-1711.
6804
Models	DM	PAS	PSD	Avg
UF1	LF1	UF1	LF1	UF1	LF1	UF1	LF1
id, basic	D&M	87.48	85.36	91.82	90.46	87.9	72.74	89.07	82.86
Ours-Sup	87.57	85.4	91.92	90.61	88.00	72.70	89.16	82.90
Ours-Semi	88.27	85.96	92.16	90.85	88.27	73.06	89.57	83.23
ood, basic	D&M	82.32	79.6	88.30	86.29	84.35	70.18	84.99	78.69
Ours-Sup	82.34	79.59	88.51	86.56	84.39	70.25	85.08	78.80
Ours-Semi	83.19	80.29	88.65	86.71	84.56	70.45	85.47	79.15
id, +Char	D&M	87.66	85.68	91.93	90.58	87.85	72.74	89.15	83.00
Ours-Sup	87.84	85.71	92.23	90.99	88.25	72.77	89.44	83.16
Ours-Semi	88.21	85.99	92.20	90.93	88.27	73.26	89.56	83.39
ood, +Char	D&M	82.50	80.07	88.27	86.33	84.51	70.48	85.09	78.96
Ours-Sup	82.60	79.99	88.71	86.84	84.59	70.46	85.30	79.09
Ours-Semi	83.23	80.41	88.74	86.79	84.84	71.00	85.60	79.40
id, +Lemma+Char	D&M	88.47	86.87	92.10	90.83	88.3	73.54	89.62	83.74
Ours-Sup	88.68	86.91	92.23	90.95	88.61	73.39	89.84	83.75
Ours-Semi	88.95	87.22	92.23	90.97	88.71	73.56	89.96	83.92
ood, +Lemma+Char	D&M	83.42	81.31	88.55	86.59	85.23	71.34	85.73	79.75
Ours-Sup	83.64	81.31	88.71	86.82	85.28	71.41	85.88	79.85
Ours-Semi	83.93	81.67	89.00	87.09	85.37	71.65	86.10	80.14
Table 7: Experimental results on all the three representations. D&M stands for the supervised model of Dozat and Manning (2018). Ours-Sup stands for our model trained on labeled data only. Ours-Semi stands for our model trained on both labeled and unlabeled data.
Derivation of the marginalized probability over all possible dependency graphs of a sentence with  $m$  words for Eq.9 is shown below.
$$
\log \sum_ {Y \in \mathcal {Y}} P (\hat {\mathbf {s}}, Y | \mathbf {s}) = \log \sum_ {Y \in \mathcal {Y}} P (Y | \mathbf {s}) P (\hat {\mathbf {s}} | Y)
$$
$$
= \log \sum_ {Y \in \mathcal {Y}} \prod_ {i, j} P (y _ {i, j} | \mathbf {s}) ^ {m + 1} \sqrt {P (\hat {s} _ {j} | \hat {\mathbf {s}} _ {0 : j - 1} , y _ {i , j})}
$$
$$
\begin{array}{l} = \log \sum_ {y _ {0, 1}} \dots \sum_ {y _ {i, j}} \dots \sum_ {y _ {m, m}} \left(\prod_ {i, j} P (y _ {i, j} | \mathbf {s}) \right. \\ \times \sqrt [ m + 1 ]{P (\hat {s} _ {j} | \tilde {\mathbf {s}} _ {0 : j - 1} , y _ {i , j})} \\ \end{array}
$$
$$
= \log \left(\left(\sum_ {y _ {0, 1}} P \left(y _ {0, 1} \mid \mathbf {s}\right) ^ {m + 1} \sqrt {P \left(\hat {s} _ {1} \mid \hat {s} _ {0} , y _ {0 , 1}\right)}\right) \right.
$$
$$
\begin{array}{l} \ldots \Big (\sum_ {y _ {m, m}} P (y _ {m, m} | \mathbf {s}) ^ {m + 1} \sqrt {P \big (\hat {s} _ {m} | \hat {\mathbf {s}} _ {0 : m - 1} , y _ {m , m} \big)} \Big) \\ = \log \prod_ {i, j} \left(\sum_ {y _ {i, j}} P (y _ {i, j} | \mathbf {s}) ^ {m + 1} \sqrt {P (\hat {s} _ {j} | \hat {\mathbf {s}} _ {0 : j - 1} , y _ {i , j})}\right) \\ = \sum_ {i, j} \log \left(\sum_ {y _ {i, j}} \left(P \left(y _ {i, j} \mid \mathbf {s}\right) \right. \right. \\ \left. \times \sqrt [ m + 1 ]{P (\hat {s} _ {j}) | \hat {\mathbf {s}} _ {0 : j - 1} , y _ {i , j})}\right) \\ \end{array}
$$
B Experiments on All Representations
Results for experiments on DM, PAS and PSD under the setting of Section 4.4 (i.e., basic, +Char and +Lemma+Char) are summarized in Table 7.