[display] 1
0px
Semi-Supervised Methods for Out-of-Domain Dependency Parsing
Juntao Yu
School of Computer Science
Introduction
Syntactic parsing is an important natural language processing (NLP) task that focuses on analysing the syntactic structures of sentences. The syntax of a sentence has been found to be important to many other NLP tasks that require deeper analysis of the sentences, such as semantic parsing BIBREF0 , BIBREF1 , anaphora resolution BIBREF2 , BIBREF3 and machine translation BIBREF4 . There are two major families of syntactic parsing, the first one is constituency parsing that generates parse trees of sentences according to phrase structure grammars, the other is dependency parsing that assigns head-child relations to the words of a sentence. Initially, the parsing community mainly focused on constituency parsing systems, as a result,Œ a number of high accuracy constituency parsers have been introduced, such as the Collins Parser BIBREF5 , Stanford PCFG Parser BIBREF6 , BLLIP reranking parser BIBREF7 and Berkeley Parser BIBREF8 . In the past decade, dependency-based systems have gained more and more attention BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , as they have a better multi-lingual capacity and are more efficient. For a long period, dependency parsing systems were mainly based on carefully selected feature sets, we denote those systems as conventional dependency parsers. In the recent years, a number of dependency parsing systems based on neural networks have also been investigated, some of which have achieved better accuracies when compared to conventional dependency parsers. We evaluated our approaches only on conventional dependency parsers, as these neural network-based systems were introduced after we finished most of the work. However, the techniques evaluated in this thesis have the potential to be adapted to neural network-based parsers as well.
Many dependency parsers are based on supervised learning techniques, which could produce high accuracy when trained on a large amount of training data from the same domain BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 . However, those models trained on the specific training data are vulnerable when dealing with data from domains different from the training data BIBREF14 , BIBREF15 . One effective way to make models less domain specific is to annotate more balanced corpora. However, the annotation work is very time-consuming and expensive. As a result of these difficulties, only very limited annotations are available to the community. As an alternative to annotating new corpora, domain adaptation techniques have been introduced to train more robust models for out-of-domain parsing. Semi-supervised methods are one family of those techniques that aim to improve the out-of-domain parsing performance by enhancing the in-domain models with a large amount of unlabelled data. Some semi-supervised methods use the unlabelled data as the additional training data, such as co-training BIBREF16 , BIBREF17 , BIBREF18 and self-training BIBREF19 , BIBREF20 , BIBREF21 . Alternatively, other research uses the unlabelled data indirectly. Word clusters BIBREF22 , BIBREF23 and word embeddings BIBREF24 , BIBREF25 are examples of this direction.
Research Questions
The focus of this thesis is on using semi-supervised techniques to bridge the accuracies between the in-domain and the out-of-domain dependency parsing. More precisely, this thesis evaluates three important semi-supervised methods, namely co-training, self-training and dependency language models. Two of the methods use unlabelled data directly as additional training data (i.e. co-/self-training). Co-training is a method that has been used in many domain adaptation tasks, it uses multiple learners to derive additional training data from unlabelled target domain data. The successful use of co-training is conditioned on learners being as different as possible. Previous work on parsing with co-training is mainly focused on using learners that are carefully designed to be very different. In this thesis, we use only off-the-shelf dependency parsers as our learners to form our co-training approaches. In total, we evaluate two co-training approaches, the normal co-training (uses two parsers) and the tri-training (uses three parsers). For both approaches, the evaluation learner is retrained on the additional training data annotated identically by two source learners. The normal co-training uses two learners, the evaluation learner is used as one of the source learners, while the tri-training uses three learners, two of which are used as source learners, the third one is used as the evaluation learner. Compare to the normal co-training, tri-training approach allows the evaluation learner to learn from the novel annotations that is not predicted by its own. For our evaluation on co-training, we trying to answer the following research questions:
Q1. Could the off-the-shelf dependency parsers be successfully used in co-training for domain adaptation?
Q2. Would tri-training be more effective for out-of-domain parsing when off-the-shelf dependency parsers are used?
In contrast to co-training, which retrains the parser on additional training data annotated by multiple learners, self-training retrains the parser on training data enlarged by its own automatically labelled data. Previous research mainly focused on applying self-training to constituency parsers BIBREF19 , BIBREF20 , BIBREF21 . Attempts to use self-training for dependency parsing either need additional classifiers BIBREF26 or only use partial parse trees BIBREF27 . In this thesis, we aim to find a more effective way to use self-training for dependency parsing. We intend to answer the following research questions for our self-training evaluation:
Q3. How could self-training be effectively used in out-of-domain dependency parsing?
Q4. If self-training works for English dependency parsing, can it be adapted to other languages?
To use auto-labelled data as additional training data is effective but comes with consequences. First of all, the re-trained models usually have a lower performance on the source domain data. Secondly, those approaches can only use a relatively small unlabelled data, as training parsers on a large corpus might be time-consuming or even intractable on a corpus of millions of sentences. To overcome those limitations we investigate dependency language models which use the unlabelled data indirectly. Dependency language models (DLM) were previously used by chen2012utilizing to leverage the performance and the efficiency of a weak second-order graph-based parser BIBREF9 . In this thesis, we adapt this method to a strong transition-based parser BIBREF12 that on its own can produce very promising accuracies. The research questions for this part are as follows:
Q5. Can dependency language models be adapted to strong transition-based parsers?
Q6. Can dependency language models be used for out-of-domain parsing?
Q7. Quality or quantity of the auto-parsed data, which one is more important to the successful use of dependency language models?
Thesis Structure
After the introduction, in Chapter SECREF7 we begin by discussing the background knowledge and previous work related to this thesis. This mainly covers two topics, dependency parsing and domain adaptation. We then introduce the Mate parser in detail. Mate is a strong transition-based parser which is used in all of our evaluations. After that, we introduce the corpora and the evaluation/analysis methods.
In Chapter SECREF14 we introduce our experiments on agreement-based co-training. It first discusses the effect of using different off-the-shelf parsers on a normal agreement-based co-training setting (i.e. only involves two parsers). And then we introduce our experiments on its variant that uses three parsers (tri-training).
Chapter SECREF20 and Chapter SECREF26 introduce our confidence-based self-training approaches. In Chapter SECREF20 , we introduce our evaluations on confidence-based self-training for English out-of-domain dependency parsing. In total, two confidence-based methods are compared in our experiments. Chapter SECREF26 introduces our experiments on multi-lingual datasets. The confidence-based self-training approach is evaluated on nine languages.
Chapter SECREF32 discusses our dependency language models method that is able to improve both in-domain and out-of-domain parsing. The evaluations on English include both in-domain and out-of-domain datasets, in addition to that, we also evaluated on the Chinese in-domain data.
Chapter SECREF38 provides a summary of the thesis and gives conclusions.
Published Work
In total, there are four publications based on this thesis. Each of the publications is related to one chapter of this thesis, pekar2014exploring is related to our evaluation on co-training (Chapter SECREF14 ). yu2015iwpt is made from our English self-training evaluation (Chapter SECREF20 ). yu2015depling is associated with our multi-lingual self-training experiments (Chapter SECREF26 ). yu2017iwpt presents our work on dependency language models (Chapter SECREF32 ).
Chapter Summary
In this chapter, we first briefly introduced dependency parsing and the problems of out-of-domain parsing that we are trying to address in this thesis. We then discussed the research questions that we intend to answer. The chapter also gave a brief introduction of the thesis structure. Finally, the chapter illustrated the published works based on this thesis.
This chapter introduced the background and the experiment set-up. The first part focused on dependency parsers, it introduced three major types of dependency parsers and gave a detailed introduction of the base parser used in this thesis. The second part discussed the problem caused by parsing out-of-domain text and the techniques that have been used by previous work to solve the problem. The third part introduced the corpora we used. The last two parts showed our evaluation methods and analysis techniques.
In this chapter we present our evaluations on two co-training approaches (co-training and tri-training). The main contribution of our evaluation on co-training is to assess the suitability of using the off-the-shelf parsers to form co-training. We first evaluated on the normal agreement based co-training with four off-the-shelf parsers. Three of them are paired with the Mate parser to generate additional training data for retraining the Mate parser. We evaluated the parser pairs by adding different number of sentences into the training data. We also evaluated the pairs with additional training data that excluded the short annotations. The results show co-training is able to improve largely on target domain and additional gains are achieved when excluding the short sentences. We then evaluated the second approach (tri-training) that retrains the Mate parser on additional training data annotated identically by MST-Malt parsers. Benefit from the novel annotations that not predicted by the Mate parser, tri-training outperforms our best co-training setting. The further evaluation on tri-training shows large improvements on all four test domains. The method achieved the largest improvement of 1.8% and 0.6% for labelled and unlabelled accuracies. We then applied both token level and sentence level analysis to find out where the improvement comes from. The analysis suggests tri-training gained particularly large improvement on label OBJ (objects) and PRD (predicative complement). The analysis of unknown words on both token level and sentence level shows only a slightly larger improvement on unknown words when compared with known words. The analysis on sentence length suggests tri-training helped mainly on sentences with a length between 15 and 30 tokens. The analysis on prepositions and conjunctions shows larger gains are achieved on sentences containing prepositions or conjunctions. Overall we demonstrated that co-/tri-training are powerful techniques for out-of-domain parsing when the off-the-shelf parsers are used.
In this chapter, we introduced two novel confidence-based self-training approaches to domain adaptation for dependency parsing. We compared a self-training approach that uses random selection and two confidence-based approaches. The random selection-based self-training method did not improve the accuracy which is in line with previously published negative results, both confidence-based methods achieved statistically significant improvements and showed relatively high accuracy gains.
We tested both confidence-based approaches on three web related domains of our main evaluation corpora (Weblogs, Newsgroups, Reviews) and the Chemical domain. Our confidence-based approaches achieved statistically significant improvements in all tested domains. For web domains, we gained up to 0.8 percentage points for both labelled and unlabelled accuracies. On average the Delta-based approach improved the accuracy by 0.6% for both labelled and unlabelled accuracies. Similarly, the parse score-based method improved labelled accuracy scores by 0.6% and unlabelled accuracy scores by 0.5%. In terms of the Chemical domain, the Delta-based and the parse score-based approaches gained 1.42% and 1.12% labelled accuracies respectively when using predicted PoS tags. When we used gold PoS tags, a larger labelled improvement of 1.62% is achieved by the Delta method and 1.48% is gained by the parse score method. The unlabelled improvements for both methods are similar to their labelled improvements for all the experiments. In total, our approaches achieved significantly better accuracy for all four domains.
We conclude from the experiments that self-training based on confidence is worth applying in a domain adaptation scenario and that a confidence-based self-training approach seems to be crucial for the successful application of self-training in dependency parsing. Our evaluation underlines the finding that the pre-selection of parse trees is probably a precondition that self-training becomes effective in the case of dependency parsing and to reach a significant accuracy gain.
The further analysis compared the behaviour of two approaches and gave a clearer picture of in which part self-training helps most. As a preliminary analysis, we assessed the overlap between the top ranked sentences of two methods. When we compared the top ranked 50% of the development set by different methods, 56% of them are identical. As there are more than 40% sentences which are selected differently by different methods, we expect some clear differences in our in-depth analysis on token and sentence level. Surprisingly, the further analysis suggested that both methods played similar roles on most of the analysis, the behaviour differences are rather small. In our token level analysis, both methods gained large improvements on the root, coordination, modifiers and unclassified relations. We also found much larger unlabelled improvements for unknown words. For sentence level analysis, we noticed that our approaches helped most the medium length sentences (10-30 tokens/sentence). Generally speaking, they also have a better performance on sentences that have certain levels of complexity, such as sentences that have more than 2 unknown words or at least 2 prepositions. This might also because of the simpler sentences have already a reasonably good accuracy when baseline model is used, thus are harder to improve.
In this chapter, we evaluated an effective confidence-based self-training approach on nine languages. Due to the lack of out-of-domain resources, we used an under-resourced in-domain setting instead. We used for all languages a unified setting, the parser is retrained on the new training set boosted by the top 50k ranked parse trees selected from a 100k auto-parsed dataset.
Our approach successfully improved accuracies of five languages (Basque, German, Hungarian, Korean and Swedish) without tuning variables for the individual language. We can report the largest labelled and unlabelled accuracy gain of 2.14% and 1.79% on Korean, on average we improved the baselines of five languages by 0.87% (LAS) and 0.78% (UAS).
We further did an in-depth analysis on Korean and French. For Korean, we did a number of analysis on both token level and sentence level to understand where the improvement comes from. The analysis on the individual label showed that the self-trained model achieved large improvement on all the major labels, and it achieved the largest gain on conjuncts (conj). The analysis of unknown words showed that the self-trained model gained a larger labelled improvement for unknown words. The analysis on sentence length suggested the self-training approach achieved larger improvements on longer sentences. For French, we aim to understand why self-training did not work. The analysis showed the confidence scores have a reasonably high correlation with the annotation quality, hence it is less likely be the reason of self-training's negative effect. While the large difference between unlabelled data and the training/test sets is more likely a major contributor to the accuracy drop.
In this chapter, we adapted the dependency language models (DLM) approach of chen2012utilizing to a strong transition-based parser. We integrated a small number of DLM-based features into the parser to allow the parser to explore DLMs extracted from a large auto-parsed corpus. We evaluated the parser with single and multiple DLMs extracted from corpora of different size and quality to improve the in-domain accuracy of the English and Chinese texts. The English model enhanced by a unigram DLM extracted from double parsed high-quality sentences achieved statistically significant improvements of 0.46% and 0.51% for labelled and unlabelled accuracies respectively. Our results outperform most of the latest systems and are close to the state-of-the-art. By using all unigram, bigram and trigram DLMs in our Chinese experiments, we achieved large improvements of 0.93% and 0.98% for both labelled and unlabelled scores. When increasing the beam size to 150, our system outperforms the best reported results by 0.2%. In addition to that, our approach gained an improvement of 0.4% on Chinese part-of-speech tagging.
We further evaluate our approach on our main evaluation corpus. The method is tested on both in-domain and out-of-domain parsing. Our DLM-based approach achieved large improvement on all five domains evaluated (Conll, Weblogs, Newsgroups, Reviews, Answers). We achieved the labelled and unlabelled improvements of up to 0.91% and 0.82% on Newsgroups domain. On average we achieved 0.6% gains for both labelled and unlabelled scores on four out-of-domain test sets. We also improved the in-domain accuracy by 0.36% (LAS) and 0.4% (UAS).
The analysis on our English main evaluation corpus suggests that the DLM model behaves differently on in-domain and out-of-domain parsing for a number of factors. Firstly, the DLM model achieved the largest improvement on label CONJ (conjunct) and LOC (locative adverbial) for in-domain parsing, while the largest improvement for out-of-domain dataset is contributed by OBJ (object) and PRD (predicative complement). Secondly, the DLM model improved more on unknown words for in-domain data but for out-of-domain text, DLM model delivered larger gains on known words. Thirdly, the analysis on sentence level shows that our model achieved most improvement on sentences of a length between 10 and 20, the range is wider (10-35) for out-of-domain data.
We also analysed the Chinese results. The analysis shows the improvement on Chinese data is mainly contributed by the objects (OBJ, POBJ), dependent of DE (DEC, DEG) and children of localizer (LC). The DLM model only shows a large improvement on the known words, it nearly does not affect the unknown words accuracy. The DLM model mostly helped the sentences that have at least 20 tokens.
In this chapter, we summarised our work of this thesis by answering seven research questions that we introduced in Chapter SECREF2 . We successfully answered all the questions using our findings in the previous chapters.
Background and Experiment Set-up
In this chapter, we first introduce the background and related work of this thesis, which includes a brief introduction of dependency parsing systems, a detailed introduction of the baseline parser BIBREF12 and previous work on out-of-domain parsing (especially those on semi-supervised approaches). We then introduce the corpora that have been used in this thesis. Finally, we introduce the evaluation metric and the analysis methods.
Dependency parsing
Dependency parsing is one important way to analyse the syntactic structures of natural language. It has been widely studied in the past decade. A dependency parsing task takes natural language (usually tokenised sentence) as input and outputs a sequence of head-dependent relations. Figure FIGREF5 shows the dependency relations of a sentence (Tom played football with his classmate .) parsed by an off-the-shelf dependency parser. During the past decade, many dependency parsing systems have been introduced, most of them are graph-based or transition-based systems. The graph-based system solves the parsing problem by searching for maximum spanning trees (MST). A first-order MST parser first assigns scores to directed edges between tokens of a sentence. It then uses an algorithm to search a valid dependency tree with the highest score. By contrast, the transition-based system solves the parsing task as a sequence of transition decisions, in each step the parser deciding the next transition. In Section SECREF6 and SECREF9 we briefly describe the two major system types. In recent years, deep learning has been playing an important role in the machine learning community. As a result, several neural network-based systems have been introduced, some of them surpassing the state-of-the-art accuracy achieved by the conventional dependency parsers based on perceptions or SVMs. We briefly touch on neural network-based systems in Section SECREF11 , although most of them are still transition/graph-based systems. The evaluation of the neural network-based parsers is beyond the scope of this thesis, as they become popular after most of the work of this thesis has been done. We mainly use the Mate parser BIBREF12 , a transition-based approach that was state-of-the-art at the beginning of this work and whose performance remained competitive even after the introduction of the parsers based on neural network. Section SECREF13 introduces the technical details of the Mate parser.
Graph-based Systems
The graph-based dependency parser solves the parsing problem by searching for maximum spanning trees (MST). In the following, we consider the first-order MST parser of mcdonald05acl. Let INLINEFORM0 be the input sentence, INLINEFORM1 be the dependency tree of INLINEFORM2 , INLINEFORM3 is the INLINEFORM4 th word of INLINEFORM5 , INLINEFORM6 is the directed edge between INLINEFORM7 (head) and INLINEFORM8 (dependent). INLINEFORM9 is used to represent the set of possible dependency trees of the input sentence where INLINEFORM10 . The parser considers all valid directed edges between tokens in INLINEFORM11 and builds the parse trees in a bottom-up fashion by applying a CKY parsing algorithm. It scores a parse tree INLINEFORM12 by summing up the scores INLINEFORM13 of all the edges INLINEFORM14 . The INLINEFORM15 is calculated according to a high-dimensional binary feature representation INLINEFORM16 and a weight vector INLINEFORM17 learned from training data INLINEFORM18 ( INLINEFORM19 ). To be more specific, the score of a parse tree INLINEFORM20 of an input sentence INLINEFORM21 is calculated as follows: INLINEFORM22
Where INLINEFORM0 consists of a set of binary feature representations associated with a number of feature templates. For example, an edge INLINEFORM1 with a bi-gram feature template INLINEFORM2 will give a value of 1 for the following feature representation: INLINEFORM3
After scoring the possible parse trees INLINEFORM0 , the parser outputs the highest-scored dependency tree INLINEFORM1 . Figure FIGREF7 shows an example of a sentence being parsed with a first-order graph-based parser.
In terms of training, the parser uses an online learning algorithm to learn the weight vector INLINEFORM0 from the training set INLINEFORM1 . In each training step, only one training instance INLINEFORM2 ( INLINEFORM3 ) is considered, the INLINEFORM4 is updated after each step. More precisely, the Margin Infused Relaxed Algorithm (MIRA) BIBREF28 is used to create a margin between the score of a correct parse tree INLINEFORM5 and the incorrect ones INLINEFORM6 ( INLINEFORM7 ). The loss INLINEFORM8 of a dependency tree is defined as the number of incorrect edges. Let INLINEFORM9 , INLINEFORM10 be the weight vector before and after the update of the INLINEFORM11 th training step, INLINEFORM12 is updated subject to keeping the margin at least as large as the INLINEFORM13 , while at the same time, keeping the norm of the changes to the INLINEFORM14 as small as possible. A more detailed training algorithm is showed in algorithm SECREF6 .
[h] INLINEFORM0 INLINEFORM1 INLINEFORM2 (*[h]N training iterations) INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8 MIRA algorithm for MST parser
The MST parser is later improved by mcdonald2006online to include second-order features, however, the system is still weaker than its successors which also include third-order features BIBREF29 . Other mostly used strong graph-based parsers include Mate graph-based parser BIBREF30 and Turbo Parser BIBREF13 .
Transition-based Systems
The transition-based parsers build the dependency trees in a very different fashion compared to graph-based systems. Instead of searching for the maximum spanning trees, transition-based systems parse a sentence with a few pre-defined transitions. The Malt parser BIBREF31 is one of the earliest transition-based parsers which has been later widely used by researchers. The parser is well engineered and can be configured to use different transition systems. We take the parser's default transition system (arc-eager) as an example to show how the transition-based parser works. The Malt parser starts with an initial configuration and performs one transition at a time in a deterministic fashion until it reaches the final configuration. The parser's configurations are represented by triples INLINEFORM0 , where INLINEFORM1 is the stack that stores partially visited tokens, INLINEFORM2 is a list of remaining tokens that are unvisited, and INLINEFORM3 stores the directed arcs between token pairs that have already been parsed. The parser's initial configuration consists of an empty INLINEFORM4 and an empty INLINEFORM5 , while all the input tokens are stored in INLINEFORM6 . The final configuration is required to have an empty INLINEFORM7 . A set of four transitions (Shift, Left-Arc, Right-Arc and Reduce) are defined to build the parse trees. The Shift transition moves the token on the top of INLINEFORM8 into INLINEFORM9 , the Left-Arc transition adds an arc from the top of INLINEFORM10 to the top of INLINEFORM11 and removes the token on the top of INLINEFORM12 , the Right-Arc transition adds an arc from the top of INLINEFORM13 to the top of INLINEFORM14 and moves the token on the top of INLINEFORM15 into INLINEFORM16 , and the Reduce transition simply removes the token on the top of INLINEFORM17 . More precisely, table TABREF10 shows the details of the transitions of an arc-eager system.
To train the parser, support vector machine classifier (SVM) with the one-versus-all strategy is used to solve the transition-based parser as a multi-classification problem. In a transition-based parsing scenario, the classes are different transitions. Each of the SVMs is trained to maximise the margin between the target transition and the other transitions, as in the one-versus-all strategy the classes other than the target class are treated the same as the negative examples. Since the data may not be linearly separable, they use in additional a quadratic kernel ( INLINEFORM0 ) to map the data into a higher dimensional space. The SVMs are trained to predict the next transition based on a given parser configuration. They used similar binary feature representations as those of the MST parser, in which the features are mapped into a high dimensional vector. The feature templates for the transition-based system are mainly associated with the configurations, for example, a feature between the INLINEFORM1 (the top of the stack) and the INLINEFORM2 (the top of the Buffer) is as follows: INLINEFORM3
Figure FIGREF8 shows an example of parsing the sentence (Tom plays football) with the Malt transition-based parser.
Benefiting from the deterministic algorithm, the Malt parser is able to parse the non-projective sentences in linear time BIBREF10 , which is much faster compared to the second-order MST parser's cubic-time parsing BIBREF9 . Although the deterministic parsing is fast, the error made in the previous transitions will largely affect the decisions taken afterwards, which results in a lower accuracy. To overcome this problem beam search has been introduced to the transition-based systems, which leads to significant accuracy improvements BIBREF12 .
Neural Network-based Systems
Neural network-based systems have only been recently introduced to the literature. chen2014neural were the first to introduce a simple neural network to a deterministic transition-based parser, yielding good results. The parser used an arc-standard transition system. Similar to arc-eager, the arc-standard is another highly used transition-based system. Many dependency parsers are based on or have options to use an arc-standard approach, which include the Malt parser we introduced in the previous section (section SECREF9 ) and our main evaluation parser (Mate parser). We will introduce the arc-standard transition system in more detail in section SECREF13 .
One of the major differences between the neural network based systems and the conventional systems is the use of feature representations. Instead of using the binary feature representations (commonly used by the conventional systems), the neural network based approaches represent the features by embeddings. During training, feature embeddings (e.g. word, part-of-speech embeddings) are capable of capturing the semantic information of the features. Take the part-of-speech tags as an example, adjective tags INLINEFORM0 will have similar embeddings. This allows the neural network-based systems to reduce the feature sparsity problem of the conventional parser systems. Conventional parsers usually represent different tokens or token combinations by independent feature spaces, thus are highly sparse.
Another advantage of using the neural network based approach is that the system allows using the pre-trained word embeddings. Word embeddings extracted from large unlabelled data carry the statistical strength of the words, this could be a better bases for the system when compared to the randomly initialised embeddings. The empirical results confirmed that large improvements can be achieved by using the pre-trained word embeddings. The idea of using the pre-trained word embeddings goes into the same direction of the semi-supervised approaches that use unlabelled data indirectly, such as dependency language models evaluated in this thesis, or word clusters.
In terms of the network architecture, chen2014neural used a single hidden layer and a softmax layer to predict the next transition based on the current configuration. To map the input layer to the hidden layer they used a cube activation function ( INLINEFORM0 ), in which INLINEFORM1 are feature embeddings of the words, part-of-speech tags and arc labels and INLINEFORM2 are the relative weights. Figure FIGREF12 shows the details of their neural network architecture.
This first attempt of using the neural network for dependency parsing leads to many subsequent research. chen2014neural's system has been later extended by weiss2015neural who introduced beam search to the system and achieved state-of-the-art accuracy. Since then a number of more complex and powerful neural networks have been evaluated, such as the stack-LSTM BIBREF32 and the bi-directional LSTM BIBREF33 . The current state-of-the-art is achieved by the parser of dozat2017deep who used the bi-directional LSTM in their system.
The Mate Parser
In this thesis, we mainly used the Mate transition-based parser BIBREF34 , BIBREF35 , BIBREF12 . The parser is one of the best performing parsers on the data set of the major shared task (CoNLL 2009) on dependency parsing BIBREF1 and it is freely available . The parser uses the arc-standard transition system, it is also integrated with a number of techniques to maximise the parser's performance. Firstly, the parser employs a beam search to go beyond the greedy approach. Secondly, it uses an additional optional graph-based model to rescore the beam entries. In their paper BIBREF34 , they name it completion model as it scores factors of the graph as soon as they are finished by the parser. Furthermore, the parser has an option for joint tagging and parsing BIBREF35 . Same as the pipeline system, the tagger model is trained separately from the parser model. However, during the parsing, instead of using only the best-predicted part-of-speech (PoS) tag, they made the n-best ( INLINEFORM0 ) PoS tags of a token available to the parser. The joint system is able to gain a higher accuracy for both PoS tagging and parsing compared to a pipeline system. In this thesis, we use the Mate parser as our baseline and make the necessary modifications, where appropriate to comply with the requirements of our approaches.
The transition-based part of the parser uses a modified arc-standard transition system. Comparing to the original arc-standard transition system (has only three transitions: Left-Arc, Right-Arc and Shift) of nivre2004incrementality, the Mate parser modified the Shift transition for joint tagging and parsing and included the Swap transition to handling non-projective parsing. More precisely, the parser tags and parses a sentence INLINEFORM0 using a sequence of transitions listed in Table TABREF15 . An additional artificial token INLINEFORM1 root INLINEFORM2 INLINEFORM3 is added to the beginning of the sentence to allow the parser assigning a Root to the sentence at the last step of the transitions. The transitions change the initial configuration ( INLINEFORM4 ) in steps until reaching a terminal configuration ( INLINEFORM5 ). bohnet2013joint used the 5-tuples INLINEFORM6 to represent all configurations, where INLINEFORM7 (the stack) and INLINEFORM8 (the buffer) refers to disjoint sublists of the sentence INLINEFORM9 , INLINEFORM10 is a set of arcs, INLINEFORM11 and INLINEFORM12 are functions to assign a part-of-speech tag to each word and a dependency label to each arc. The initial configuration ( INLINEFORM13 ) has an empty stack, the buffer consists of the full input sentence INLINEFORM14 , and the arc set INLINEFORM15 is empty. The terminal configuration ( INLINEFORM16 ) is characterised by an empty stack and buffer, hence no further transitions can be taken. The arc set INLINEFORM17 consists of a sequence of arc pairs ( INLINEFORM18 ), where INLINEFORM19 is the head and INLINEFORM20 is the dependent. They use Tree INLINEFORM21 to represent the tagged dependency tree defined for INLINEFORM22 by INLINEFORM23 .
As shown in Table TABREF15 , the Left-Arc INLINEFORM0 adds an arc from the token ( INLINEFORM1 ) at the top of the stack ( INLINEFORM2 ) to the token ( INLINEFORM3 ) at the second top of the stack and removes the dependent ( INLINEFORM4 ) from the stack. At the same time, the INLINEFORM5 function assigns a dependency label ( INLINEFORM6 ) to the newly created arc INLINEFORM7 . The Left-Arc INLINEFORM8 transition is permissible as long as the token at the second top of the stack is not the INLINEFORM9 root INLINEFORM10 (i.e. INLINEFORM11 ). The Right-Arc INLINEFORM12 adds a labelled arc from the token ( INLINEFORM13 ) at the second top of the stack to the token ( INLINEFORM14 ) at the top of the stack and removes the later. The Shift INLINEFORM15 transition assigns a PoS tag INLINEFORM16 to the first node of the buffer and moves it to the top of the stack. The Swap transition that is used to handling non-projective tree extracts the token ( INLINEFORM17 ) at the second top of the stack and moves it back to the buffer. The Swap transition is only permissible when the top two tokens of the stack are in the original word order (i.e. INLINEFORM18 ), this prevents the same two tokens from being swapped more than once. In additional, the artificial INLINEFORM19 root INLINEFORM20 token is not allowed to be swapped back to the buffer (i.e. INLINEFORM21 ). Figure FIGREF16 shows an example of joint tagging and parsing a sentence by the Mate parser.
The graph-based completion model consists of a number of different second- and third-order feature models to rescore the partial parse tree Tree INLINEFORM0 . Some feature models are similar to carreras07 and koo10acl. Take one of the models INLINEFORM1 as an example, which consists of the second-order factors of carreras07:
The head and the dependent.
The head, the dependent and the right/left-most grandchild in between.
The head, the dependent and the right/left-most grandchild away from the head.
The head, the dependent and between those words the right/left-most sibling.
Feature models are independent to each other and can be easily turned on/off by configuration. The score of a parse tree Tree INLINEFORM0 or a partial parse tree Tree INLINEFORM1 is then defined as the sum of the scores from the both parts: INLINEFORM2
Where INLINEFORM0 is the score of the transition-based part of the parser and INLINEFORM1 is the score from the graph-based completion model.
[t] INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8 INLINEFORM9 INLINEFORM10 INLINEFORM11 INLINEFORM12 INLINEFORM13 INLINEFORM14 INLINEFORM15 return INLINEFORM16 Beam search algorithm for the Mate parser
Mate parser uses similar binary feature representations as those of the MST/Malt parser (the features are represented by a high dimensional feature vector ( INLINEFORM0 )). A learned weight vector ( INLINEFORM1 ) is used with the feature vector ( INLINEFORM2 ) to score the configurations in conjunction with the next transition. In addition, the parser uses the beam search to mitigate error propagation. Comparing with the deterministic parsing algorithm that only keeps the best partial parse tree, the beam search approach keeps the n-best partial parse trees during the inference. By using the beam search, errors made in the early stage can potentially be recovered in the late stage, as long as the correct configuration has not fallen out of the beam. The beam search algorithm takes a sentence ( INLINEFORM3 ), the weight vector ( INLINEFORM4 ) and the beam size parameter ( INLINEFORM5 ) and returns the best scoring parse tree (Tree INLINEFORM6 ). A parse hypothesis ( INLINEFORM7 ) of a sentence consists of a configuration ( INLINEFORM8 ), a score ( INLINEFORM9 ) and a feature vector ( INLINEFORM10 ). Initially the Beam only consists of the initial hypothesis ( INLINEFORM11 ), in which INLINEFORM12 contains a initial configuration of the sentence ( INLINEFORM13 ), a score of INLINEFORM14 and a initial feature vector ( INLINEFORM15 ). The transitions ( INLINEFORM16 ) change the hypotheses in steps and create new hypotheses by applying different permissible transitions to them. For each step, the top INLINEFORM17 scoring hypotheses are kept in the Beam. The beam search terminates when every hypothesis in the Beam contains a terminal configuration ( INLINEFORM18 ). It then returns the top scoring parse tree (Tree INLINEFORM19 ). Algorithm SECREF13 outlines the details of the beam search algorithm used by the Mate parser.
In order to learn the weight vector, the parser goes through the training set ( INLINEFORM0 ) for INLINEFORM1 iterations. The weight vector is updated for every sentence INLINEFORM2 when an incorrect parse is returned (i.e. the highest scoring parse INLINEFORM3 is different from the gold parse INLINEFORM4 ). More precisely, the passive-aggressive update of crammer2006online is used: INLINEFORM5
In this thesis, unless specified, we used the default settings of the parser:
We use all the graph-based features of the completion model.
We use the joint PoS-tagging with two-best tags for each token.
We use a beam of 40.
We use 25 iterations of training.
We do not change the sentence order of the training data during training.
Out-of-domain Parsing
The release of the large manually annotated Penn Treebank (PTB) BIBREF36 and the development of the supervised learning techniques enable researchers to work on the supervised learning based parsing systems. Over the last two decades, the parsing accuracy has been significantly improved. A number of strong parsing systems for both constituency and dependency families have been developed BIBREF6 , BIBREF8 , BIBREF12 , BIBREF13 , BIBREF25 , BIBREF33 . The parsers based on supervised learning techniques capture statistics from labelled corpora to enable the systems to correctly predict parse trees when input the corresponding sentences. Since the PTB corpus contains mainly texts from news domain, the supervised learning based parsers trained on PTB corpus are sensitive to domain shifting. Those systems are able to achieve high accuracies when tested on the PTB test set (i.e. in-domain parsing). However, when applying them on data from different sources (i.e. out-of-domain parsing), such as web domain BIBREF15 and chemical text BIBREF14 , the accuracy drops significantly. Table TABREF27 shows a comparison of the in-domain and out-of-domain parsing performance of three parsers that have been frequently used by researchers (i.e. MST BIBREF9 , Malt BIBREF10 , and Mate parser BIBREF12 ). Those parsers are trained on the training data from the major shared task on dependency parsing (i.e. CoNLL 2009 BIBREF1 ). The training set contains mainly the news domain data from the Penn Treebank. In our evaluation, we first test them on the CoNLL test set which denotes our in-domain examples; for our out-of-domain examples we test the parsers on a number of different domains from the OntoNotes v5.0 corpus. As we can see from the results, the accuracies on out-of-domain texts are much lower than that of in-domain texts, with the largest accuracy difference of more than 15% (i.e. Mate parser has an accuracy of 90.1% on in-domain texts and an accuracy of 74.4% on texts from broadcast conversations). How can we reduce the accuracy gap between the in-domain and the out-of-domain parsing? The most straightforward way would be annotating more text for the target domain, however, this approach is very expensive and time-consuming. There are only very limited manually annotated corpora available, which confirms the high costs of the annotation process. Domain adaptation is a task focused on solving the out-of-domain problems but without the need for manual annotation. There are a number of directions to work on the domain adaptation task, each of them focusing on a different aspect. These directions include semi-supervised techniques, domain specific training data selection, external lexicon resources and parser ensembles. Each direction has its own advantages and disadvantages, we briefly discuss in Section SECREF28 . In this thesis, we mainly focus on one direction that improves the out-of-domain accuracy by using unlabelled data (Semi-supervised approaches). Similar to other domain adaptation approaches, semi-supervised approaches do not require to manually annotate new data, but instead, they use the widely available unlabelled data. Some semi-supervised approaches focus on boosting the training data by unlabelled data that is automatically annotated by the base models, others aid the parsers by incorporating features extracted from the large unlabelled data. In Section SECREF29 we discuss both approaches in detail.
Approaches to Out-of-Domain Parsing
As stated above, the domain adaptation techniques are designed to fill the accuracy gaps between the source domain and the target domain. Previous work on domain adaptation tasks is mainly focused on four directions: semi-supervised techniques BIBREF16 , BIBREF19 , BIBREF20 , BIBREF17 , BIBREF37 , BIBREF21 , BIBREF22 , BIBREF18 , target domain training data selection BIBREF38 , BIBREF39 , BIBREF40 , external lexicon resources BIBREF41 , BIBREF42 , BIBREF23 and parser ensembles BIBREF14 , BIBREF43 , BIBREF18 , BIBREF15 .
The semi-supervised techniques focus on exploring the largely available unlabelled data. There are two major ways to use the unlabelled data. The first family aims to boost the training data. Data that has been automatically annotated by the base models are used directly in re-training as the additional training set, up-training, self-training and co-training are techniques of this family. The other family uses the features extracted from unlabelled data to aid the base model, this type of techniques include word embeddings, word clusters and dependency language models. In this thesis, we use semi-supervised techniques from both families and we will discuss them in detail in Section SECREF29 .
Domain specific training data selection is a technique based on the assumption that similarity methods are able to derive a subset of the source domain training data that fits an individual test domain. plank2011effective investigated several similarity methods to automatically select sentences from training data for the target domain, which gain significant improvements when comparing with random selection. Positive impacts are also found by khan13towards when they experimented with training data selection on parsing five sub-genres of web data. The advantage of this technique is that it does not need any extra data, however, it is also restricted to learn only from the source domain training set.
Lack of the knowledge of the unknown words is one of the well-known problems faced by domain adaptation tasks, i.e. target domain test sets usually contain more unknown words (vocabularies which did not appear in the training data) than source domain test sets BIBREF14 , BIBREF15 . One way to solve this problem is to use the external lexicon resources created by the linguistics. External lexicons provide additional information for tokens, such as word lemma, part-of-speech tags, morphological information and so on. This information can be used by parsers directly to help making the decision. Previously, lexicons have been used by szolovits2003adding and pyysalo2006 to improve the link grammar parser on the medical domain. Both approaches showed large improvements on parsing accuracy. Recently, pekar2014exploring extracted a lexicon from a crowd-sourced online dictionary (Wiktionary) and applied it to a strong dependency parser. Unfortunately, in their approach, the dictionary achieved a moderate improvement only.
The fourth direction of domain adaptation is parser ensembles, it becomes more noticeable, due to its good performance in shared tasks. For example, in the first workshop on syntactic analysis of non-canonical language (SANCL), the ensemble-based systems on average produced much better results than that of single parsers BIBREF43 , BIBREF18 , BIBREF15 . However, those ensemble-based systems are not used in real-world tasks, due to the complex architectures and high running time.
Semi-Supervised Approaches
Semi-supervised approaches use unlabelled data to bridge the accuracy gap between in-domain and out-of-domain. In recent years, unlabelled data has gained large popularity in syntactic parsing tasks, as it can easily and inexpensively be obtained, cf. BIBREF16 , BIBREF44 , BIBREF45 , BIBREF37 , BIBREF46 , BIBREF15 , BIBREF47 , BIBREF25 . This is in stark contrast to the high costs of manually labelling new data. Some techniques such as self-training BIBREF45 and co-training BIBREF16 use auto-parsed data as additional training data. This enables the parser to learn from its own or other parsers' annotations. Other techniques include word clustering BIBREF37 and word embedding BIBREF48 which are generated from a large amount of unlabelled data. The outputs can be used as features or inputs for parsers. Both groups of techniques have been shown effective on syntactic parsing tasks BIBREF49 , BIBREF20 , BIBREF21 , BIBREF46 , BIBREF50 , BIBREF25 .
The first group uses unlabelled data (usually parsed data) directly in the training process as additional training data. The most common approaches in this group are co-training and self-training.
Co-training is a technique, that has been frequently used by domain adaptation for parsers BIBREF16 , BIBREF17 , BIBREF18 , BIBREF15 . The early version of co-training uses two different 'views' of the classifier, each 'view' has a distinct feature set. Two 'views' are used to annotate unlabelled set after trained on the same training set. Then both classifiers are retrained on the newly annotated data and the initial training set BIBREF51 . blum98 first applied a multi-iteration co-training on classifying web pages. Then it was extended by collins99 to investigate named entity classification. At that stage, co-training strongly depended on the splitting of features BIBREF52 . One year after, goldman00 introduced a new variant of co-training which used two different learners, but both of them took the whole feature sets. One learner's high confidence data are used to teach the other learner. After that, zhou2005tri proposed another variant of co-training (tri-training). Tri-training used three learners, each learner is designed to learn from data on which the other two learners have agreed.
In terms of the use of co-training in the syntactic analysis area, sarkar01 first applied the co-training to a phrase structure parser. He used a subset (9695 sentences) of labelled Wall Street Journal data as initial training set and a larger pool of unlabelled data (about 30K sentences). In each iteration of co-training, the most probable INLINEFORM0 sentences from two views are added to the training set of the next iteration. In their experiments, the parser achieved significant improvements in both precision and recall (7.79% and 10.52% respectively) after 12 iterations of co-training.
The work most close to ours was presented by BIBREF17 in the shared task of the conference on computational natural language learning (CoNLL). They used two different settings of a shift-reduce parser to complete a one iteration co-training, and their approach successfully achieved improvements of approximately 2-3%. Their outputs have also scored the best in the out-of-domain track BIBREF14 . The two settings they used in their experiments are distinguished from each other in three ways. Firstly, they parse the sentences in reverse directions (forward vs backward). Secondly, the search strategies are also not the same (best-first vs deterministic). Finally, they use different learners (maximum entropy classifier vs support vector machine). The maximum entropy classifier learns a conditional model INLINEFORM0 by maximising the conditional entropy ( INLINEFORM1 ), while the support vector machines (SVMs) are linear classifiers trained to maximise the margin between different classes. In order to enable the multi-class classification, they used the all-versus-all strategy to train multiple SVMs for predicting the next transition. In addition, a polynomial kernel with degree 2 is used to make the data linearly separable. sagae07 proved their assumptions in their experiments. Firstly, the two settings they used are different enough to produce distinct results. Secondly, the perfect agreement between two learners is an indication of correctness. They reported that the labelled attachment score could be above 90% when the two views agreed. By contrast, the labelled attachment scores of the individual view were only between 78% and 79%.
Tri-training is a variant of co-training. A tri-training approach uses three learners, in which the source learner is retrained on the data produced by the other two learners. This allows the source learner to explore additional annotations that are not predicted by its own, thus it has a potential to be more effective than the co-training. Tri-training is used by BIBREF18 in the first workshop on syntactic analysis of non-canonical language (SANCL) BIBREF15 . They add the sentences which the two parsers agreed on into the third parser's training set, then retrain the third parser on the new training set. However, in their experiments, tri-training did not significantly affect their results.
Recently, weiss2015neural used normal agreement based co-training and tri-training in their evaluation of a state-of-the-art neural network parser. Their evaluation is similar to the Chapter SECREF14 of this thesis, although they used different parsers. Please note their paper is published after our evaluation on co-training BIBREF23 . In their work, the annotations agreed by a conventional transition-based parser (zPar) BIBREF53 and the Berkeley constituency parser BIBREF8 have been used as additional training data. They retrained their neural network parser and the zPar parser on the extended training data. The neural network parser gained around 0.3% from the tri-training, and it outperforms the state-of-the-art accuracy by a large 1%. By contrast, their co-training evaluation on the zPar parser found only negative effects.
Self-training is another semi-supervised technique that only involves one learner. In a typical self-training iteration, a learner is firstly trained on the labelled data, and then the trained learner is used to label some unlabelled data. After that, the unlabelled data with the predictions (usually the high confident predictions of the model) are added to the training data to re-train the learner. The self-training iteration can also be repeated to do a multi-iteration self-training. When compared with co-training, self-training has a number of advantages. Firstly unlike the co-training that requires two to three learners, the self-training only requires one learner, thus it is more likely we can use the self-training than co-training in an under resourced scenario. Secondly, to generate the additional training data, co-training requires the unlabelled data to be double annotated by different learners, this is more time-consuming than self-training's single annotation requirement. In term of the previous work on parsing via self-training, charniak1997statistical first applied self-training to a PCFG parser, but this first attempt of using self-training for parsing failed. steedman2003semi implemented self-training and evaluated it using several settings. They used a 500 sentences training data and parsed only 30 sentences in each self-training iteration. After multiple self-training iterations, it only achieved moderate improvements. This is caused probably by the small number of additional sentences used for self-training.
mcclosky06naacl reported strong self-training results with an improvement of 1.1% f-score by using the Charniak-parser, cf. BIBREF7 . The Charniak-parser is a two stage parser that contains a lexicalized context-free parser and a discriminative reranker. They evaluated on two different settings. In the first setting, they add the data annotated by both stages and retrain the first stage parser on the new training set, this results in a large improvement of 1.1%. In the second setting, they retrain the first stage parser on its own annotations, the result shows no improvement. Their first setting is similar to the co-training as the first stage parser is retrained on the annotation co-selected by the second stage reranker, in which the additional training data is more accurate than the predictions of first stage parser. mcclosky2006reranking applied the same method later on out-of-domain texts which show good accuracy gains too.
reichart2007self showed that self-training can improve the performance of a constituency parser without a reranker for the in-domain parsing. However, their approach used only a rather small training set when compared to that of mcclosky06naacl.
sagae2010self investigated the contribution of the reranker for a constituency parser in a domain adaptation setting. Their results suggest that constituency parsers without a reranker can achieve statistically significant improvements in the out-of-domain parsing, but the improvement is still larger when the reranker is used.
In the workshop on syntactic analysis of non-canonical language (SANCL) 2012 shared task, self-training was used by most of the constituency-based systems, cf. BIBREF15 . The top ranked system is also enhanced by self-training, this indicates that self-training is probably an established technique to improve the accuracy of constituency parsing on out-of-domain data, cf. BIBREF43 . However, none of the dependency-based systems used self-training in the SANCL 2012 shared task.
One of the few successful approaches to self-training for dependency parsing was introduced by chen2008learning. They improved the unlabelled attachment score by about one percentage point for Chinese.chen2008learning added parsed sentences that have a high ratio of dependency edges that span only a short distance, i.e. the head and dependent are close together. The rationale for this procedure is the observation that short dependency edges show a higher accuracy than longer edges.
kawahara2008learning used a separately trained binary classifier to select reliable sentences as additional training data. Their approach improved the unlabelled accuracy of texts from a chemical domain by about 0.5%.
goutam2011exploring applied a multi-iteration self-training approach on Hindi to improve parsing accuracy within the training domain. In each iteration, they add a small number (1,000) of additional sentences to a small initial training set of 2,972 sentences, the additional sentences were selected due to their parse scores. They improved upon the baseline by up to 0.7% and 0.4% for labelled and unlabelled attachment scores after 23 self-training iterations.
While many other evaluations on self-training for dependency parsing are found unhelpful or even have negative effects on results. bplank2011phd applied self-training with single and multiple iterations for parsing of Dutch using the Alpino parser BIBREF54 , which was modified to produce dependency trees. She found self-training produces only a slight improvement in some cases but worsened when more unlabelled data is added.
plank2013experiments used self-training in conjunction with dependency triplets statistics and the similarity-based sentence selection for Italian out-of-domain parsing. They found the effects of self-training are unstable and does not lead to an improvement.
cerisara2014spmrl and bjorkelund2014spmrl applied self-training to dependency parsing on nine languages. cerisara2014spmrl could only report negative results in their self-training evaluations for dependency parsing. Similarly, bjorkelund2014spmrl could observe only on Swedish a positive effect.
The second group uses the unlabelled data indirectly. Instead of using the unlabelled data as training data, they incorporate the information extracted from large unlabelled data as features to the parser. Word clusters BIBREF37 , BIBREF55 and word embeddings BIBREF24 , BIBREF25 are most well-known approaches of this family. However, other attempts have also been evaluated, such as dependency language models (DLM) BIBREF56 .
Word Clustering is an unsupervised algorithm that is able to group the similar words into the same classes by analysing the co-occurrence of the words in a large unlabelled corpus. The popular clustering algorithm includes Brown BIBREF57 , BIBREF58 and the Latent dirichlet allocation (LDA) BIBREF59 clusters.
koo08 first employed a set of features based on brown clusters to a second-order graph-based dependency parser. They evaluated on two languages (English and Czech) and yield about one percentage improvements for both languages. The similar features have been adapted to a transition-based parser of bohnet2012emnlp. The LDA clusters have been used by cerisara2014spmrl in the workshop on statistical parsing of morphologically rich languages (SPMRL) 2014 shared tasks BIBREF60 on parsing nine different languages, their system achieved the best average results across all non-ensemble parsers.
Word embeddings is another approach that relies on the co-occurrence of the words. Instead of assigning the words into clusters, word embedding represent words as a low dimensional vector (such as 50 or 300 dimensional vector), popular word embedding algorithms include word2vec BIBREF61 and global vectors for word representation (GloVe) BIBREF62 . Due to the nature of the neural networks, word embeddings can be effectively used in the parsers based on neural networks. By using pre-trained word embeddings the neural network-based parsers can usually achieve a higher accuracy compared with those who used randomly initialised embeddings BIBREF24 , BIBREF25 , BIBREF33 .
Other Approaches that use different ways to extract features from unlabelled data have also been reported.
mirroshandel12 used lexical affinities to rescore the n-best parses. They extract the lexical affinities from parsed French corpora by calculating the relative frequencies of head-dependent pairs for nine manually selected patterns. Their approach gained a labelled improvement of 0.8% over the baseline.
chen2012utilizing applied high-order DLMs to a second-order graph-based parser. This approach is most close to the Chapter SECREF32 of this thesis. The DLMs allow the new parser to explore higher-order features without increasing the time complexity. The DLMs are extracted from a 43 million words English corpus BIBREF63 and a 311 million words corpus of Chinese BIBREF64 parsed by the baseline parser. Features based on the DLMs are used in the parser. They gained 0.66% UAS for English and an impressive 2.93% for Chinese.
chen2013feature combined the basic first- and second-order features with meta features based on frequencies. The meta features are extracted from auto-parsed annotations by counting the frequencies of basic feature representations in a large corpus. With the help of meta features, the parser achieved the state-of-the-art accuracy on Chinese.
Corpora
As mentioned previously, one contribution of this thesis is evaluating major semi-supervised techniques in a unified framework. For our main evaluation, we used English data from the conference on computational natural language learning (Conll) 2009 shared task BIBREF1 as our source of in-domain evaluation. For out-of-domain evaluation, we used weblogs portion of OntoNotes v5.0 corpus (Weblogs) and the first workshop on syntactic analysis of non-canonical language shared task data (Newsgroups,Reviews,Answers) BIBREF15 . Section SECREF35 introduces our main evaluation corpora in detail. For comparison and multi-lingual evaluation, we also evaluated some of our approaches in various additional corpora. Our self-training approach has been evaluated on chemical domain data (Chemical) from the conference on computational natural language learning 2007 shared task BIBREF14 and nine languages datasets from the workshop on statistical parsing of morphologically rich languages (Spmrl) 2014 shared task BIBREF60 . Our dependency language models approach has been evaluated in addition on Wall Street Journal portion of Penn English Treebank 3 (Wsj) BIBREF36 and Chinese Treebank 5 (Ctb) BIBREF65 . As both treebanks do not contain unlabelled data, we used the data of chelba13onebillion and the Xinhua portion of Chinese Gigaword Version 5.0 for our English and Chinese tests respectively. We introduce those corpora in the experiment set-up section of the relevant chapters.
The Main Evaluation Corpora
In this section, we introduce our main evaluation corpora that have been used in all of the semi-supervised approaches evaluated in this thesis.
The Conll English corpus built on the Penn English Treebank 3 BIBREF36 which contains mainly Wall Street Journals but also included a small portion of Brown corpus BIBREF66 . The training set contains only Wall Street Journals, the small subset of the Brown corpus has been included in the test set. The constituency trees from Penn English Treebank are converted to dependency representation by the LTH constituent-to-dependency conversion tool, cf. BIBREF67 . A basic statistic of the corpus can be found in Table TABREF36 .
For our Weblogs domain test we used the Ontonotes v5.0 corpus. The Ontonotes corpus contains various domains of text such as weblogs, broadcasts, talk shows and pivot texts. We used the last 20% of the weblogs portion of the Ontonotes v5.0 corpus as our target domain development set and the main test set. The selected subset allows us to build sufficient sized datasets similar to the source domain test set. More precisely, the first half of the selected corpus is used as a test set while the second half is used as the development set. Table TABREF37 shows some basic statistic of those datasets.
Newsgroups, Reviews and Answers domain data are used as additional test sets for our evaluation. Those additional test domains are provided by the first workshop on syntactic analysis of non-canonical language (SANCL) shared task BIBREF15 . The shared task is focused on the parsing English web text, in total, they prepared five web domain datasets, two of them are development datasets (Email, Weblogs) and the other three (Newsgroups, Reviews and Answers) are used as test sets. For each of the domains, a small labelled set and a large unlabelled set are provided. In this thesis, we used all three test datasets (both labelled and unlabelled data). In addition, one of the unlabelled texts (Weblogs) from the development portion of the shared task is also used. We used for each domain a similar sized unlabelled dataset to make the evaluation more unified. The only exception is the answers domain, as its unlabelled dataset is much smaller than the other three domains, thus we used all of the data provided. A basic statistic of the labelled test sets and unlabelled data can be found in Table TABREF37 and TABREF38 respectively.
In term of the dependency representation, we used the LTH conversion for our main evaluation corpora. Same as the CoNLL 2009 shared task we converted all the labelled data from constituent trees to dependency representation by the LTH constituent-to-dependency conversion tool BIBREF67 when needed.
Evaluation Methods
To measure the parser's performance, we report labelled attachment scores (LAS) and unlabelled attachment scores (UAS). For our evaluation on the main corpora, we use the official evaluation script of the CoNLL 2009 shared task, in which all punctuation marks are included in the evaluation. The LAS and UAS are the standard ways to evaluate the accuracy of a dependency parser. Due to the single-head property of the dependency trees, the dependency parsing can be seen as a tagging task, thus the single accuracy metric is well suited for the evaluation. Both LAS and UAS measure the accuracy by calculating the percentage of the dependency edges that have been correctly attached. The UAS considers an edge is correct if the attachment is correct, it does not take the label into account, while the LAS counts only the edges that are both correctly attached and the correct label also assigned. The LAS is more strict than UAS thus we mainly focus on LAS in our evaluation. Let INLINEFORM0 be the number of edges that are correctly attached, INLINEFORM1 be the number of edges that are both correctly attached and have the correct label, INLINEFORM2 be the total number of edges, we compute: DISPLAYFORM0
DISPLAYFORM0
For significance testing, we use the randomised parsing evaluation comparator from a major shared task on dependency parsing BIBREF14 . The script takes predictions annotated by two different models of the same dataset. Let the first input be the one which has a higher overall accuracy. The null hypothesis of the script is that the accuracy difference between the first input and the second input is not statistically significant. And the p-values represent the probability that the null hypothesis is correct. We use the script's default setting of 10,000 iterations ( INLINEFORM0 ), for each iteration, the comparator randomly selects one sentence from the dataset and compares the accuracies of the sentence predicted in the two different inputs. Let INLINEFORM1 be the number of randomly selected instances that are predicted less accurately in the first input when compared to the predictions in the second input. The p-value is calculated by: INLINEFORM2
We mark the significance levels based on their p-values, * for INLINEFORM0 , ** for INLINEFORM1 .
Analysis Techniques
To understand the behaviour of our methods, we assess our results on a number of tests. We analyse the results on both token level and sentences level. For token level, we focus on the accuracies of individual syntactic labels and the known/unknown words accuracies. For sentence level, we used the methods from mcclosky06naacl to evaluate sentences in four factors. We used all four factors from their analysis, i.e. sentence length, the number of unknown words, the number of prepositions and number of conjunctions.
Token Level Analysis. Our token level analysis consists of two tests, the first test assesses the accuracy changes for individual labels. The goal of this test is to find out the effects of our semi-supervised methods on different labels. For an individual label, we calculate the recall, precision and the f-score. Let INLINEFORM0 be the number of the label INLINEFORM1 predicted by the parser, INLINEFORM2 be the count of label INLINEFORM3 presented in the gold data and INLINEFORM4 be the number of the label predicted correctly. The precision ( INLINEFORM5 ), recall ( INLINEFORM6 ) and the f-score ( INLINEFORM7 ) are calculated as follows: DISPLAYFORM0
DISPLAYFORM0
DISPLAYFORM0
We compute for each label, the score differences between our enhanced model and the base model. The results for the most frequent labels are visualised by the bar chart. Figure FIGREF45 is an example of the bar chart we used, the x-axis shows the names of the relevant label, the y-axis shows the accuracy changes in percentage. For each of the labels, we report the accuracy changes of all three scores (recall, precision and f-score), the left (blue) bar represents the recall, the middle (red) bar represents the precision and the right (brown) bar is for f-score.
The second test assesses the overall accuracy of known words and unknown words. The unknown words are defined as the words that are not presented in the initial training set. The initial training set is the one we used to train the base model. To compute the accuracy for known and unknown words, we first assign all the tokens in the dataset into two groups (known and unknown) and then we calculate the labelled and unlabelled accuracies for each of the groups separately. We compare the improvements achieved by our enhanced model on known and unknown words to understand the ability of our model on handling unknown words.
Sentence Level Analysis. For our sentence level analysis, we evaluate on four factors (sentence length, the number of unknown words, the number of prepositions and the number of conjunctions) that are known to be problematic in parsing. We use a method similar to mcclosky06naacl in our analysis. For each of the factors, we assign sentences to different classes according to their property, sentences that have the same property are assigned to the same class. Take unknown words factor as an example, sentences which contain the same number of unknown words are grouped together. For each group, we calculate the percentage of sentences that are improved, worsened or unchanged in accuracy by our enhanced model. The reason for using the percentage instead of the number of sentences that were used by mcclosky06naacl is mainly because the absolute numbers vary greatly both within the factor and between factors, thus is not suitable for comparison. The percentage, on the other hand, can be easily compared. In addition to the above values, we also report the number of the sentences in each class. Figure FIGREF46 shows an example of our sentence level analysis on the different number of unknown words per sentence. The x-axis shows the conditions of the classes. In this example, it represents the different number of unknown words in a single sentence. The y-axis to the left is the percentage and the y-axis to the right is the number of sentences. The blue dashed line represents the percentage of the sentences that are parsed better by our enhanced model, the red dotted line represent the portion that is parsed less accurate, the black dash-dotted line shows the portion of sentences whose accuracy are unchanged. The black solid line is the number of sentences in the individual classes.
Co-training
In this chapter, we introduce our co-training approach. Co-training is one of the popular semi-supervised techniques that has been applied to many natural language processing tasks, such as named entity recognition BIBREF68 , constituency parsing BIBREF16 and dependency parsing BIBREF17 , BIBREF15 . Although co-training approaches are popular, they do not always bring positive effects BIBREF18 , BIBREF25 . Improvements on results are usually reported by learners that are carefully designed to be as different as possible. Such as in sagae07's approach, they form the co-training with parsers consisting of different learning algorithms and search strategies. However, off-the-shelf parsers use many similar features, the output of these parsers are more likely to agree with each other. Thus it is unclear whether the off-the-shelf parsers are suitable for co-training.
In this work we evaluate co-training with a number of off-the-shelf parsers that are freely available to the research community, namely Malt parser BIBREF10 , MST parser BIBREF9 , Mate parser BIBREF12 , and Turbo parser BIBREF11 . We evaluate those parsers on agreement based co-training algorithms. The evaluation learner is retrained on the training set that is boosted by automatically annotated sentences agreed by two source learners. We investigate both normal agreement based co-training and a variant called tri-training. In a normal co-training setting the evaluation learner is used as one of the source learners, and in a tri-training scenario, the source learners are different from the evaluation learner.
In the following sections we introduce our approaches in Section SECREF15 . We then introduce our experiment settings and results in Section SECREF16 and Section SECREF17 respectively. After that, in Section SECREF18 we analyse the results and trying to understand how co-training helps. In the last section (Section SECREF19 ), we summarise our finding.
Agreement Based Co-training
In this work, we apply an agreement based co-training to out-of-domain dependency parsing. Our agreement based co-training is inspired by the observation from sagae07 in which the two parsers agreeing on an annotation is an indication of a higher accuracy. We proposed two types of agreement based approaches: one uses parser pairs (normal co-training), the other uses three parsers which is also known as tri-training.
Two approaches use a similar algorithm, which involves two source learners and one evaluation learner. Two source learners are used to produce additional training data for retraining the evaluation learner. More precisely, our algorithm is as follows:
Although both approaches share the similar algorithm, the major differences between them are: both parsers involved by normal co-training are used as the source learners, in which one of them is also used as the evaluation learner; by contrast, tri-training uses three parsers in total, in which two of them are used as the source learners and the third one is used as the evaluation learner.
In terms of parsers selection, we selected four public available dependency parsers, which include two benchmark parsers (Malt parser BIBREF10 and MST parser BIBREF9 ), one transition-based Mate parser BIBREF12 , and one graph-based Turbo parser BIBREF11 . These parsers have been widely used by researchers. A more detailed discussion of the dependency parser can be found in section SECREF8 .
The agreement based co-training depends on the assumption that identical annotations between two learners indicate the correctness. To confirm the suitability of selected parsers, in the preliminary evaluation we assessed the accuracy of identical analysis generated by parser pairs. Because we intend to use the Mate parser as our evaluation parser, we paired each of the other three parsers with Mate parser to create three co-training pairs. We assess our assumption by annotating our Weblogs development set, the development set is parsed by all four parsers. We then extract the identical annotations (whole sentence) from parser pairs. We show the accuracy of individual parsers and the accuracy of identical annotations in Table TABREF51 . The second row shows the labelled accuracy of each parser on the Weblogs development set. The third row shows the labelled accuracy of the identical annotations between the named parser and Mate parser. The fourth row shows the agreement rate of the parser pairs. The last row shows the average sentence length of the identical annotations. As we can see from the table, our assumption is correct on all the parser pairs. Actually, when they agreed on the annotations, the accuracies can be 16% higher than that of individual parsers. However, we also noticed that the average sentence length of the identical annotations is in stark contrast with that of the entire development set (19.6 tokens/sentence). We will discuss this potential conflict in the later section.
Experiment Set-up
In our evaluation on co-training we use our main evaluation corpora that consists of a source domain training set (Conll), a Weblogs domain development set, a in-domain test set (Conll) and four out-of-domain test sets (Weblogs, Newsgroups, Reviews and Answers). For each target domains, we used in addition a large unlabelled dataset to supply the additional training set. We evaluate various different settings on the development set to tune the best configuration, after that, we apply the best setting to all the test domains.
As mentioned before, we used four parsers in our experiments. cf. the Malt parser BIBREF10 , MST parser BIBREF9 , Mate parser BIBREF12 , and the Turbo parser BIBREF11 . We use the default settings for all the parsers. The part-of-speech tags is annotated by Mate parser's internal tagger.
To create the additional training corpus, the unlabelled datasets are annotated by all the parsers which are trained on the Conll source domain training set. The Mate parser is used as our evaluation learner, the baseline for all the domains are generated by Mate parser trained on the same Conll training set and applied directly to target domains.
We mainly report the labelled attachment scores (LAS), but also include the unlabelled attachment scores (UAS) for our evaluations on test sets. We mark the significance levels according to the p-values, * indicates significance at the p < 0.05 level, ** for the p < 0.01 level.
For our evaluation on self-training, we used our main evaluation corpora and the Chemical domain text from the domain adaptation track of CoNLL 2007 shared task. We mainly evaluated on our main evaluation corpora and the best setting is also tuned on the development set of the main evaluation corpora. The Chemical domain evaluation is only used for comparison with previous work, we do not optimise our approaches specifically for this domain.
For the main evaluation corpora, we used the Conll source domain training set, the Weblogs domain development set, the Conll source domain test set and Weblogs, Newsgroups, Reviews domain test sets. We do not evaluate our approach on the Answers domain as the unlabelled data for this domain is not large enough for our self-training.
The evaluation corpus for Chemical domain is taken from the domain adaptation track of the CoNLL 2007 shared task BIBREF14 . The shared task is the second year running for the dependency parsing task. Besides the multi-lingual parsing track introduced from the previous year, the 2007 shared task also included a track on domain adaptation task. The domain adaptation track provided mainly two domains (Biomedical and Chemical), in which the biomedical domain is used as development set and the chemical domain is used as evaluation set. The source domain training set consists of sections 2-11 of the Wall Street Journal section of the Penn Treebank BIBREF36 . A sufficient size of unlabelled data are also provided by the organiser, we used the first 256k sentences in our work. The labelled data are converted to dependency relations by the LTH constituent-to-dependency conversion tool BIBREF67 . Table TABREF79 shows the basic statistics of the training, development and the test set. For the Chemical domain test we used only the data from the CoNLL 2007 shared task to make a fair comparison with kawahara2008learning's results.
We use the Mate transition-based parser in our experiments. The parser is modified to output the confidence scores, other than that we used its default settings. For part-of-speech tagging, we use predicted tags from Mate's internal tagger for all the evaluated domains. For Chemical domain we evaluated additionally on gold tags as they are used by previous work. The baselines are trained only on the respective source domain training data.
For the evaluation of the parser's accuracy, we report both labelled (LAS) and unlabelled (UAS) attachment scores, but mainly focus on the labelled version. We included all punctuation marks in the evaluation. The significance levels are marked according to the p-values, * and ** are used to represent the p-value of 0.05 and 0.01 levels respectively.
We evaluate our adjusted parse score-based self-training approach with the Spmrl multi-lingual corpora. The Spmrl multi-lingual corpora consist of nine languages (Arabic, Basque, French, German, Hebrew, Hungarian, Korean, Polish, Swedish) in-domain datasets available from 2014 Shared Task at the workshop on statistical parsing of morphologically rich languages (SPMRL), cf. BIBREF60 . We have chosen the datasets as there are no multi-lingual out-of-domain corpora available. Actually, even the in-domain corpora for many languages are rather small. We used the 5k smaller training set from the shared task, to make the scenario similar to the domain adaptation task that assumes a small number of target domain data is available. This setting is also a good basis for exploration for improving parsing accuracy of under-resourced languages. For each language, the shared task also provided a sufficient unlabelled data which is required by our evaluation. We evaluate nine languages in a unified setting, in which the 5k training set and a 100k unlabelled dataset are used for all the languages. For additional training set, we parse all 100k sentences for each of the languages and use 50k of them as the additional training set. For tuning the INLINEFORM0 value of our adjusted parse score-based method, we used only the German development set, as we intend to use a unified setting for all languages and the German development set is the largest in size. Table TABREF103 shows statistics about the corpora that we used in our experiments.
We evaluate all nine languages on the Mate parser BIBREF12 , the default settings are used in all the experiments. To output the confidence scores we slightly modified the parser, however, this does not affect the parser's accuracy. For part-of-speech tagging, we use the Mate parser's internal tagger for all the evaluations. The baselines are obtained from models trained only on the 5k initial training data.
We report both labelled (LAS) and unlabelled (UAS) attachment scores, and mainly focus on the labelled accuracy. In line with the shared task official evaluation method, we include all the punctuations in our evaluation. The statistically significance levels are marked according to their p-values, (*) p-value < 0.05, (**) p-value < 0.01.
For our experiments on English in-domain text, we used the Wall Street Journal portion (Wsj) of the Penn English Treebank BIBREF36 . The constituency trees are converted to the Stanford style dependency relations. The Stanford conversion attracts more attention during the recent years, it has been used in the SANCL 2012 shared tasks BIBREF15 and many state-of-the-art results were also reported using this conversion BIBREF25 , BIBREF78 , BIBREF33 . We follow the standard splits of the corpus, section 2-21 are used for training, section 22 and 23 are used as the development set and the test set respectively. We used the Stanford parser v3.3.0 to convert the constituency trees into Stanford style dependencies BIBREF79 . For unlabelled data, we used the data of chelba13onebillion which contains around 30 million sentences (800 million words) from the news domain. Table TABREF126 shows the basic statistics about the corpus;
In addition to the Wsj corpus, we also evaluate our approach on the main evaluation corpus of this thesis. Our main evaluation corpus consists of a Conll source domain training set, a source domain test set and four target domain test sets (Weblogs, Newsgroups, Reviews and Answers). Unlike our Wsj corpus that uses Stanford dependencies, the main evaluation corpus is based on the LTH conversion BIBREF67 . Experimenting on different conversions and domains allow us to evaluate our method's robustness. For unlabelled data, we use the same dataset as in our Wsj evaluation.
For Chinese, we evaluate our approach only on the in-domain scenario, this is due to the lack of out-of-domain corpus. We use Chinese Treebank 5 (CTB5) BIBREF65 as the source of our gold standard data. The Chinese Treebank 5 corpus mainly consists of articles from Xinhua news agency but also contains some articles from Sinorama magazine and information services department of HKSAR. We follow the splits of zhang11, the constituency trees are converted to dependency relations by the Penn2Malt tool using head rules of zhang08. We use the Xinhua portion of Chinese Gigaword Version 5.0 as our source for unlabelled data. We noticed that the unlabelled data we used actually contains the Xinhua portion of the CTB5; to avoid potential conflict we removed them from the unlabelled data. After the pre-processing, our Chinese unlabelled data consists of 20 million sentences which are roughly 450 million words. We use ZPar v0.7.5 as our pre-processing tool. The word segmentor of ZPar is trained on the CTB5 training set. Table TABREF128 gives some statistics about the corpus.
We use a modified version of the Mate transition-based parser in our experiments. We enhance the parser with our DLM-based features; other than this we used the parser's default setting. The part-of-speech tags are supplied by Mate parser's internal tagger. The baselines are trained only on the initial training set. In most of our experiments, DLMs are extracted from data annotated by the base model of Mate parser. For the evaluation on higher quality DLMs, the unlabelled data is additionally tagged and parsed by Berkeley parser BIBREF8 and is converted to dependency trees with the same tools as for gold data.
We report both labelled (LAS) and unlabelled (UAS) attachment scores for our evaluation. The punctuation marks are excluded for our English and Chinese in-domain evaluations. For English evaluation on our main evaluation corpus we include the punctuations. The significance levels are marked due to their p-values, we use * and ** to represent the p-value of 0.05 and 0.01 levels respectively.
Empirical Results
Agreement based co-training. We first evaluate the parser pairs on the normal agreement based co-training. Each of the other three parsers is paired with Mate parser to be the source learners of our co-training. For each pairwise parser combinations, the unlabelled Weblogs text is double parsed by the parser pairs. The sentences that are annotated identically by both parsers are used as candidates for the additional training set. We take different amount of additional training sentences from the candidates pool to retrain the Mate parser. Figure FIGREF52 shows the co-training results of adding 10k to 30k additional training data for all three parser pairs. As we can see from the figure, all the co-training results achieved improvements when compared with the Mate baseline. The largest improvement of one percentage point is achieved by Mate-Malt parser pair when adding 20k or 30k additional training data. We also notice a negative correlation between the improvement and the identical rate mentioned previously in Table TABREF51 . The Turbo parser has the highest identical rate, in which it annotated 479 out of 2150 sentences (22.28%) exactly the same as Mate parser when evaluated on the development set. This is 2% higher than that of MST parser and 2.5% higher than the Malt parser. However, the improvements achieved by the pairs are shown to be negatively correlated, i.e. the Mate-Malt pair gains the largest improvement, the Mate-Turbo pair achieved the lowest gain. This finding is in-line with the fundamental of co-training that requires the learners to be as different as possible.
Removing short sentences from identical data. The identical annotations between the parsers are like a double-edged sword, they consist of a higher accuracy but in the same time shorter in average sentence length. Take our Mate-Malt pair as an example, the average sentence length of the identical annotations is only 8 tokens, this is much lower than the development set's 19.6 tokens/sentence and the Conll training set's 24.4 tokens/sentence. To make the additional training data more similar to the manually annotated data, we exclude the extremely short sentences from the pool. More precisely we set three minimal sentence length thresholds (4, 5 and 6 tokens), sentences shorter than the thresholds are removed from the pool. We then take 30k sentences from the remaining pool as the additional training data. By taking out the short sentences the average sentence length of the selected sentences is closer to that of the development set. As shown in Table TABREF53 , the average sentence length reached 13 tokens/sentence. One of the major concerns when we exclude the short sentences from the pool is that the accuracy of the remaining pool might drop. The short sentences are easier to parse, thus they usually have a higher accuracy. However, an evaluation on the development set shows that there is almost no effect on the accuracies (see Table TABREF53 ). In term of the results, we gained a 0.27% additional improvement when discarding short sentences (Figure FIGREF54 ).
Three learners co-training. In the normal co-training setting, the Mate parser is used as one of the source learners to provide additional training data for retraining itself. Based on this setting the Mate parser can learn only from the annotations it has already known. The tri-training algorithm is on the other hand designed to allow the evaluation learner to learn from sources other than itself. This gives the Mate parser the potential to explore novel examples from other parsers. In our tri-training experiments, we used the Malt parser and the MST parser as our source learners. The sentences that are annotated identically by these parsers are added to the pool for retraining the Mate parser. To assess the quality of the identical annotations between Malt and MST parsers we apply them to our development set. We also assessed the sentences that are annotated identically by Malt and MST parsers but different to Mate parser's annotation, this allows us to know the scale of the novel examples. As shown in Table TABREF55 , the accuracy of the sentences agreed by Malt and MST parsers is even slightly higher than that of Mate and Malt parsers, this is surprising as MST parser is less accurate than Mate parser. The analysis also showed that half of the identical annotations from Malt and MST parsers are actually novel to Mate parser. We compared our tri-training and co-training results in Figure FIGREF56 , the tri-training results constantly outperform the normal co-training. The best result of 79.12% is achieved by retraining the Mate parser with 20k additional training data agreed by Malt-MST parsers (tri-training). The best tri-training result is 0.24% higher than that of co-training and nearly 1.6% higher than the Mate baseline.
Evaluating on test domains. We then evaluated our best configuration (tri-training) on our four test domains. Under the tri-training setting, the unlabelled datasets of each domain are double parsed by Malt-MST pairs, the first 20k identical annotations are used as additional training data to retrain the Mate parser. The only exception is for answers domain. Due to the lack of unlabelled data the additional training data is much smaller, we used all 3k identical sentences for retraining. Table TABREF57 shows our tri-training results accompanied by the baselines. The tri-training setting achieved large labelled improvements up to 1.8 percentage points. For unlabelled attachment scores, the models gained up to 0.59% absolute improvements. We also tested the retrained Weblogs domain model on the in-domain test set. The results show the tri-trained model does not affect the in-domain accuracy.
Random Selection-based Self-training. To have an idea of the performance of basic self-training, we first evaluated with randomly selected additional training data. The triangle marked curve in Figure FIGREF80 shows the accuracy of the random selection-based self-training. We used from 50k to 200k randomly selected additional training data to retrain the Mate parser. The retrained models obtain some small improvements when compared with the baseline. The improvements achieved by the different number of additional training data are very similar: they all around 0.2%. Those small improvements obtained by the basic self-training are not statistically significant. This finding is in line with previous work of applying non-confidence-based self-training approaches to dependency parsing, cf. BIBREF55 , BIBREF70 .
Parse Score-based Self-training. We then evaluate with our first confidence-based method, that uses parse scores. As proposed the automatically annotated sentences are ranked in descending order by the adjusted parse scores before they are used as additional training data. As shown in Figure FIGREF80 , we add between 50k to 300k top ranked sentences from the Weblogs auto-annotated dataset. The method achieved 0.52% improvement when we use 50k additional training data and the improvement increased to 0.66% when 250k sentences are used. After that, the improvement decreased. We use an auto-labelled dataset of 500k sentences. After we rank the sentences by our confidence-based methods, the first half is expected to have an accuracy higher than the average, and the second half is expected to have one lower than average. Thus we should avoid using sentences from the second half of the ranked dataset.
Delta-based self-training. For our Delta-based approach, we select additional training data with the Delta method. We train the parser by adding between 50k to 300k sentences from the target domain. Same as the parse score-based method, we gain the largest improvement when 250k sentences are used, which improves the baseline by 0.73% (cf. Figure FIGREF80 ). Although this improvement is slightly higher than that of the parse score-based method, the accuracies are lower than the baseline when we use 50k and 100k ranked sentences from Delta based method. Our error analysis shows that these parse trees are mainly short sentences consisting of only three words. These sentences contribute probably no additional information that the parser can exploit.
Evaluating on test domains. We adapt our best settings of 250k additional sentences for both approaches and apply them to three test sets (Weblogs, Newsgroups and Reviews). As illustrated in Table TABREF81 , nearly all the results produced by both approaches are statistically significant improvements when compared to the baselines. The only exception is the unlabelled improvement of the parse score approach on Reviews domain which has a p-value of 0.08. Both approaches achieved the largest improvements on Weblogs domain. The largest labelled improvement of 0.81% is achieved by the parse score-based method, while the largest unlabelled improvement of 0.77% is achieved by the Delta method. For Newsgroups domain both approaches gained the similar labelled and unlabelled improvements of 0.6%. For Reviews domain the Delta method achieved 0.4 - 0.5% improvements on labelled and unlabelled accuracies. The parse score-based approach achieved lower improvements of 0.3%. In terms of the in-domain evaluation, the accuracies of both approaches are lower than the baseline.
We further evaluate our best settings on Chemical texts provided by the CoNLL 2007 shared task. We adapt the best settings of the main evaluation corpora and apply both confidence-based approaches to the Chemical domain. For the constant INLINEFORM0 , we use 0.015 and we use 125k additional training data out of the 256k from the unlabelled data of the Chemical domain. We evaluate our confidence-based methods on both predicted and gold part-of-speech tags. After retraining, both confidence-based methods achieve significant improvements in all experiments. Table TABREF82 shows the results for the Chemical domain. When we use predicted part-of-speech tags, the Delta-based method gains a labelled improvement of 1.42%, while the parse score-based approach gains 1.12%. For the experiments based on gold tags, we achieved larger labelled improvements of 1.62% for the Delta-based and 1.48% for the parse score-based methods. For all experiments, the unlabelled improvements are similar to that of labelled ones.
Table TABREF82 compares our results with that of kawahara2008learning. We added also the results of sagae07 but those are not directly comparable since they were gained with co-training. sagae07 gained additional training data by parsing the unlabelled data with two parsers and then they select those sentences where the parsers agree on.
kawahara2008learning reported positive results for self-training. They used a separately trained binary classifier to select additional training data and are evaluated only on gold tags. Our baseline is higher than kawahara2008learning's self-training result. Starting from this strong baseline, we could improve by 1.62% LAS and 1.52% UAS which is an error reduction of 9.6% on the UAS (cf. Table TABREF82 ). The largest improveme

Question: Which English domains do they evaluate on?
Answer: Conll, Weblogs, Newsgroups, Reviews, Answers