Transition Based Dependency Parser for Amharic Language Using Deep Learning

Abstract: Researches shows that attempts done to apply existing dependency parser on morphological rich languages including Amharic shows a poor performance. In this study, a dependency parser for Amharic language is implemented using arc-eager transition system and LSTM network. The study introduced another way of building labeled dependency structure by using a separate network model to predict dependency relation. This helps the number of classes to decrease from 2n+2 into n, where n is the number of relationship types in the language and increases the number of examples for each class in the data set. Evaluation of the parser model results 91.54 and 81.4 unlabeled and labeled attachment score respectively. The major challenge in this study was the decrease of the accuracy of labeled attachment score. This is mainly due to the size and quality of the tree-bank available for Amharic language.  Improving the tree-bank by increasing the size and by adding morphological information can make the performance of parser better.

Reviews and author responses:

The paper describes a dependency parser for Amharic text trained on the Yimam et al. 2018 treebank.

The proposed method is an unlabelled arc-eager transition-based dependency parser followed by a dependency label classifier. Both models are based on LSTM architectures. Unfortunately there is no clear description of these models, which makes it difficult to understand their structure.

The experimental section contains a questionable selection on the train/test split (which should have been done blindly before looking at the results) and lacks any baseline comparison with previous approaches. The authors argue that performing arc prediction and arc labeling in two separate stages is beneficial, but they don't compare with a method that perform both actions at the same time.

The motivation of the paper is also questionable: the introduction section describes some of the peculiarities of the Amharic language, such as being morphologically rich, but the proposed method seems completely generic and does not address or exploits any of these peculiarities. If the authors want to argue that Amharic would benefit from specialized parser architecture, the methods and results reported in this paper fail to provide evidence for this position.

The citation format is also broken.


The paper claims to build a transition-based dependency parser for Amharic. To produce a parse tree, the paper takes a two-stage approach, first producing the structure of the tree, followed by annotating each individual arcs on the tree. The paper also explores attention and finds it useful for improving the performance.

I am giving a score 1, because the paper lacks precision and the contribution is not significant.

The major problem of the paper is precision. There is a lack of formal descriptions of the parser, e.g., the representation of the input and output, what each individual block in the figures are doing, how the arc-eager transition parser works. The text description is certainly helpful, but without formal descriptions, it is impossible to reproduce the results in the paper. Besides, without setting up a precise language/notation, the paper is not easy to follow.

In terms of contribution, the paper begins with the motivation that Amharic has certain language characteristics that are absent in languages, such as English, where most research effort has been spent. However, the paper still uses a parser for English, and does not utilize the special features of Amharic. Generic architectural modification, such as adding attention, has been explored, but this has little contribution on its own and has little to do with Amharic.

In addition to the above, other minor weaknesses of the paper includes typos, poor experimental practices, missing citations and related work, and other wording issues.

In particular, choosing the split of the data set that achieves the best result, as quoted below, is concerning.

"We experimented on 60/40, 70/30 and 80/20 train-test splitting ratios ... Therefore, we select 70/30 train-test splitting ratio throughout the experiments."

* overview 
- This paper describes a model for transition-based dependency parser, and tested on the Amharic treebank. It proposes a modification that is different from the common practice of combining the transition with dependency relation, namely to predict the transitions first, then the dependency relations.
    
* strengthens
 - Unfortunately, I can't find any strength in this paper. It is more like a course project than a research paper for ICLR.

* weaknesses
- The main idea of first unlabeled parsing then predicting labels is hardly novel, and it generally won't help, since the predicted labels could be useful features for determining the transitions. There is also no experiment to compare the proposed model to normal parsing model with 2n+2 predictions to support the author's claim.
- Generally, the architecture of the model is poorly explained. For example, it seems that the "embeddings" for the words are just one-hot index vectors (since the number of dimensions are the same as the vocabulary size), which is not what we mean by embeddings. It is also unclear what the dot product of the embeddings means. The attention layer is again not explained at all, e.g. what are the input and output.
- The splitting of train/test set does not make sense, the selection should not be influenced by the results at all. There is no reason not to use the standard split in UD. 
- I don't see what is the challenge in the discussion section, since UAS is by definition lower than LAS.
- There numerous typos and grammar mistakes, and the citation format is broken.
- In many places, the quoted presumably Amharic words are not shown.

Meta-review: The paper builds a transition-based dependency parser for Amharic, first predicting transitions and then dependency labels. The model is poorly motivated, and poorly described. The experiments have serious problems with their train/test splits and lack of baseline. The reviewers all convincingly argue for reject. The authors have not responded. 

Decision: reject