Abstract: In this paper, we study semantic role labelling (SRL), a subtask of semantic parsing of natural language sentences and its application for the Vietnamese language. We present our effort in building Vietnamese PropBank, the first Vietnamese SRL corpus and a software system for labelling semantic roles of Vietnamese texts. In particular, we present a novel constituent extraction algorithm in the argument candidate identification step which is more suitable and more accurate than the common node-mapping method. In the machine learning part, our system integrates distributed word features produced by two recent unsupervised learning models in two learned statistical classifiers and makes use of integer linear programming inference procedure to improve the accuracy. The system is evaluated in a series of experiments and achieves a good result, an $F_1$ score of 74.77%. Our system, including corpus and software, is available as an open source project for free research and we believe that it is a good baseline for the development of future Vietnamese SRL systems.

Introduction: In this paper, we study semantic role labelling (SRL), a subtask of semantic parsing of natural language sentences. SRL is the task of identifying semantic roles of arguments of each predicate in a sentence. In particular, it answers a question Who did what to whom, when, where, why?. For each predicate in a sentence, the goal is to identify all constituents that fill a semantic role, and to determine their roles, such as agent, patient, or instrument, and their adjuncts, such as locative, temporal or manner. Figure 1 shows the SRL of a simple Vietnamese sentence. In this example, the arguments of the predicate giúp (helped) are labelled with their semantic roles. The meaning of the labels will be described in detail in Section "Building a Vietnamese PropBank" . SRL has been used in many natural language processing (NLP) applications such as question answering BIBREF0 , machine translation BIBREF1 , document summarization BIBREF2 and information extraction BIBREF3 . Therefore, SRL is an important task in NLP. The first SRL system was developed by Gildea and Jurafsky BIBREF4 . This system was performed on the English FrameNet corpus. Since then, SRL task has been widely studied by the NLP community. In particular, there have been two shared-tasks, CoNLL-2004 BIBREF5 and CoNLL-2005 BIBREF6 , focusing on SRL task for English. Most of the systems participating in these shared-tasks treated this problem as a classification problem which can be solved by supervised machine learning techniques. There exists also several systems for other well-studied languages like Chinese BIBREF7 or Japanese BIBREF8 . This paper covers not only the contents of two works published in conference proceedings BIBREF9 (in Vietnamese) and BIBREF10 on the construction and the evaluation of a first SRL system for Vietnamese, but also an extended investigation of techniques used in SRL. More concretely, the use of integer linear programming inference procedure and distributed word representations in our semantic role labelling system, which leads to improved results over our previous work, as well as a more elaborate evaluation are new for this article. Our system includes two main components, a SRL corpus and a SRL software which is thoroughly evaluated. We employ the same development methodology of the English PropBank to build a SRL corpus for Vietnamese containing a large number of syntactically parsed sentences with predicate-argument structures. We then use this SRL corpus and supervised machine learning models to develop a SRL software for Vietnamese. We demonstrate that a simple application of SRL techniques developed for English or other languages could not give a good accuracy for Vietnamese. In particular, in the constituent identification step, the widely used 1-1 node-mapping algorithm for extracting argument candidates performs poorly on the Vietnamese dataset, having $F_1$ score of 35.93%. We thus introduce a new algorithm for extracting candidates, which is much more accurate, achieving an $F_1$ score of 84.08%. In the classification step, in addition to the common linguistic features, we propose novel and useful features for use in SRL, including function tags and distributed word representations. These features are employed in two statistical classification models, maximum entropy and support vector machines, which are proved to be good at many classification problems. In order to incorporate important grammatical constraints into the system to improve further the performance, we combine machine learning techniques with an inference procedure based on integer linear programming. Finally, we use distributed word representations produced by two recent unsupervised models, the Skip-gram model and the GloVe model, on a large corpus to alleviate the data sparseness problem. These word embeddings help our SRL software system generalize well on unseen words. Our final system achieves an $F_1$ score of 74.77% on a test corpus. This system, including corpus and software, is available as an open source project for free research and we believe that it is a good baseline for the development of future Vietnamese SRL systems. The remainder of this paper is structured as follows. Section "Existing English SRL Corpora" describes the construction of a SRL corpus for Vietnamese. Section "Vietnamese SRL System" presents the development of a SRL software, including the methodologies of existing systems and of our system. Section "Evaluation" presents the evaluation results and discussion. Finally, Section "Conclusion" concludes the paper and suggests some directions for future work.

Question: How are EAC evaluated?
Answer: Qualitatively through efficiency, effectiveness and satisfaction aspects and quantitatively through metrics such as precision, recall, accuracy, BLEU score and even human judgement.