Türkçe eşgönderge çözümlemesi
Turkish coreference resolution
Yazar: TUĞBA PAMAY
Danışman: DR. ÖĞR. ÜYESİ GÜLŞEN ERYİĞİT
İstanbul Teknik Üniversitesi / Fen Bilimleri Enstitüsü / Bilgisayar Mühendisliği Ana Bilim Dalı / Bilgisayar Mühendisliği Bilim Dalı
Derece: Yüksek Lisans, 2018
Konu: Bilgisayar Mühendisliği Bilimleri-Bilgisayar ve Kontrol = Computer Engineering and Computer Science and Control
Dizin: Adıl çözümleme = Pronoun resolution

Özet:
Bu çalışmada Türkçe Eşgönderge Çözümleme (EÇ) problemi üzerinde veri öğrenmesine dayalı farklı algoritmik yaklaşımlar incelenmiştir. Literatürde sıklıkla geçen ifade-çifti ve varlık-ifade modelleri yaklaşımlarıyla, önerilen yeni öznitelik değerleri kullanılarak iki farklı sistem mimarisi oluşturulmuştur. Etiketli veri kümesi olarak, Türkçe EÇ çalışmaları için yakın geçmişte oluşturulan Marmara Türkçe Eşgönderge Derlemi (MTED) (Schuller, 2017) kullanılmıştır.

İfade-çifti modelleri temelde ifade ikilileri üzerinde çalışır ve mevcut çifti oluşturan ifadelerin birbirleriyle eşgöndergesel olup olmadığına, daha önce verdiği kararlardan bağımsız olarak karar vermektedir. Bu model, sonişleme aşaması olarak bir kümeleme mekanizmasına ihtiyaç duymaktadır. Bu sonişleme sırasında, ifadelerin eşgönderge durumları hakkında sınıflandırıcının ürettiği çıktılar üzerinden ilgili ifade kümeleri oluşturulmaktadır. Bu tez çalışmasında, öğrenme tabanlı bir ifade-çifti modeli oluşturulmuştur. İlk aşama olan ifade çiftlerinin eşgönderge durumlarının saptanması işlemi için iki farklı sınıflandırıcı (Karar Ağacı (KA) ve Destek Vektör Makinesi (DVM)) ile çalışılmış, ifade kümelerinin oluşturulması aşamasında ise kural tabanlı bir sonişlemciden yararlanılmıştır. İlk olarak, farklı negatif örnekleme yöntemi ve kullanılan DVM algoritmasının çekirdek fonksiyonunun parametrelerinin etkileri araştırılmıştır. Bu incelemeden elde edilen sonuçlar kullanılarak, farklı öznitelik seçimlerinin problem üzerindeki etkileri incelenmiş ve ortaya çıkarılan ifade-çifti modeli yaklaşımıyla Türkçe EÇ üzerinde en başarılı sonuçlar elde edilmiştir.

Varlık-ifade modelleri, bir ifade ile ifade kümesinin eşgönderge ilişkisi üzerinde bir karar vermektedir. Bu modelde ifade kümelerinin oluşturulması için bir sonişlemciye ihtiyaç duyulmamaktadır; çünkü ifade kümeleri modelin tahmin aşamasında dinamik şekilde artarak oluşturulmaktadır. Tez çalışması kapsamında bir varlık-ifade modeli geliştirilmiş, ifade-çifti modelinin geliştirilmesi aşamasında elde edilen en iyi öznitelikler bu modele aktarılmış ve etkileri incelenmiştir.

Sistemin değerlendirilmesi, referans ifadeler üzerinde MUC, $B^3$, BLANC ve LEA değerlendirme ölçütleri kullanılarak yapılmıştır. İfade-çifti modelinde elde edilen %90.68 (MUC), %86.89 (B3), %85.13 (BLANC) ve %78.34 (LEA) F1 ölçütüyle, mevcut Türkçe EÇ sistemi (Schuller, 2017) üzerinde sırasıyla 9.12, 16.06, 13.08 ve 12.57 yüzde puanlık bir gelişim sağlanmaktadır. 

Varlık-ifade modelinde ise sırasıyla %75.83 (MUC), %62.37 ($B^3$), %57.15 (BLANC) ve %50.69 (LEA) F1 ölçütleri elde edilmiştir. Gerçeklenen varlık-ifade modelinde elde edilen sonuçlar ile ifade-çifti modelinin başarısı geçilememiştir; ancak bu yöntemi kullanan Türkçe EÇ çalışmaları için temel bir sonuç oluşturulmuştur.

Abstract:
Coreference resolution (CR) is the task of determining sets of
mentions which identify the same real-world entity (e.g. a
person, a place, a thing, an event). CR is one of the natural
language processing (NLP) tasks at semantic level. Resolving
textual mentions as a pre-processing step helps in several
higher level NLP applications. For example, machine translation,
text summarization, information extraction, question
answering are shown as examples of such applications getting
benefit of CR systems.

Studies categorized as linguistic or theoretical approaches in
the literature which uses strict language grammar rules do not
seem to be suitable for Turkish CR because of the agglutinative
and morphologically very rich nature of the language. This
study presents a data-driven coreference resolution system
which learns from annotated data by use of machine-learning
techniques. Although not strictly encoded, the system also
uses linguistic analysis results (e.g. morphological, syntactic)
during its decision process. In contrary to some studies which
could be listed under Turkish CR task and which cover only
limited types of relations (such as pronominal relations), this
study do not limit itself to a specific coreference type and
covers many types of relations occurring between different
types of linguistic entities such as pronouns, adjectival clauses,
named entities and possessive constructions.

In this study, we use Marmara Turkish Coreference Corpus
(MTCC) (Schuller et al., 2017) (a subcollection from METU Corpus (MC) (Say et al. 2002)) where mentions, the coreference relations between them and the formed chains are manually annotated.

This thesis basically focuses on pairwise classification technique with two different models used in coreference resolution task: mention-pair model (MPM) and entity-mention model (EMM). Mention-pair models basically process pair of mentions and determine whether the current mention couple is coreferential with each other without considering previous judgments. Also, mention-pair models need a clustering mechanism as a post-processing step to create mention chains from the classification output. However, entity-mention models focus on coreference relation between an entity and a mention unlike MPMs. At this point, entity may be thought as a group of mentions referred to same real world entity. Therefore, EMMs make a decision on coreference relation between a mention and a group of mentions and classifer answers this question "Should this mention be inserted to this chain?". EMMs do not need a post-processor to create chains; because chains are obtained cumulatively during the prediction.

Our chain creator is a rule-based post-processor which
works on the classification outputs and aims to form chains according to prediction results. Chain creator accepts a mention pair which is classified positively. If these mentions are not assigned to any chain previously, then a new chain is generated and these two mentions are inserted to that chain. When one of these mentions are already added to a chain, the other mention is also appended to that chain. If two mentions occur in two different chains, creator accepts its previous decision and does not change their locations.


This study proposes a much rich feature set comprising detailed linguistic information (such as morphological and syntactic properties) extracted from a Turkish NLP pipeline (Eryiğit, 2014) inspiring from earlier studies
on Turkish dependency parsing (Eryiğit et al., 2008)-(Sulubacak et al., 2016). The selected features for each mention are: head's lemma, head's coarse and fine POS tags, the status of being/containing a multi-word expression, the status of being an adjectival clause/a noun phrase, string similarity of mentions, the status of having the same head forms/lemmas (see Section 3.1 for usage of these information in MPMs and EMMs).

Our initial set of experiments is to see the impact of the
negative sampling size and kernels on the CR task with MPMs. With this purpose, we start our experiments with the baseline introduced in (Schuller et al., 2017) by replicating it with exactly the same parameters using SVM with linear kernel and negative sampling within +-100 window size. With our replicated baseline, we obtain slightly better scores than the ones reported in (Schuller et al., 2017) which may be attributed to the difference in the used data set (see Section 2.2 for details). Therefore, mention-pair models are prepared with different negative sampling sizes (+-100, +-70, +-40 and +-10) and polynomial, RBF and linear kernels. Highest results are obtained with +-10 window size and polynomial kernel which yields a 6.49, 8.09, 7.62 and 8.81 percentage points improvement in MUC, B3, BLANC and LEA, respectively in case of linear kernel. The use of polynomial kernel also makes an improvement of more than 1 percentage point in all metrics except MUC.

In our second set of experiments, we experiment our MPM to evaluate the influence of our newly introduced feature set (see Section 3.1 for details) by using both Decision Trees (DTs) and Support Vector Machines (SVMs) as the binary classifiers. In this step, we continue with +-10 negative sampling window size and polynomial kernel for SVM. We see that SVMs are much more powerful (based of F-scores) than DTs in this setting where we use many categorical features such as lemma information. In terms of precision, DTs provide much higher results than SVMs but its recall values fall very behind. When we compare the results with the baseline feature model, we observe improvements in all metrics reaching almost 5 percentage points.

In our final set of experiments on MPMs, we investigate the impact of our proposed features for the Turkish CR task. We see that the most powerful feature is "The status of having the same head lemmas" which is also a feature existing in the baseline model. Thus, the gain may not be attributed to
this feature alone. Since Turkish is an agglutinative language, we see that checking the similarity of lemmas is more powerful than checking surface forms for the head tokens. As a result, the introduced MPM approach obtains 90.68% (MUC), 86.89% (B3), 85.13% (BLANC) and 78.34% (LEA) F1 scores with SVM algorithm by settings of +-10 window size negative sampling, polynomial kernel and newly the selected features yielding an improvement of 9.12, 16.06, 13.08 and 12.57 percentage points respectively over a recent baseline MPM system on Turkish CR (Schuller et al., 2017).


After experiments are completed on Turkish MPMs, we also focus on the Turkish CR with entity-mention models. In EMMs, we also experience two different negative sampling approach. Firstly, the existing order of mentions in the document is ignored during the preparation of negative sampling, and as the second approach, this property is also considered. In this manner, the system is improved 9.7 (B3), 3.7 (BLANC) and 7.03 (LEA) percentage points over related evaluation metrics. The best EMM provides 75.83% (MUC), 62.37% (B3), 57.15% (BLANC) ve 50.69% (LEA) F-scores. However, the main drawback of this model is creating many singletons. Therefore, EMMs are still open for improvement for Turkish.

As a future work, it is aimed to extend the feature set with word embeddings and test with different deep neural networks, and also we determine to focus on semantic based feature engineering and new approaches to make an improvement on EMMs, especially by reducing the number of creating singletons.