Venue: EMNLP 2017

# A Mention-Ranking Model for Abstract Anaphora Resolution

Ana Marasovic, Leo Born, Juri Opitz, Anette Frank\({}^{\dagger}\)

Research Training Group AIPHES

Department of Computational Linguistics

Heidelberg University

69120 Heidelberg, Germany

{marasovic,born,opitz,frank}@cl.uni-heidelberg.de

Leo Born, Juri Opitz and Anette Frank contributed equally to this work.

###### Abstract

Resolving abstract anaphora is an important, but difficult task for text understanding. Yet, with recent advances in representation learning this task becomes a more tangible aim. A central property of _abstract anaphora_ is that it establishes a relation between the _anaphor_ embedded in the _anaphoric sentence_ and its (typically non-nominal) _antecedent_. We propose a mention-ranking model that learns how abstract anaphors relate to their antecedents with an LSTM-Siamese Net. We overcome the lack of training data by generating artificial anaphoric sentence-antecedent pairs. Our model outperforms state-of-the-art results on _shell noun resolution_. We also report first benchmark results on an abstract anaphora subset of the ARRAU corpus. This corpus presents a greater challenge due to a mixture of nominal and pronominal anaphors and a greater range of confounders. We found model variants that outperform the baselines for nominal anaphors, without training on individual anaphor data, but still lag behind for pronominal anaphors. Our model selects syntactically plausible candidates and - if disregarding syntax - discriminates candidates using deeper features.

+
Footnote †: Leo Born, Juri Opitz and Anette Frank contributed equally to this work.

## 1 Introduction

Current research in anaphora (or coreference) resolution is focused on resolving noun phrases referring to concrete objects or entities in the real world, which is arguably the most frequently occurring type. Distinct from these are diverse types of _abstract_ anaphora (AA) Asher (1993) where reference is made to propositions, facts, events or properties. An example is given in (1) below.1

Footnote 1: Example drawn from ARRAU Uryupina et al. (2016).

While recent approaches address the resolution of selected abstract _shell nouns_Kolnatkar and Hirst (2014), we aim to resolve a wide range of abstract anaphors, such as the NP _this trend_ in (1), as well as pronominal anaphors (_this_, _that_, or _it_).

Henceforth, we refer to a sentence that contains an abstract anaphor as _the anaphoric sentence (AnaphS)_, and to a constituent that the anaphor refers to as _the antecedent (Antec)_ (cf. (1)).

1. Ever-more powerful desktop computers, designed with one or more microprocessors as their "brains", are expected to increasingly take on functions carried out by more expensive minicompeters and mainframes. "[_Latcc The gaps that make traditional hardware are really being obseleted by microprocessor-based machines_]", said Mr. Benton. [_AnaphS_ As a result of **this trend\({}_{AA}\)**, longtime powerhouses HP, IBM and Digital Equipment Corp. are scrambling to counterattack with microprocessor-based systems of their own.]

A major obstacle for solving this task is the lack of sufficient amounts of annotated training data. We propose a method to generate large amounts of training instances covering a wide range of abstract anaphor types. This enables us to use neural methods which have shown great success in related tasks: coreference resolution Clark and Manning (2016a), textual entailment Bowman et al. (2016), learning textual similarity Mueller and Thyagarajan (2016), and discourse relation sense classification Rutherford et al. (2017).

Our model is inspired by the mention-ranking model for coreference resolution Wiseman et al. (2015); Clark and Manning (2015, 2016a,b) and combines it with a Siamese Net Mueller and Thyagarajan (2016), Neculoiu et al. (2016) for

learning similarity between sentences. Given an _anphoric sentence_ (AntecS in (1)) and a candidate antecedent (any constituent in a given context, e.g. _being obsoleted by microprocessor-based machines_ in (1)), the LSTM-Siamese Net learns representations for the candidate and the anaphoric sentence in a shared space. These representations are combined into a joint representation used to calculate a score that characterizes the relation between them. The learned score is used to select the highest-scoring antecedent candidate for the given anaphoric sentence and hence its anaphor. We consider one anaphor at a time and provide the embedding of the _context of the anaphor_ and the embedding of the _head of the anaphoric phrase_ to the input to characterize each individual anaphor - similar to the encoding proposed by Zhou and Xu (2015) for individuating multiply occurring predicates in SRL. With deeper inspection we show that the model learns a relation between the anaphor in the anaphoric sentence and its antecedent. Fig. 1 displays our architecture.

In contrast to other work, our method for generating training data is not confined to specific types of anaphora such as shell nouns Kolhatkar and Hirst (2014) or anaphoric connectives Stede and Grishina (2016). It produces large amounts of instances and is easily adaptable to other languages. This enables us to build a robust, knowledge-lean model for abstract anaphora resolution that easily extends to multiple languages.

We evaluate our model on the shell noun resolution dataset of Kolhatkar et al. (2013b) and show that it outperforms their state-of-the-art results. Moreover, we report results of the model (trained on our newly constructed dataset) on unrestricted abstract anaphora instances from the ARRAU corpus Poesio and Artstein (2008); Uryupina et al. (2016). To our knowledge this provides the first state-of-the-art benchmark on this data subset.

Our TensorFlow2 implementation of the model and scripts for data extraction are available at: [https://github.com/amarasovic/neural-abstract-anaphora](https://github.com/amarasovic/neural-abstract-anaphora).

Footnote 2: Abadi et al. (2015)

## 2 Related and prior work

**Abstract anaphora** has been extensively studied in linguistics and shown to exhibit specific properties in terms of semantic antecedent types, their degrees of abstractness, and general discourse properties Asher (1993); Webber (1991). In contrast to nominal anaphora, abstract anaphora is difficult to resolve, given that agreement and lexical match features are not applicable. Annotation of abstract anaphora is also difficult for humans Dipper and Zinsmeister (2012), and thus, only few smaller-scale corpora have been constructed. We evaluate our models on a subset of the ARRAU corpus Uryupina et al. (2016) that contains abstract anaphors and the shell noun corpus used in Kolhatkar et al. (2013b).3 We are not aware of other freely available abstract anaphora datasets.

Footnote 3: We thank the authors for making their data available.

Little work exists for the **automatic resolution of abstract anaphora**. Early work Eckert and Strube (2000); Strube and Muller (2003); Byron (2004); Muller (2008) has focused on spoken language, which exhibits specific properties. Recently, **event coreference** has been addressed using feature-based classifiers Jauhar et al. (2015); Lu and Ng (2016). Event coreference is restricted to a subclass of _events_, and usually focuses on coreference between verb (phrase) and noun (phrase) mentions of similar abstractness levels (e.g. _purchase_ - _acquire_) with no special focus on (pro)nominal anaphora. Abstract anaphora typically involves a full-fledged clausal antecedent that is referred to by a highly abstract (pro)nominal anaphor, as in (1).

Rajagopal et al. (2016) proposed a model for resolution of events in biomedical text that refer to a single or multiple clauses. However, instead of selecting the correct antecedent clause(s) (our task) for a given event, their model is restricted to classifying the event into six abstract categories: this these _changes, responses, analysis, context, finding, observation_, based on its surrounding context. While related, their task is not comparable to the full-fledged abstract anaphora resolution task, since the events to be classified are known to be coreferent and chosen from a set of restricted abstract types.

More related to our work is Anand and Hardt (2016) who present an antecedent ranking account for **slucing** using classical machine learning based on a small training dataset. They employ features modeling distance, containment, discourse structure, and - less effectively - content and lexical correlates.4

Footnote 4: Their data set was not publicized.

Closest to our work is Kolhatkar et al. (2013b) 

(KZH13) and Kolhatkar and Hirst (2014) (KH14) on **shell noun resolution**, using classical machine learning techniques. Shell nouns are abstract nouns, such as _fact_, _possibility_, or _issue_, which can only be interpreted jointly with their _shell_ content (their embedded clause as in (2) or antecedent as in (3)). KZH13 refer to shell nouns whose antecedent occurs in the prior discourse as _anaphoric shell nouns_ (ASNs) (cf. (3)), and _cataphoric shell nouns_ (CSNs) otherwise (cf. (2)).5

Footnote 5: We follow this terminology for their approach and data representation.

* and that it takes away money from regular education_.]
* Environmental Defense [...] notes that [\({}_{Attoc}\)_Mowing the lawn with a gas mower produces as much pollution [...] as driving a car 172 miles_.] [\({}_{AnughS}\)_This fact_ may [...] explain the recent surge in the sales of [...] old-fashioned push monomers [...].

KZH13 presented an approach for resolving six typical shell nouns following the observation that CSNs are easy to resolve based on their syntactic structure alone, and the assumption that ASNs share linguistic properties with their embedded (CSN) counterparts. They manually developed rules to identify the embedded clause (i.e. cataphoric antecedent) of CSNs and trained SVM\({}^{rank}\)(Joachims, 2002) on such instances. The trained SVM\({}^{ank}\) model is then used to resolve ASNs. KH14 generalized their method to be able to create training data for any given shell noun, however, their method heavily exploits the specific properties of shell nouns and does not apply to other types of abstract anaphora.

Stede and Grishina (2016) study a related phenomenon for German. They examine inherently anaphoric connectives (such as _demzufolge - according to which_) that could be used to access their abstract antecedent in the immediate context. Yet, such connectives are restricted in type, and the study shows that such connectives are often ambiguous with nominal anaphors and require sense disambiguation. We conclude that they cannot be easily used to acquire antecedents automatically.

In our work, we explore a different direction: we construct artificial training data using a general pattern that identifies embedded sentence constituents, which allows us to extract relatively secure training data for abstract anaphora that captures a wide range of anaphora-antecedent relations, and apply this data to train a model for the resolution of unconstrained abstract anaphora.

Recent work in entity coreference resolution has proposed powerful neural network-based models that we will adapt to the task of abstract anaphora resolution. Most relevant for our task is the **mention-ranking neural coreference model** proposed in Clark and Manning (2015), and their improved model in Clark and Manning (2016a), which integrates a loss function Wiseman et al. (2015) which learns distinct feature representations for anaphoricity detection and antecedent ranking.

**Siamese Nets** distinguish between similar and dissimilar pairs of samples by optimizing a loss over the metric induced by the representations. It is widely used in vision Chopra et al. (2005), and in NLP for semantic similarity, entailment, query normalization and QA Mueller and Thyagarajan (2016); Neculoiu et al. (2016); Das et al. (2016).

## 3 Mention-Ranking Model

Given an anaphoric sentence \(s\) with a marked anaphor (mention) and a candidate antecedent \(c\), the mention-ranking (MR) model assigns the pair \((c,s)\) a score, using representations produced by an LSTM-Siamese Net. The highest-scoring candidate is assigned to the marked anaphor in the anaphoric sentence. Fig. 1 displays the model.

We learn representations of an anaphoric sentence \(s\) and a candidate antecedent \(c\) using a bi-directional Long Short-Term Memory Hochreiter and Schmidhuber (1997); Graves and Schmidhuber (2005). One bi-LSTM is applied to the anaphoric sentence \(s\) and a candidate antecedent \(c\), hence the term siamese. Each word is represented with a vector \(\mathbf{w_{i}}\) constructed by concatenating embeddings of the word, of the context of the anaphor (average of embeddings of the anaphoric phrase, the previous and the next word), of the head of the anaphoric phrase6, and, finally, an embedding of the constituent tag of the candidate, or the \(S\) constituent tag if the word is in the anaphoric sentence. For each sequence \(s\) or \(c\), the word vectors \(\mathbf{w_{i}}\) are sequentially fed into the bi-LSTM, which produces outputs from the forward pass, \(\overrightarrow{\mathbf{h_{i}}}\), and outputs \(\overleftarrow{\mathbf{h_{i}}}\) from the backward pass. The final output of the i-th word is defined as \(\mathbf{h_{i}}=[\overleftarrow{\mathbf{h_{i}}};\overleftarrow{\mathbf{h_{i}}}]\). To get a representation of the full sequence, \(\mathbf{h_{s}}\) or \(\mathbf{h_{c}}\), all outputs are averaged, except for those that correspond to padding tokens.

 

To prevent forgetting the constituent tag of the sequence, we concatenate the corresponding tag embedding with \({\mathbf{h}_{s}}\) or \({\mathbf{h}_{c}}\) (we call this a _shortcut_ for the tag information). The resulting vector is fed into a feed-forward layer of exponential linear units (ELUs) Clevert et al. (2016) to produce the final representation \(\tilde{\mathbf{h}_{s}}\) or \(\tilde{\mathbf{h}_{c}}\) of the sequence.

From \(\widetilde{\mathbf{h}_{c}}\) and \(\widetilde{\mathbf{h}_{s}}\) we compute a vector \({\mathbf{h}_{c,s}}=[|\tilde{\mathbf{h}_{c}}-\widetilde{\mathbf{h}_{s}}|;\widetilde{\mathbf{h} _{c}}\odot\widetilde{\mathbf{h}_{s}}]\)Tai et al. (2015), where \(|\!-\!|\) denotes the absolute values of the element-wise subtraction, and \(\odot\) the element-wise multiplication. Then \({\mathbf{h}_{c,s}}\) is fed into a feed-forward layer of ELUs to obtain the final joint representation, \(\widetilde{\mathbf{h}_{c,s}}\), of the pair \((c,s)\). Finally, we compute the score for the pair \((c,s)\) that represents relatedness between them, by applying a single fully connected linear layer to the joint representation:

\[score(c,s)=W\widetilde{\mathbf{h}_{c,s}}+b\in\mathbb{R}, \tag{1}\]

where W is a \(1\times d\) weight matrix, and \(d\) the dimension of the vector \(\widetilde{\mathbf{h}_{c,s}}\).

We train the described mention-ranking model with the max-margin training objective from Wiseman et al. (2015), used for the antecedent ranking subtask. Suppose that the training set \(\mathcal{D}=\{(a_{i},s_{i},\mathcal{I}(a_{i}),\mathcal{N}(a_{i})\}^{n}_{i=1}\), where \(a_{i}\) is the i-th abstract anaphor, \(s_{i}\) the corresponding anaphoric sentence, \(\mathcal{I}(a_{i})\) the set of antecedents of \(a_{i}\) and \(\mathcal{N}(a_{i})\) the set of candidates that are not antecedents (negative candidates). Let \(\tilde{t_{i}}=\arg\max_{t\in\mathcal{I}(a_{i})}score(t_{i},s_{i})\) be the highest scoring antecedent of \(a_{i}\). Then the loss is given by

\[\sum_{i=1}^{n}\max(0,\max_{c\in\mathcal{N}(a_{i})}\{1+score(c,s_{i})-score( \tilde{t_{i}},s_{i})\}).\]

## 4 Training data construction

We create large-scale training data for abstract anaphora resolution by exploiting a common construction, consisting of a verb with an embedded sentence (complement or adverbial) (cf. Fig. 2). We detect this pattern in a parsed corpus, 'cut off' the S\({}^{\prime}\) constituent and replace it with a suitable anaphor to create the anaphoric sentence (AnaphS), while S yields the antecedent (Antec). This method covers a wide range of anaphorameteced constellations, due to diverse semantic or discourse relations that hold between the clause hosting the verb and the embedded sentence.

First, the pattern applies to verbs that embed sentential arguments. In (4), the verb _doubt_ establishes a specific semantic relation between the embedding sentence and its sentential complement.

* He doubts [\({}_{S^{\prime}}\)[\({}_{S}\) a Bismarckian super state will emerge that would dominate Europe], but warns of "a risk of profound change in the [..] European Community from a Germany that is too strong, even if democratic"].

From this we extract the artificial antecedent _A Bismarckian super state will emerge that would dominate Europe_, and its corresponding anaphoric sentence _He doubts this, but warns of "a risk of profound change ... even if democratic"_, which we construct by randomly choosing one of a predefined set of appropriate anaphors (here: _this, that, it_), cf. Table 1. The second row in Table 1 is used when the head of S\({}^{\prime}\) is filled by an overt complementizer (_doubts that_), as opposed to (4). The remaining rows in Table 1 apply to adverbial clauses of different types.

Adverbial clauses encode specific discourse relations with their embedding sentences, often indicated by their conjunctions. In (5), for example, the causal conjunction **as** relates a cause (embedded sentence) and its effect (embedding sentence):

Figure 1: Mention-ranking architecture for abstract anaphora resolution (MR-LSTM).

Figure 2: A general pattern for artificially creating anaphoric sentence–antecedent pairs.

 

5. There is speculation that property casualty firms will sell even more munis [\({}_{S^{\prime}}\)**as**] [\({}_{S}\) they scramble to raise cash to pay claims related to Hurricane Hugo [..] I].

We randomly replace causal conjunctions _because, as_ with appropriately adjusted anaphors, e.g. _because of that, due to this_ or _therefore_ that make the causal relation explicit in the anaphor.7

Footnote 6: In case of ambiguous conjunctions (e.g. _as_ interpreted as causal or temporal), we generally choose the most frequent interpretation.

Compared to the shell noun corpus of KZH13, who made use of a carefully constructed set of extraction patterns, a downside of our method is that our artificially created antecedents are uniformly of type S. However, the majority of abstract anaphora antecedents found in the existing datasets are of type S. Also, our models are intended to induce semantic representations, and so we expect syntactic form to be less critical, compared to a feature-based model.8 Finally, the general extraction pattern in Fig. 2, covers a much wider range of anaphoric types.

Footnote 7: This also alleviates problems with languages like German, where (non-)embedded sentences differ in surface position of the finite verb. We can either adapt the order or ignore it, when producing anaphoric sentence – antecedent pairs.

Using this method we generated a dataset of artificial anaphoric sentence-antecedent pairs from the WSJ part of the PTB Corpus Marcus et al. (1993), automatically parsed using the Stanford Parser Klein and Manning (2003).

## 5 Experimental setup

### Datasets

We evaluate our model on two types of anaphora: (a) _shell noun anaphora_ and (b) (pro)nominal abstract anaphors extracted from ARRAU.

#### a. Shell noun resolution dataset

For comparability we train and evaluate our model for _shell noun resolution_, using the original training (CSN) and test (ASN) corpus of Kolhatkar et al. (2013a, b).9 We follow the data preparation and evaluation protocol of Kolhatkar et al. (2013b) (**KZH13**).

Footnote 9: We thank the authors for providing the available data.

The **CSN corpus** was constructed from the NYT corpus using manually developed patterns to identify the antecedent of cataphoric shell nouns (CSNs). In KZH13, all syntactic constituents of the sentence that contains both the CSN and its antecedent were considered as candidates for training a ranking model. Candidates that differ from the antecedent in only one word or one word and punctuation were as well considered as antecedents10. To all other candidates we refer to as _negative_ candidates. For every shell noun, KZH13 used the corresponding part of the CSN data to train SVM\({}^{rank}\).

Footnote 10: We obtained this information from the authors directly.

The **ASN corpus** serves as the test corpus. It was also constructed from the NYT corpus, by selecting anaphoric instances with the pattern "_this \(\langle\)shell noun\(\rangle\)_" for all covered shell nouns. For validation, Kolhatkar et al. (2013a) crowdsourced annotations for the sentence which contains the antecedent, which KZH13 refer to as a _broad region_. Candidates for the antecedent were obtained by using all syntactic constituents of the broad region as candidates and ranking them using the SVM\({}^{rank}\) model trained on the CSN corpus. The top 10 ranked candidates were presented to the crowd workers and they chose the best answer that represents the ASN antecedent. The workers were encouraged to select _None_ when they did not agree with any of the displayed answers and could provide information about how satisfied they were with the displayed candidates. We consider this dataset as gold, as do KZH13, although it may be biased towards the offered candidates.11

Footnote 11: The authors provided us with the workers’ annotations of the broad region, antecedects chosen by the workers and links to the NYT corpus. The extraction of the anaphoric sentence and the candidates had to be redone.

**b. Abstract anaphora resolution data set.** We use the automatically constructed data from the WSJ corpus (Section 4) for training.12 Our test data for unrestricted abstract anaphora resolution is obtained from the **ARRAU corpus** Uryupina et al. (2016). We extracted all abstract anaphoric instances from the WSJ part of ARRAU that are marked with the category _abstract_ or _plan_,13 and call the subcorpus **ARRAYAAU-AA**.

Footnote 12: We excluded any documents that are part of ARRAU.

\begin{table}
\begin{tabular}{l l l} \hline \hline type & head of S\({}^{\prime}\) & possible anaphoric phrase \\ \hline empty & \(\emptyset\) & this, that \\ general & that, this & that, this \\ causal & because, as & therefore, because of this/that, \\ temporal & while, since, etc. & during this/that \\ conditional & if, whether & if this/that is true \\ \hline \hline \end{tabular}
\end{table}
Table 1: S\({}^{\prime}\)-heads and the anaphoric types and phrases they induce (most frequent interpretation).

 

Candidates extraction.Following KZH13, for every anaphor we create a list of candidates by extracting all syntactic constituents from sentences which contain antecedents. Candidates that differ from antecedents in only one word, or one word and punctuation, were as well considered as antecedents. Constituents that are not antecedents are considered as negative candidates.

Data statistics.Table 2 gives statistics of the datasets: the number of anaphors (row 1), the median length (in tokens) of antecedents (row 2), the median length (in tokens) for all anaphoric sentences (row 3), the median of the number of antecedents and candidates that are not antecedents (negatives) (rows 4-5), the number of pronominal and nominal anaphors (rows 6-7). Both training sets, artificial and CSN, have only one possible antecedent for which we accept two minimal variants differing in only one word or one word and punctuation. On the contrary, both test sets by design allow annotation of more than one antecedent that differ in more than one word. Every anaphor in the artificial training dataset is pronominal, whereas anaphors in CSN and ASN are nominal only. ARRAU-AA has a mixture of nominal and pronominal anaphors.

Data pre-processing.Other details can be found in Supplementary Materials.

### Baselines and evaluation metrics

Following KZH13, we report _success@n_ (\(s\)@\(n\)), which measures whether the antecedent, or a candidate that differs in one word14, is in the first \(n\) ranked candidates, for \(n\in\{1,2,3,4\}\). Additionally, we report the preceding sentence baseline (PS\({}_{BL}\)) that chooses the previous sentence for the antecedent and _TAGbaseline_ (TAG\({}_{BL}\)) that randomly chooses a candidate with the constituent tag label in {S, VP, ROOT, SBAR}. For TAG\({}_{BL}\) we report the average of 10 runs with 10 fixed seeds. PS\({}_{BL}\) always performs worse than the KZH13 model on the ASN, so we report it only for ARRAU-AA.

Footnote 14: We obtained this information in personal communication with one of the authors.

### Training details for our models

Hyperparameters tuning.We recorded performance with manually chosen HPs and then tuned HPs with Tree-structured Parzen Estimators (TPE) [8]. TPE chooses HPs for the next (out of \(10\)) trails on the basis of the _s@_1 score on the devset. As devsets we employ the ARRAU-AA corpus for shell noun resolution and the ASN corpus for unrestricted abstract anaphora resolution. For each trial we record performance on the test set. We report the best test _s@_1 score in \(10\) trials if it is better than the scores from default HPs. The default HPs and prior distributions for HPs used by TPE are given below. The (exact) HPs we used can be found in Supplementary Materials.

Input representation.To construct word vectors \(\mathbf{w_{i}}\) as defined in Section 3, we used \(100\)-dim. GloVe word embeddings pre-trained on the Gigaword and Wikipedia [9], and did not fine-tune them. Vocabulary was built from the words in the training data with frequency in \(\{3,~{}\mathcal{U}(1,10)\}\), and OOV words were replaced with an _UNK_ token. Embeddings for tags are initialized with values drawn from the uniform distribution \(\mathcal{U}\big{(}-\frac{1}{\sqrt{d+t}},\frac{1}{\sqrt{d+t}}\big{)}\), where \(t\) is the number of tags16 and \(d\in\{50,\textit{qlog-$\mathcal{U}(30,100)$}\}\) the size of the tag embeddings.17 We experimented with removing embeddings for tag, anaphor and context.

Footnote 17: [https://github.com/hyperopt/hyperopt](https://github.com/hyperopt/hyperopt).

Footnote 18: We used a list of tags obtained from the Stanford Parser.

Footnote 17: \(\textit{qlog-$\mathcal{U}$}\) is the so-called qlog-uniform distribution.

**Weightes initialization.The size of the LSTMs hidden states was set to \(\{100,\textit{qlog-$\mathcal{U}(30,150)$}\}\). We initialized the weight matrices of the LSTMs with random orthogonal matrices [10], all other weight matrices with the initialization proposed in He et al. [10]. The first feed-forward layer size is set to a value in \(\{400,\textit{qlog-$\mathcal{U}(200,800)$}\}\), the second to a value in \(\{1024,\textit{qlog-$\mathcal{U}(400,2000)$}\}\). Forget biases in the LSTM were initialized with 1s [8], all other biases with 0s.**

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline  & \multicolumn{2}{c}{shell noun} & \multicolumn{2}{c}{abstract anaphone} \\ \cline{2-5}  & \begin{tabular}{c} **CSN** \\ train \\ \end{tabular} & \begin{tabular}{c} **ASN** \\ test \\ \end{tabular} & \begin{tabular}{c} **artifical** \\ train \\ \end{tabular} & \begin{tabular}{c} **ARRAU-AA** \\ test \\ \end{tabular} \\ \hline \# shell nouns / anaphors & \begin{tabular}{c} 114492 \\ \end{tabular} & \begin{tabular}{c} 2303 \\ \end{tabular} & \begin{tabular}{c} 8527 \\ \end{tabular} & \begin{tabular}{c} 600 \\ \end{tabular} \\ \hline media & Antec & 12.75 & 13.87 & 11 & 20.5 \\ \# of tokens & AnaB\(\overline{\text{s}}\) & 11.5 & 24 & 19 & 28 \\ \hline median & Antec & 2 & 4.5 & 2 & 1 \\ \# & negatives & 44.5 & 39 & 15 & 48 \\ \hline \multirow{2}{*}{\#} & nominal & \begin{tabular}{c} 114492 \\ \end{tabular} & \begin{tabular}{c} 2303 \\ \end{tabular} & \begin{tabular}{c} 0 \\ \end{tabular} & 
\begin{tabular}{c} 397 \\ \end{tabular} \\ \cline{2-5}  & promoinal & 0 & 0 & 8527 & 203 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Data statistics. For the ASN and CSN we report statistics over all shell nouns, but classifiers are trained independently.

 



**Optimization.** We trained our model in mini-batches using Adam (Kingma and Ba, 2015) with the learning rate of \(10^{-4}\) and maximal batch size \(64\). We clip gradients by global norm (Pascanu et al., 2013), with a clipping value in \(\{1.0,~{}\mathcal{U}(1,100)\}\). We train for \(10\) epochs and choose the model that performs best on the devset.

**Regularization.** We used the \(l_{2}\)-regularization with \(\lambda\in\{10^{-5},~{}log\text{-}\mathcal{U}(10^{-7},10^{-2})\}\). Dropout (Srivastava et al., 2014) with a keep probability \(k_{p}\in\{0.8,~{}\mathcal{U}(0.5,1.0)\}\) was applied to the outputs of the LSTMs, both feed-forward layers and optionally to the input with \(k_{p}\in\mathcal{U}(0.8,1.0)\).

## 6 Results and analysis

### Results on shell noun resolution dataset

Table 3 provides the results of the mention-ranking model (MR-LSTM) on the ASN corpus using default HPs. Column 2 states which model produced the results: KZH13 refers to the best reported results in Kolhatkar et al. (2013b) and TAG\({}_{BL}\) is the baseline described in Section 5.2.

In terms of s@1 score, MR-LSTM outperforms both KZH13's results and TAG\({}_{BL}\) without even necessitating HP tuning. For the outlier _reason_ we tuned HPs (on ARRAU-AA) for different variants of the architecture: the full architecture, without embedding of the context of the anaphor (ctx), of the anaphor (aa), of both constituent tag embedding and shortcut (tag,cut), dropping only the shortcut (cut), using only word embeddings as input (ctx,aa,tag,cut), without the first (ffl1) and second (ff12) layer. From Table 4 we observe: (1) with HPs tuned on ARRAU-AA, we obtain results well beyond KZH13, (2) all ablated model variants perform worse than the full model, (3) a large performance drop when omitting syntactic information (tag,cut) suggests that the model makes good use of it. However, this could also be due to a bias in the tag distribution, given that all candidates stem from the single sentence that contains antecedents. The median occurrence of the S tag among both antecedents and negative candidates is \(1\), thus the model could achieve \(50.00\)_s@1_by picking S-type constituents, just as TAG\({}_{BL}\) achieves \(42.02\) for _reason_ and \(48.66\) for _possibility_.

Tuning of HPs gives us insight into how different model variants cope with the task. For example, without tuning the model with and without syntactic information achieves \(71.27\) and \(19.68\) (not shown in table) _s@_1_score, respectively, and with tuning: \(87.78\) and \(68.10\). Performance of \(68.10\)_s@_1_score indicates that the model is able to learn without syntactic guidance, contrary to the \(19.68\)_s@_1_score before tuning.

### Results on the ARRAU corpus

Table 5 shows the performance of different variants of the MR-LSTM with HPs tuned on the ASN corpus (always better than the default HPs), when evaluated on \(3\) different subparts of the ARRAU-AA: all \(600\) abstract anaphors, \(397\) nominal and \(203\) pronominal ones. HPs were tuned on the ASN corpus for every variant separately, without shuffling of the training data. For the best performing variant, without syntactic information (tag,cut), we report the results with HPs that yielded the best _s@_1_test score for all anaphors (row 4), when training with those HPs on shuffled training data (row 5), and with HPs that yielded the best _s@_1_test score for all anaphors (row 6), when training with those HPs on shuffled training data (row 5). In addition, the results obtained with the same set of models are not consistent with the other two baselines.

\begin{table}
\begin{tabular}{c|c|c|c c||c c c c} \hline  & & \multicolumn{6}{c}{**reason**} \\ ctx & aa & tag & cut & ff1 & ff2 & s@1 & s@2 & s@ 3 & s@ 4 \\ \hline ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & **87.78** & **91.63** & **93.44** & **93.89** \\ ✗ & ✓ & ✓ & ✓ & ✓ & ✓ & 85.97 & 87.56 & 89.14 & 89.82 \\ ✓ & ✗ & ✓ & ✓ & ✓ & ✓ & 86.65 & ss91 & 91.18 & 91.40 \\ ✓ & ✓ & ✗ & ✗ & ✓ & ✓ & 68.10 & 80.32 & 85.29 & 89.37 \\ ✓ & ✓ & ✓ & ✗ & ✓ & ✓ & 85.52 & ss8.24 & 89.59 & 90.05 \\ ✗ & ✗ & ✗ & ✓ & ✓ & ✓ & 66.67 & 80.54 & 85.75 & 88.24 \\ ✓ & ✓ & ✓ & ✓ & ✗ & ✓ & 87.56 & 91.63 & 92.76 & 94.12 \\ ✓ & ✓ & ✓ & ✓ & ✓ & ✗ & 85.97 & 88.69 & 89.14 & 90.05 \\ \hline \end{tabular}
\end{table}
Table 4: Architecture ablation for _reason_.

\begin{table}
\begin{tabular}{c|c|c c c c} \hline  & & s @ 1 & s @ 2 & s @ 3 & s @ 4 \\ \hline

- \\ \hline

- \\ \hline

- \\ \hline

- \\ \hline

- \\ \hline

- \\ \hline \end{tabular}
\end{table}
Table 3: Shell noun resolution results.


score for pronominal anaphors (row 6).

The MR-LSTM is more successful in resolving nominal than pronominal anaphors, although the training data provides only pronominal ones. This indicates that resolving pronominal abstract anaphora is harder compared to nominal abstract anaphora, such as shell nouns. Moreover, for shell noun resolution in KZH13's dataset, the MR-LSTM achieved _s@1_ scores in the range \(76.09\text{--}93.14\), while the best variant of the model achieves \(51.89~{}s\)_s@1_ score for nominal anaphors in ARRAU-AA. Although lower performance is expected, since we do not have specific training data for individual nominals in ARRAU-AA, we suspect that the reason for better performance for shell noun resolution in KZH13 is due to a larger number of positive candidates in ASN (cf. Table 2, rows: antecedents/negatives).

We also note that HPs that yield good performance for resolving nominal anaphors are not necessarily good for pronominal ones (cf. rows 4-6 in Table 5). Since the TPE tuner was tuned on the nominal-only ASN data, this suggest that it would be better to tune HPs for pronominal anaphors on a different dataset or stripping the nouns in ASN.

Contrary to shell noun resolution, omitting syntactic information boosts performance in ARRAU-AA. We conclude that when the model is provided with syntactic information, it learns to pick S-type candidates, but does not continue to learn deeper features to further distinguish them or needs more data to do so. Thus, the model is not able to point to exactly one antecedent, resulting in a lower _s@1_ score, but does well in picking a few good candidates, which yields good _s@_2-4 scores. This is what we can observe from row 2 vs. row 6 in Table 5: the MR-LSTM without context embedding (ctx) achieves a comparable _s@_2 score with the variant that omits syntactic information, but better _s@_3-4 scores. Further, median occurrence of tags not in {S, VP, ROOT, SBAR} among top-4 ranked candidates is \(0\) for the full architecture, and \(1\) when syntactic information is omitted. The need for discriminating capacity of the model is more emphasized in ARRAU-AA, given that the median occurrence of S-type candidates among negatives is \(2\) for nominal and even \(3\) for pronominal anaphors, whereas it is \(1\) for ASN. This is in line with the lower TAG\({}_{BL}\) in ARRAU-AA.

Finally, not all parts of the architecture contribute to system performance, contrary to what is observed for _reason_. For nominal anaphors, the anaphor (aa) and feed-forward layers (ffl1, ffl2) are beneficial, for pronominals only the second ffl.

### Exploring the model

We finally analyze deeper aspects of the model: (1) whether a learned representation between the anaphoric _sentence_ and an antecedent establishes a relation between _a specific anaphor we want to resolve_ and the antecedent and (2) whether the max-margin objective enforces a separation of the joint representations in the shared space.

(1) We claim that by providing embeddings of both the anaphor and the sentence containing the anaphor we ensure that the learned relation between antecedent and anaphoric sentence is dependent on the anaphor under consideration. Fig. 3 illustrates the heatmap for an anaphoric sentence with two anaphors. The i-th column of the heatmap corresponds to absolute differences between the output of the bi-LSTM for the i-th word in the anaphoric sentence when the first vs. second anaphor is resolved. Stronger color indi

\begin{table}
\begin{tabular}{c|c c|c c|c c c c|c c c c|c c c c} \hline  & \multicolumn{10}{c}{**all (600)**} & \multicolumn{10}{c}{**nominal (397)**} & \multicolumn{3}{c}{**pronominal (203)**} \\ ctx & aa & tag & cut & ffl1 & ffl2 & s@1 & s@2 & s@ 3 & s@ 4 & s@1 & s@2 & s@ 3 & s@ 4 & s@1 & s@2 & s@ 3 & s@ 4 \\ \hline ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & 24.17 & 43.67 & 54.50 & 63.00 & 29.47 & 50.63 & 62.47 & 72.04 & 13.79 & 30.05 & 38.92 & 45.32 \\ ✗ & ✓ & ✓ & ✓ & ✓ & ✓ & 29.67 & 52.50 & 66.00 & **75.00** & 33.50 & 58.19 & 72.04 & **80.86** & 22.17 & 41.38 & **54.19** & **63.55** \\ ✓ & ✗ & ✓ & ✓ & ✓ & ✓ & ✓ & 22.83 & 39.00 & 52.00 & 61.33 & 22.42 & 41.31 & 54.66 & 64.48 & 23.65 & 34.48 & 46.80 & 55.17 \\ ✓ & ✓ & ✗ & ✗ & ✓ & ✓ & 38.33 & 54.83 & 63.17 & 69.33 & 46.60 & **64.48** & 72.54 & 79.09 & 22.17 & 35.96 & 44.83 & 50.25 \\ ✓ & ✓ & ✗ & ✗ & ✓ & ✓ & **43.83** & **56.33** & **66.33** & 73.00 & **51.89** & **64.48** & **73.55** & 79.98 & 28.08 & 40.39 & 52.22 & 59.61 \\ ✓ & ✓ & ✗ & ✗ & ✓ & ✓ & 38.17 & 52.50 & 61.33 & 68.67 & 43.07 & 57.43 & 65.49 & 72.04 & 28.57 & **42.86** & 53.20 & 62.07 \\ ✓ & ✓ & ✓ & ✗ & ✓ & ✓ & 30.17 & 48.00 & 57.83 & 67.33 & 30.73 & 50.88 & 61.21 & 71.54 & **29.06** & 42.36 & 51.23 & 59.11 \\ ✗ & ✗ & ✗ & ✓ & ✓ & ✓ & 26.33 & 40.50 & 50.67 & 85.67 & 28.46 & 41.81 & 52.14 & 59.70 & 22.17 & 37.93 & 47.78 & 56.65 \\ ✓ & ✓ & ✓ & ✓ & ✗ & ✓ & 21.33 & 41.77 & 53.17 & 60.33 & 23.43 & 47.36 & 60.45 & 69.52 & 17.24 & 29.06 & 38.92 & 42.36 \\ ✓ & ✓ & ✓ & ✓ & ✓ & ✗ & 12.00 & 24.67 & 33.50 & 41.50 & 13.35 & 27.20 & 37.28 & 45.84 & 9.36 & 19.70 & 26.11 & 33.00 \\ \hline \end{tabular}
\end{table}
Table 5: Results table for the ARRAU-AA test set. Refer to text for explanation of duplicated rows.

 

cates larger difference, the blue rectangle represents the column for the head of the first anaphor, the dashed blue rectangle the column for the head of the second anaphor. Clearly, the representations differ when the first vs. second anaphor is being resolved and consequently, joint representations with an antecedent will differ too.

(2) It is known that the max-margin objective separates the best-scoring positive candidate from the best-scoring negative candidate. To investigate what the objective accomplishes in the MR-LSTM model, we analyze the joint representations of candidates and the anaphoric sentence (i.e., outputs of ffl2) after training. For a randomly chosen instance from ARRAU-AA, we plotted outputs of ffl2 with the tSNE algorithm (v.d. Maaten and Hinton, 2008). Fig. 4 illustrates that the joint representation of the first ranked candidate and the anaphoric sentence is clearly separated from other joint representations. This shows that the max-margin objective separates the best scoring positive candidate from the best scoring negative candidate by separating their respective joint representations with the anaphoric sentence.

## 7 Conclusions

We presented a neural mention-ranking model for the resolution of unconstrained abstract anaphora, and applied it to two datasets with different types of abstract anaphora: the shell noun dataset and a subpart of ARRAU with (pro)nominal abstract anaphora of any type. To our knowledge this work is the first to address the unrestricted abstract anaphora resolution task with a neural network. Our model also outperforms state-of-the-art results on the shell noun dataset.

In this work we explored the use of purely artificially created training data and how far it can bring us. In future work, we plan to investigate mixtures of (more) artificial and natural data from different sources (e.g. ASN, CSN).

On the more challenging ARRAU-AA, we found model variants that surpass the baselines for the entire and the nominal part of ARRAU-AA, although we do not train models on individual (nominal) anaphor training data like the related work for shell noun resolution. However, our model still lags behind for pronominal anaphors. Our results suggest that models for nominal and pronominal anaphors should be learned independently, starting with tuning of HPs on a more suitable devset for pronominal anaphors.

We show that the model can exploit syntactic information to select plausible candidates, but that when it does so, it does not learn how to distinguish candidates of equal syntactic type. By contrast, if the model is not provided with syntactic information, it learns deeper features that enable it to pick the correct antecedent without narrowing down the choice of candidates. Thus, in order to improve performance, the model should be enforced to first select reasonable candidates and then continue to learn features to distinguish them, using a larger training set that is easy to provide.

In future work we will design such a model, and offer it candidates chosen not only from sentences containing the antecedent, but the larger context.

#### Acknowledgments

This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1. We would like to thank anonymous reviewers for useful comments and especially thank Todor Mihaylov for the model implementations advices and everyone in the Computational Linguistics Group for helpful discussion.

Figure 3: Visualizing the differences between outputs of the bi-LSTM over time for an anaphoric sentence containing two anaphors.

 



## References

* Abadi et al. (2015) Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherby Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
* Anand and Hardt (2016) Pranav Anand and Daniel Hardt. 2016. Antecedent selection for sluicing: Structure and content. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1234-1243, Austin, Texas.
* Asher (1993) Nicholas Asher. 1993. _Reference to Abstract Objects in Discourse_. Kluwer Academic Publishers.
* Bergstra et al. (2011) James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl. 2011. Algorithms for hyper-parameter optimization. In _Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NIPS)_, Granada, Spain.
* Bowman et al. (2016) Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1466-1477, Berlin, Germany.
* Byron (2004) Donna K. Byron. 2004. _Resolving pronominal reference to abstract entities_. Ph.D. thesis, University of Rochester, Rochester, New York.
* Chopra et al. (2005) Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In _Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on_, volume 1, pages 539-546. IEEE.
* Clark and Manning (2015) Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL)_, Beijing, China.
* Clark and Manning (2016a) Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, Austin, Texas.
* Clark and Manning (2016b) Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_, Berling, Germany.
* Clevert et al. (2016) Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and accurate deep network learning by exponential linear units (elus). In _Proceedings of the 4th International Conference on Learning Representations (ICLR)_, San Juan, Puerto Rico.
* Das et al. (2016) Arpita Das, Harish Yenala, Manoj Kumar Chinnakotla, and Manish Shrivastava. 2016. Together we stand: Siamese networks for similar question retrieval. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_, Berling, Germany.
* Dipper and Zinsmeister (2012) Stefanie Dipper and Heike Zinsmeister. 2012. Annotating Abstract Anaphora. _Language Resources and Evaluation_, 46(1):37-52.
* Eckert and Strube (2000) Miriam Eckert and Michael Strube. 2000. Dialogue acts, synchronising units and anaphora resolution.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise Phoneme Classification With Bidirectional LSTM And Other Neural Network Architectures. _Neural Networks_, 18:602-610.
* He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In _2015 IEEE International Conference on Computer Vision (ICCV)_.
* Henaff et al. (2016) Mikael Henaff, Arthur Szlam, and Yann LeCun. 2016. Recurrent orthogonal networks and long-memory tasks. In _Proceedings of the the 33rd International Conference on Machine Learning (ICML)_, New York City, USA.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Juhaar et al. (2015) Sujas Kumar Jauhar, Raul Guerra, Edgar Gonzalez Pellicer, and Marta Recasens. 2015. Resolving discourse-deictic pronouns: A two-stage approach to do it. In _Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics_, pages 299-308, Denver, Colorado.
* Joachims (2002) Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In _Proceedings of the 8th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 133-142.
* Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In _Proceedings of the 32nd International Conference on Machine Learning (ICML)_, Lille, France

* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_, San Diego, USA.
* Klein and Manning (2003) Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1_, pages 423-430. Association for Computational Linguistics.
* Kolhatkar and Hirst (2014) Varada Kolhatkar and Graeme Hirst. 2014. Resolving shell nouns. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 499-510, Doha, Qatar.
* Kolhatkar et al. (2013a) Varada Kolhatkar, Heike Zinsmeister, and Graeme Hirst. 2013a. Annotating anaphoric shell nouns with their antecedents. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 112-121, Sofia, Bulgaria.
* Kollhatkar et al. (2013b) Varada Kolhatkar, Heike Zinsmeister, and Graeme Hirst. 2013b. Interpreting anaphoric shell nouns using antecedents of cataphoric shell nouns as training data. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 300-310, Seattle, Washington, USA.
* Lu and Ng (2016) Jing Lu and Vincent Ng. 2016. Event Coreference Resolution with Multi-Pass Sieves. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC)_, pages 3996-4003, Portoroz.
* Maaten and Hinton (2008) Laurens v.d. Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(2579-2605):85.
* Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. _Computational linguistics_, 19(2):313-330.
* Mueller and Thyagarajan (2016) Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In _Proceedings of the 13th Conference on Artificial Intelligence (AAAI)_, pages 2786-2792, Phoenix, Arizona.
* Muller (2008) Christoph Muller. 2008. _Fully Automatic Resolution of It_, This _and That in Unrestricted Multi-Party Dialog_. Ph.D. thesis, Universitat Tubingen, Tubingen.
* Neculoiu et al. (2016) Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. 2016. Learning Text Similarity with Siamese Recurrent Networks. In _Proceedings of the 1st Workshop on Representation Learning for NLP_, pages 148-157, Berlin, Germany.
* Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In _Proceedings of the 30th International Conference on Machine Learning (ICML)_, Atlanta, USA.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Poesio and Artstein (2008) Massimo Poesio and Ron Artstein. 2008. Anaphoric Annotation in the ARRAU Corpus. In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)_, Marrakech, Morocco.
* Rajagopal et al. (2016) Dheeraj Rajagopal, Eduard Hovy, and Teruko Mitamura. 2016. Unsupervised event coreference for abstract words. In _Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods_, pages 22-26, Austin, Texas.
* Rutherford et al. (2017) Attapol T. Rutherford, Vera Demberg, and Nianwen Xue. 2017. A Systematic Study of Neural Discourse Models for Implicit Discourse Relation. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_, 15:1929-1958.
* Stede and Grishina (2016) Manfred Stede and Yulia Grishina. 2016. Anaphoricity in Connectives: A Case Study on German. In _Proceedings of the Coreference Resolution Beyond OntoNotes (CORBON) Workshop_, San Diego, California.
* Strube and Muller (2003) Michael Strube and Christoph Muller. 2003. A machine learning approach to pronoun resolution in spoken dialogue. In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_, pages 168-175, Sapporo, Japan.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL)_, Beijing, China.
* Tryupina et al. (2016) Olga tryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Kepa J Rodriguez, and Massimo Poesio. 2016. ARRAU: Linguistically-Motivated Annotation of Anaphoric Descriptions. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC)_, pages 2058-2062, Portoroz.
* Webber (1991) Bonnie Lynn Webber. 1991. Structure and ostension in the interpretation of discourse deixis. _Language and Cognitive processes_, 6(2):107-135.
* Wiseman et al. (2015) Sam Joshua Wiseman, Alexander Matthew Rush, Stuart Merrill Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for 

coreference resolution. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL)_, Beijing, China.
* Zhou and Xu [2015] Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1127-1137, Beijing, China.