Venue: NAACL 2016

# The Instantiation Discourse Relation:

A Corpus Analysis of Its Properties and Improved Detection

Junyi Jessy Li & Ani Nenkova

University of Pennsylvania

{ljunyi,nenkova}@seas.upenn.edu

###### Abstract

instantiation is a fairly common discourse relation and past work has suggested that it plays special roles in local coherence, in sentiment expression and in content selection in summarization. In this paper we provide the first systematic corpus analysis of the relation and show that relation-specific features can improve considerably the detection of the relation. We show that sentences involved in instantiation are set apart from other sentences by the use of gradable (subjective) adjectives, the occurrence of rare words and by different patterns in part-of-speech usage. Words across arguments of instantiation are connected through hypernym and meronym relations significantly more often than in other sentences and that they stand out in context by being significantly less similar to each other than other adjacent sentence pairs. These factors provide substantial predictive power that improves the identification of implicit instantiation relation by more than 5% F-measure.

## 1 Introduction

In an instantiation relation, one text span explains in further detail the events, reasons, behaviors and attitudes mentioned in the other Miltsakaki et al. (2008), as illustrated by the segments below:

**[a]** Other fundamental "reforms" of the 1986 act have been threatened as well.

**[b]** The House seriously considered raising the top tax rate paid by individuals with the highest incomes.

Sentence **[a]** mentions "other reforms" and a threat to them, but leaves unspecified what are the reforms or how they are threatened. Sentence **[b]** provides sufficient detail for the reader to infer more concretely what has happened.

The instantiation relation has some special properties. A study of discourse relations as indicators for content selection in single document summarization revealed that the first sentences from instantiation pairs are included in human summaries significantly more often than other sentences Louis et al. (2010) and that being a first sentence in an instantiation relation is _the_ most powerful indicator for content selection related to discourse relation sense. The sentences between which the relation holds also contain more sentiment expressions than other sentences Trnavac and Taboada (2013), making it a special target for sentiment analysis applications. Moreover, instantiation relations appear to play a special role in local coherence Louis and Nenkova (2010), as the flow between instantiation sentences is not explained by the major coherence theories Kehler (2004); Grosz et al. (1995). Many of the sentences in instantiation relation contain entity instantiations (complex examples of set-instance anaphora), such as "several EU countries"--"the UK", "footballers"--"Wayne Rooney" and "most cosmetic purchase"--"lipstick McKinlay and Markert (2011), raising further questions about the relationship between instantiations and key discourse phenomena.

Detecting an instantiation, however, is hard. In the Penn Discourse Treebank (PDTB) Prasad et al. (2008), Instantiation is one of the few relations that are more often _implicit_, i.e., expressed without a discourse marker such as "for exam

ple". Identifying implicit discourse relation is an acknowledged difficult task [1, 26, 30, 49], but the challenge is exacerbated due to the lack of _explicit_ instantiations: explicit relations are shown to improve their implicit counterparts using data source expansion [57]. Moreover, detecting instantiation also involves the skewed class distribution problem [18] because although it is one of the largest class of implicit relations, it constitutes less than 10% of all the implicit relations annotated in the PDTB.

In this work, we identify a rich set of factors that sets apart each sentence in an implicit instantiation and the pair as a whole. We show that these factors improve the identification of implicit instantiation by at least 5% in F-measure and 8% in balanced accuracy compared to prior systems.

## 2 Presence of instantiation

We use the Penn Discourse Tree Bank (PDTB) [21] for the analysis and experiments presented in this paper. There are a total of 1,747 instantiation relations in the PDTB, of which 83% are implicit. Instantiations make up 8.7% of all implicit relations and is the 5th largest among the 16 second-level relations in the PDTB.

## 3 Characteristics of instantiation

We identify significant factors1 that characterize: _(i)_\(s_{1}\) and \(s_{2}\): the first and second sentence in an instantiation pair vs. all other sentences; _(ii)_\(s_{1}\) vs. \(s_{2}\): adjacent sentence pairs in instantiation relation vs. all other adjacent sentence pairs.

Footnote 1: \(p<0.05\) according to paired Wilcoxon signed rank test for real valued comparison between the two sentences in a relation, non-paired Wilcoxon rank sum for real valued factors in different types of sentences, and Kruskal-Wallis for binary valued features across different types of sentences.

Our analysis is conducted on the PDTB except section 23, which is reserved for testing as in prior work [18, 19]. In total, there are 1,337 instantiation sentence pairs and 43,934 non-instantiation sentences for the corpus analysis.

Sentence length.Intuitively, longer sentences are more likely to involve details. Table 1 demonstrates that there is an average of 8.4-word difference in length between the two sentences in an instantiation relation; moreover, \(s_{1}\)s are significantly shorter (more than 5 words on average) than other sentences, and \(s_{2}\)s are significantly longer.

Rare words.For each sentence, we compute the percentage of words that are not present in the 400K vocabulary of the Glove vector representations [10]. Table 2 shows that \(s_{1}\) of instantiations contain significantly _fewer_ out-of-vocabulary words compared to either \(s_{2}\) and non-instantiations. We also compare the difference in unigram probability2 of content word pairs across sentence pairs, i.e., \((w_{i},w_{j}),w_{i}\in s_{1},w_{j}\in s_{2}\). Compared to non-instantiation, words across instantiation arguments show significantly larger average unigram log probability difference (1.24 vs. 1.22). These numbers show that the first sentences of instantiation do not involve many unfamiliar words -- an indication of higher readability [18].

Footnote 2: We use a unigram language model on year 2006 of the New York Times Annotated Corpus [26].

Gradable adjectives.The use of gradable adjectives [26, 27, 28]--_popular, high, likely_-- may require further explanation to justify the appropriateness of their use. Here we compute the average percentage of gradable adjectives in a sentence. The list of adjectives is from Hatzivassiloglou2000 and the respective percentages are shown in Table 3. Compared to other sentences, \(s_{1}\) of instantiation involves significantly m 

Parts of speech.We study word categories that are heavily or rarely used with instantiation by inspecting the percentage of part-of-speech tags found in each sentence. In Table 4, we show POS tags whose presence is significantly different across arguments in instantiation but not so across non-instantiation, with significance in non-instantiation in the reverse direction denoted in \({\dagger}\). Four cases of POS occurrences are inspected:

* more often in \(s_{1}\) compared to \(s_{2}\),
* more often in \(s_{2}\) compared to \(s_{1}\),
* more (+) or less (-) in \(s_{1}\) compared to non-instantiation,
* more (+) or less (-) in \(s_{2}\) compared to non-instantiation.

We see that \(s_{1}\) of instantiation contains more characteristic POS usage than \(s_{2}\). There are more comparative adjectives and adverbs as well as fewer nouns in \(s_{1}\) compared to \(s_{2}\) in instantiation pairs. The usage of verbs is also different between two arguments and \(s_{1}\) contains more conjunctions and existential there. On the other hand, \(s_{2}\) contains more nouns, numbers, determiners, personal pronouns and proper nouns, intuitively associated with the presence of detailed information.

Wordnet relations.Here we consider word-level relationships across arguments using Wordnet [7]. For each noun, verb, adjective and adverb content word pairs across arguments, we calculate the percentage of sentences with each type of Wordnet relation. Shown in Table 5, among instantiation sentence pairs there are significantly more noun-noun pairs with hypernym or meronym relationships and verbs with indirect hypernym relationship. We also observe significantly more semantically similar verbs (_group (v)_).

Lexical similarity.Finally, we inspect the similarity between sentences in each pair as well as between each sentence in a pair and their immediate prior context; specifically:

* Between \(s_{1}\) and \(s_{2}\);
* Between \(s_{1}\) and \(C\) and between \(s_{2}\) and \(C\), where C denotes up to two sentences immediately before \(s_{1}\).

We compute the Jaccard similarity between sentences using their nouns, verbs, adjectives and adverbs. Instantiation arguments are significantly less similar than other adjacent sentence pairs (0.335 vs. 0.505), indicating higher difference in content.

\begin{table}
\begin{tabular}{c|c|c|c}  & \(s_{1}\) & \(s_{2}\) & \(\Delta\text{sim}\) \\ \hline Inst. & 0.282\({}^{*}\) & 0.275\({}^{*}\) & 0.007 \\ \(\neg\) Inst. & 0.390 & 0.358 & 0.042\({}^{*}\) \\ \end{tabular}
\end{table}
Table 6: Average Jaccard similarity of an adjacent sentence pair \(s_{1}\), \(s_{2}\) with immediate context. [\(*\)]: significant (\(p<0.05\)) compared to non-instantiation sentence pairs.

\begin{table}
\begin{tabular}{l|l|c}  & \(s_{1}\) & \(s_{2}\) & \(\neg\) Inst. \\ \hline \multicolumn{3}{l|}{%gradable adj} & 2.96\({}^{*}\) & 2.22 & 2.22 \\ \cline{2-3} \multicolumn{1}{r|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\ \end{tabular}
\end{table}
Table 3: Average % of gradable adjectives per sentence. [\(*\)]: significant (\(p<0.05\)) compared to non-instantiation sentences.

\begin{table}
\begin{tabular}{c|c|c}  & CC EX JJR JS NNS PDT RB\({}^{\dagger}\) RBR VBG \\  & VBN PBP VBZ\({}^{\dagger}\) \\ \hline \multicolumn{3}{l|}{\(s_{1}<s_{2}\)} & NN NNP\({}^{\dagger}\) PRP TO VBD WRB \\ \hline \(s_{1}\) vs \(\neg\)Inst. & CD\({}^{-}\) JJ\({}^{-}\) MD\({}^{-}\) NN\({}^{-}\) NNP\({}^{-}\) NNS\({}^{+}\) PRP\({}^{-}\) \\  & RB\({}^{+}\) TO\({}^{-}\) VB\({}^{-}\) VBD\({}^{-}\) VBG\({}^{+}\) VBP\({}^{+}\) \\  & VBZ\({}^{\pm}\) WDT\({}^{-}\) \\ \hline \(s_{2}\) vs \(\neg\)Inst. & CD\({}^{+}\) DT\({}^{+}\) MD\({}^{-}\) NNP\({}^{+}\) NSN\({}^{+}\) PRP\({}^{+}\) RB\({}^{-}\) \\  & VB\({}^{-}\) VBN\({}^{-}\) \\ \end{tabular}
\end{table}
Table 4: POS tag significantly different in percentage compared to non-instantiation. [\({\dagger}\)]: significance in non-instantiation pairs in the other direction. [\({\dagger}\)+/\({-}\)]: used more/less often than non-instantiation.

\begin{table}
\begin{tabular}{c|c|c} Relation (pos) & Inst. & \(\neg\) Inst. \\ \hline hypernym (n) & 18.01\({}^{*}\) & 21.63 \\ meronym (n) & 18.66\({}^{*}\) & 15.48 \\ holonym (n) & 17.14 & 15.07 \\ indirect hypernym (n) & 19.17 & 21.03 \\ hyponym (n) & 20.33 & 21.75 \\ \hline group (v) & 72.5\({}^{*}\) & 68.7 \\ indirect hypernym (v) & 38.5\({}^{*}\) & 41.7 \\ hypernym (v) & 76.25 & 74.79 \\ hyponym (v) & 80.76 & 78.44 \\ entailment (v) & 4.94 & 4.18 \\ cause (v) & 17.65 & 17.00 \\ \hline similar to (adj) & 3.78 & 2.77 \\ also see (adj) & 6.25 & 5.78 \\ \end{tabular}
\end{table}
Table 5: Percentage of sentence pairs with Wordnet relationships. [\(*\)]: significant (\(p<0.05\)) compared to non-instantiation sentence pairs.

 

Shown in Table 6, both arguments of instantiation are less similar to the immediate context. While other sentence pairs follow the pattern that is much less similar to's immediate context, this phenomenon is not significant for instantiation.

## 4 Experiments

In this section, we demonstrate the benefit of exploiting instantiation characteristics in the identification of the relation.

Settings.Following prior work that identifies the more detailed (second-level) relations in the PDTB Biran and McKeown [2015]; Lin et al. [2014], we use sections 2-21 as training, section 23 as testing. The rest of the corpus is used for development. The task is to predict if an implicit instantiation relation holds between pairs of adjacent sentences in the same paragraph. Sentence pairs with instantiation relation constitute the positive class; all other non-explicit relations3 constitute the negative class. We use Logistic Regression with class weights inversely proportional to the size of each class.

Footnote 3: including AltLex, EntRel and NoRel

Features.The factors discussed in SS 3 are adopted as the _only_ features in the classifier. We use the average values of \(s_{1}\) and \(s_{2}\) and their difference for: the number of words, difference in number of words compared to the sentence before \(s_{1}\), the percentage of OOVs, gradable adjectives, POS tags and Jaccard similarity to immediate context. We use the minimum, maximum and average differences in word-pair unigram log probability, and average Jaccard similarity across sentence pairs. For Wordnet relations, we use binary features indicating the presence of a relation.

Results.To compare with our instantiation-specific classifier (_Inst. specific_), we show results from two state-of-the-art PDTB discourse parsers that identify second-level relations: Biran and McKeown [2015] (_B&M_) and Lin et al. [2014]. We also compare the results with the classifier from our prior work Li and Nenkova [2014b] (_L&N_). In that work we introduce syntactic production-stick features, which minimize the occurrence of zero values in instance representation. Furthermore, we re-implemented Brown-cluster features (concatenation of clusters in each sentence) that have been shown to perform well in identifying instantiation's parent class expansion Braud and Denis [2015].4

Footnote 4: The dimension of clusters are tuned on the development set. As in prior work, we use clusters in Turian et al. [2010].

Table 7 shows the precision, recall, F-measure and balanced accuracy (average of the accuracies for the positive and negative class respectively) for each system. We show balanced accuracy rather than overall accuracy due to the highly skewed class distribution. For _Inst. specific_, we use a threshold of 0.65 for positive labels5. We also use it along with L&N for a _soft voting_ classifier where the label is assigned to the class with larger weighted posterior probability sum from each classifier6. Both classifiers achieved at least 5% improvement of F-measure and 8%-10% improvement of balanced accuracy compared to other systems. These improvements mostly come from a dramatic improvement in recall. The improvement achieved by the voting classifier also indicate that _Inst. specific_ provide complementary signals to syntactic production rules. Note that compared to Lin et al., _Inst. specific_ behaves very differently in precision and recall, indicating potential for further system combination.

Footnote 5: Tuned on development set.

Finally, we analyze the confusion matrix induced by false positives and false negatives across Lin et al., B&M, _Inst. specific_ and soft vote7. In Table 8, we list relations contributing at least 10% to false positives for at least one system. Remarkably, instantiatio

\begin{table}
\begin{tabular}{c|c|c|c|c} System & P & R & F & BA \\ \hline Inst. specific & 0.3072 & 0.6986 & **0.4268** & **0.7862** \\ Vote (L\&N) & 0.3052 & 0.6438 & 0.4141 & 0.7632 \\ \hline L\&N & 0.3028 & 0.4521 & 0.3626 & 0.6843 \\ B\&M & 0.2542 & 0.2055 & 0.2273 & 0.5786 \\ Lin et al. & 0.5500 & 0.1507 & 0.2366 & 0.5704 \\ Brown-concat & 0.1333 & 0.3836 & 0.1979 & 0.5919 \\ \end{tabular}
\end{table}
Table 7: Precision, recall, F-measure and balanced accuracy of identifying instantiation.

 

a constant set of relations: restatement, cause and EntRel. Different from other systems, the _Inst. specific_ classifier demonstrates more confusion towards cause than restatement. On the false negative side, all 62 mistakes are not annotated (i.e., NoRel/EntRel) in Lin et al. For the 58 false negatives in B&M, we observe above 10%: NoRel/EntRel (56.9%), cause (19.0%), restatement (13.8%), consistent with relations relations involved in false positives.

## 5 Conclusion

We have characterized the implicit instantiation relation by studying significant factors that discriminate individual arguments and the sentence pairs connected by the relation. We show distinctive patterns in sentence length, word usage, semantic relationships between words as well as cross-argument and contextual similarity associated with instantiation. Using these factors as features we demonstrate significant improvement on the detection of implicit instantiation relation.

## References

* Biran and McKeown [2013] Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: Short Papers_, pages 69-73.
* Brian and McKeown [2015] Or Biran and Kathleen McKeown. 2015. PDTB discourse parsing as a tagging task: The two taggers approach. In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 96-104.
* Braud and Denis [2015] Chloe Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2201-2211.
* de Marneffe et al. [2010] Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. "Was it good? It was provocative." Learning the meaning of scalar adjectives. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 167-176.
* Fellbaum [1998] Christiane Fellbaum. 1998. _WordNet_. Wiley Online Library.

* Grosz et al. [1995] Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. _Computational Linguistics_, 21(2).
* Volume 1_, pages 299-305.
* Ji and Eisenstein [2015] Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. _Transactions of the Association for Computational Linguistics_, 3:329-344.
* Kehler [2004] Andrew Kehler. 2004. Discourse coherence. _The handbook of pragmatics_, pages 241-265.
* Li and Nankova [2014a] Junyi Jessy Li and Ani Nankova. 2014a. Addressing class imbalance for improved recognition of implicit discourse relations. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 142-150.
* Li and Nenkova [2014b] Junyi Jessy Li and Ani Nenkova. 2014b. Reducing sparsity improves the recognition of implicit discourse relations. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 199-207.
* Lin et al. [2009] Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_, pages 343-351.
* Lin et al. [2014] Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A PDTB-styled end-to-end discourse parser. _Natural Language Engineering_, 20:151-184.
* Louis and Nenkova [2010] Annie Louis and Ani Nenkova. 2010. Creating local coherence: An empirical assessment. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 313-316.
* Louis et al. [2010] Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In _Proceedings of the SIGDIAL 2010 Conference_, pages 147-156.

\begin{table}
\begin{tabular}{c|c|c|c|c} Relation & Lin et al. & B\&M & Inst. & Vote \\ \hline Restatement & 33.3 & 34.1 & 28.7 & 30.8 \\ Cause & 22.2 & 25.0 & 33.9 & 33.6 \\ Contrast & 22.2 & 6.8 & 9.6 & 10.3 \\ Conjunction & 11.1 & 4.5 & 6.1 & 2.8 \\ EntRel & 11.1 & 18.2 & 17.4 & 19.6 \\ \end{tabular}
\end{table}
Table 8: Relations involved in false positives, \(\geq 10\%\) for least one system.

 

Andrew McKinlay and Katja Markert. 2011. Modelling entity instantiations. In _Proceedings of the International Conference Recent Advances in Natural Language Processing 2011_, pages 268-274.
* Miltsakaki et al. (2008) Eleni Miltsakaki, Livio Robaldo, Alan Lee, and Aravind Joshi. 2008. Sense annotation in the Penn Discourse Treebank. In _Computational Linguistics and Intelligent Text Processing_, volume 4919 of _Lecture Notes in Computer Science_, pages 275-286. Springer Berlin Heidelberg.
* Park and Cardie (2012) Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In _Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 108-112.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. _Proceedings of the Empirical Methods in Natural Language Processing_, 12:1532-1543.
* Pitler and Nenkova (2008) Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 186-195.
* Pitler et al. (2009) Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 683-691.
* Prasad et al. (2008) Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In _Proceedings of the International Conference on Language Resources and Evaluation_.
* Rutherford and Xue (2014) Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_, pages 645-654.
* Rutherford and Xue (2015) Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 799-808.
* Sandhaus (2008) Evan Sandhaus. 2008. The New York Times Annotated Corpus LDC2008T19. _Linguistic Data Consortium, Philadelphia_.
* Trinavac and Taboada (2013) Radoslava Trinavac and Maite Taboada. 2013. Discourse relations and affective content in the expression of sentiment in texts. In _11th ICGL Conference-Workshop on The semantic field of emotions: Interdisci_.
* Turian et al. (2010) Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 384-394.