Venue: NAACL 2016

# Unsupervised Learning of Prototypical Fillers

for Implicit Semantic Role Labeling

 Niko Schenk and Christian Chiarcos

Applied Computational Linguistics Lab

Goethe University Frankfurt am Main, Germany

{n.schenk,chiarcos}@em.uni-frankfurt.de

###### Abstract

Gold annotations for supervised implicit semantic role labeling are extremely sparse and costly. As a lightweight alternative, this paper describes an approach based on unsupervised parsing which can do without iSRL-specific training data: We induce prototypical roles from large amounts of explicit SRL annotations paired with their distributed word representations. An evaluation shows competitive performance with supervised methods on the SemEval 2010 data, and our method can easily be applied to predicates (or languages) for which no training annotations are available.

## 1 Introduction

Semantic role labeling (SRL) [5] has become a well-established and highly important NLP component which directly benefits various downstream applications, such as text summarization [8], recognizing textual entailment [36] or QA systems [49, 7]. Its goal is to detect verbal or nominal predicates, together with their associated arguments and semantic roles, either by PropBank/Nombank [34, 48] or FrameNet [3] analysis. In its traditional form, however, SRL is restricted to the _local syntactic context of the predicate_ as in the following example from Ruppenhofer et al. rnp:RuppenhoferEtAl:2010:

\begin{tabular}{l l}  & **[GOLANIn the centre of this room] there was an upright beam, [THEREwhich] had been placed**[THEREat some period] as a support for the old worm-eaten bauk of timber which spanned the roof. \\ \end{tabular} In a FrameNet-style analysis of the sentence, the predicate _place_ evokes the Placing frame, with two frame elements (roles) overtly expressed (Theme and Time) but with one role - Goal - beyond the embedded relative clause and thus beyond the scope of the SRL parser. Such implicit roles, or _null instantiations_ (NIs) [15, 7] are much harder to detect automatically, as they require to broaden the analysis to the surrounding discourse, commonly also to preceding (or following) sentences.

State-of-the-art approaches to implicit SRL (iSRL) are supervised and need a groundwork of hand-annotated training data - which is costly, extremely sparse, limited to only a handful of predicates, and requires careful feature engineering [1, 49, 5]. A first attempt has been made to combine the scarce resources available [10], but given the great diversity of predicate-specific roles and enormous complexity of the task, the main issues remain [3].

A promising exploratory effort recently made by gornisakietal2013 aims to overcome the annotation bottleneck by using distributional methods to infer evidence for elements filling null instantiated roles. The authors do not rely on gold annotations but instead learn distributional properties of fillers induced from a large corpus.

**Our Contribution**: We propose an extension of the distributional idea for unsupervised iSRL to loosen the need for annotated training data. Specifically, we

propose to induce predicate and role-specific ** prototypical fillers** from large amounts of SRL annotated texts in order to resolve null instantiations as (semantically and syntactically) similar elements found in the context. Parts of our approach have been successfully applied in traditional SRL Hermann et al. (2014), but not yet to implicit roles. Our work differs from Gorinski et al. (2013) in that we extend discrete context vectors to SRL-guided embeddings and experiment with a variety of different configurations. We intend _not_ to set a new benchmark beating the current state-of-the-art for _supervised_ iSRL, but rather provide a simple and alternative strategy which does not rely on manually annotated gold data. Still, we demonstrate that our method is highly competitive with supervised methods on one out of two standard evaluation sets and that it can easily be extended to other predicates for which no implicit gold annotations are available.

## 2 Method

### Prototypical Fillers

We use large amounts of _explicit_ SRL annotations to compute predicate-specific _protofillers_ (prototypical fillers) for each frame element (role) individually:

\[\overline{v}^{\textit{protofilter}}=\frac{1}{N}\sum_{i=0}^{N}E(w_{i}) \tag{1}\]

where \(N\) is the total number of tokens filling a particular role and \(E(\cdot)\) is an embedding function which maps a word \(w_{i}\) to its distributed representation, i.e., a precomputed vector of \(d\) dimensions. Note that only those words contribute to the protofiller of a frame element which occur in this role.

### Identifying Null Instantiations

Our approach generalizes over labeled filler instances of the frame (Placing in the example) as found in corpus data, e.g., _placed on the middle picture_, _planted on the top of the church_, _hung over the river_, _laid on the table_, etc. We exploit their syntactic (here: prepositional) and semantic properties (inanimate, spacial NPs) in order to capture a composed meaning and thus to approximate the correct implicit role _in the centre of this room_. We measure similarity between a trained protofiller \(\overrightarrow{v}^{c}\) and a candidate constituent \(\overrightarrow{v}^{c}\) by cosine similarity \(\cos(\theta)=\frac{\overline{v}^{p}\cdot\overrightarrow{v}^{c}}{\|\vec{v}^{ c}\|}\|\frac{1}{\|\vec{v}^{c}\|}\|\) and predict a candidate as null instantiation which maximizes the inner product with the protofiller. As candidate constituents for an implicit argument we initially consider all terminal and non-terminal nodes in a context window of the predicate, ruling out those categories which never occur as implicit arguments, which do contain the target predicate and/or which are already overt arguments. The result set comprises mainly nouns, verbs and PPs. Candidate constituents in our evaluation data are available from their respective (manual) syntax annotation, but could easily be extracted using automated phrase-structure parsers. The candidate _vectors_ for arbitrary length n-grams are derived in the same way (by means of Equation 1).

### Training Resources & Tools

In accordance with domain-specific evaluation data, we chose to learn protofillers on two distinct corpora: _The Corpus of Late Modern English Texts, CLMET_Smet (2005) (\(\approx\)35M tokens, 18th-20th century novels) and a subset of the English _Gigaword corpus_Graff and Cieri (2003) (\(\approx\)500M tokens of Newswire texts). We label the first one with SEMAFOR1Das et al. (2014), a FrameNet-style semantic parser. We employ MATE2Bjorkelund et al. (2009) to obtain a PropBank/NomBank analysis for each sentence in Gigaword.

Footnote 1: [http://www.cs.cmu.edu/~ark/SEMAFOR/](http://www.cs.cmu.edu/~ark/SEMAFOR/)

Footnote 2: [https://code.google.com/p/mate-tools/](https://code.google.com/p/mate-tools/)

Table 1 highlights general statistics on the number of predicates collected from both corpora. Two observations are worth noting: While on average the number of explicitly realized roles/frame elements per predicate/frame in both data sets is similar, we find more predicate instances in CLMET than in Gigaword. This is due to the FrameNet lexicon and its more fine-grained modeling of lexical units, as

\begin{table}
\begin{tabular}{l r|r}  & CLMET & Gigaword \\ \hline \# explicit roles & 21.9M & 264.0M \\ \# predicate instances & 9.5M & 122.5M \\ \# roles per predicate & 2.3 & 2.2 \\ \# predicates per sentence & 7.6 & 4.2 \\ \end{tabular}
\end{table}
Table 1: Statistics on the number of explicit fillers used for training protofillers.

 

opposed to PropBank. Also note that FrameNet currently specifies 9.7 frame elements per lexical frame3 which - despite the fact that this number also comprises non-core arguments - is much larger than what can explicitly be labeled by the SRL systems.

Footnote 3: [https://framenet.icsi.berkeley.edu/findrupal/current_status](https://framenet.icsi.berkeley.edu/findrupal/current_status), accessed March 2016.

Regarding the distributional component, we experimented with a variety of distributed word representations: We chose out of the box vectors; Collobert et al. (2011), dependency-based word embeddings Levy and Goldberg (2014) and the pre-trained Google News vectors from \(word2vec^{4}\)Mikolov et al. (2013). Using the same tool, we also trained custom embeddings (bag-of-words and skip-gram) with 50 dimensions on our two corpora.

## 3 Evaluation

In order to assess the usefulness of our approach, a quantitative evaluation has been conducted on two iSRL test sets which have become a de facto standard in this domain: a collection of fiction novels from the SemEval 2010 Shared Task with manual annotations of null instantiations Ruppenhofer et al. (2010), and Gerber and Chai (2010)'s augmented NomBank data set. Table 2 shows some general statistics on the number of implicit roles and candidate phrases involved in our experiments. As to have a comparison with the supervised approaches referred to in this study, we also provide the size of the training data.

The figures in Table 4 suggest that our approach clearly outperforms the vector-based method by Gorinski et al. (2013) and is best in terms of overall recognition rate (recall) among all systems. One potential reason for that might be that, in contrast to the vec resolver, we do not compute mere context vectors but do rely on the valuable annotations obtained from explicit SRL structures. Also, we do not restrict our analysis to _head_ words only, as we have seen that syntactic information from function words is crucial for the resolution of null instantiated roles, too. Moreover, our distributional protofiller method is highly competitive with state-of-the-art performance by Silberer and Frank (2012), yet does not yield better results in terms of \(F_{1}\) score. Note however that, in contrast to their approach, ours is largely unsupervised and does neither rely on gold coreference chains, nor do we need to train on im

\begin{table}
\begin{tabular}{l c c c}  & SemEval & NomBank \\ \hline \# predicate instances & & & \\ in training set & 1,370 & 816 \\ in test set & 1,703 & 437 \\ \hline \#implicit arguments & & & \\ in training set & 245 & 650 \\ in test set & 259 & 246 \\ \hline \hline \# of candidate phrases & 27.6 & 52.2 \\ per predicate instance & & & \\ proportion of single tokens & 63.4\% & 47.9\% \\ proportion of phrases & 36.6\% & 52.1\% \\ \(\varnothing\) length of candidate phrase & & 5.8 & 7.1 \\ (in tokens) & & & \\ \end{tabular}
\end{table}
Table 2: Statistics on implicit arguments and candidate phrases from the test sections of the two evaluation sets.

 

plicit semantic roles in a supervised setting. An error analysis of our method reveals that it is particularly effective for NIs encountered in the _same sentence_ as the target predicate (44.4% accuracy), which seems plausible given the contextual setup in which protofillers are derived.

### NomBank Data

Compared to the SemEval data, Gerber and Chai (2010)'s augmented NomBank resource covers only ten nominal predicates, which allows us to nicely visualize the distributional profile based on their prototypical fillers. For each predicate, we simply concatenate all per role computed protofillers and apply multidimensional scaling to project the so obtained vectors onto two dimensions (cf. Figure 1).

We observe that the predicate grouping is now based on the prototypical fillers that they co-occur with: In the Wall Street Journal texts, _loss_, _loan_ and _investment_ are similar because their proto-agents (A0 fillers) who lose, lend and invest resp. are semantically shared (i.e. companies, banks). Similarly, _bid_, _cost_ and _fund_ are related in that the targets or commodities (A2) are all money-financed. Finally, the predicates _sale_ and _plan_ are to be expected as outliers as they are less homogeneous in their prototypical argument structure.

We have empirically evaluated our protofiller method also on this data set: Table 3 reports the classification scores for implicit argument resolution compared to the state-of-the-art (Laparra and Rigau, 2013). We restrict the search for implicit arguments to certain predicate-specific parts-of-speech, since some syntactic constituents (e.g., SBAR) never occur as implicit arguments. For choosing the final implicit arguments for each individual predicate instance, we follow the same deterministic strategy as described in Gerber and Chai (2010), which informally states that, if a certain role is not overtly expressed (within a chain of mentions of the same predicate in previous sentences), it is an implicit

Figure 1: Clustered projection of the ten nominal predicates from Gerber and Chai (2010) in protofiller space.

 

candidate. POS lists and cosine similarity thresholds which trigger an actual prediction have been optimized on the development set. The context window for candidate NIs is optimal for the current and previous two sentences in our setting, which explains why the the number of candidate constituents is approximately twice as large for the NomBank predicates (cf. Table 2).

Our best-performing protofillers are again obtained by Collobert et al. (2011) embeddings substituting explicit SRL annotations in the Gigaword corpus, and with custom-trained embeddings using the continuous bag-of-words model. Overall, our results significantly exceed the highly informed baseline but cannot beat the state-of-the art on this test set. For some predicates, the protofillers seem to generalize better (higher recall), and in particular for the low-frequency predicates (_fund_), precision can be increased. Also, we found that the dependency-based word embeddings do perform slightly worse (not shown), compared to our optimal two configurations. This might be due to the fact that the inherent properties of dependency-based contexts mostly focus on relations between semantically valuable nouns, ignoring ("skipping") functional words and categories.6 The same pertains to the pre-computed Google News vectors which come with a frequency cutoff excluding stop words, again a constraint which is harmful for the correct identification of implicit roles. Furthermore, skipgram embeddings perform significantly worse than the embeddings derived by the continuous bag-of-words implementation (relative decrease in \(F_{1}\) by more than 30%). Finally, we observed that inferring implicit roles for nominal predicates is much more challenging because our collected fillers exhibit a much greater variation. For example, the protoagents of _loan_ can roughly be divided into two categories, institutions and countries. This in turn introduces noise and has a negative effect on the quality of the singleton protofillers which by vector average capture neither of the two groups perfectly. Promising alternatives could operate on (topic-like) protofiller **clusters** which we leave for future work.

Footnote 6: This is also nicely illustrated in Levy and Goldberg (2014).

## 4 Summary & Conclusion

We have described a lightweight approach for the resolution of implicit semantic roles which does not rely on manual gold annotations. For each predicate-specific role, our method generalizes over explicit SRL-guided annotations incorporating pretrained word embeddings. This allows us to capture their idiosyncratic properties and use the so-inferred protofillers to find null instantiated roles by means of distributional similarity.

Our method has proven to be generally useful, in particular on the SemEval data, where it is competitive with supervised systems. Its greatest benefit stems from its simplicity and from that fact that it allows to induce null-instantiated roles for arbitrary predicates. As it is applicable even if no iSRL training data is available, it represents a promising technique to address iSRL data scarcity issues.

In our experiment, we employed PropBank/NomBank-style (i)SRL annotations, and our general design clearly benefits from using small-scale inventories of semantic roles. It should be noted though, that our approach is not restricted to any particular SRL tagset, but can be equally applied to other role inventories with similar degrees of consistence and size. Beyond SRL annotations in a strict sense, this might even extend to syntactic dependency annotations that are occasionally taken as a substitute for semantic roles proper. In particular, we see potential in combining our experiments with on-going efforts to cross-lingual projection, adaptation and harmonization of syntax annotations along the lines of Sukhareva and Chiarcos (2014, 2016) and related approaches based on frameworks such as the Universal Dependencies (Nivre, 2015, UD).7 If successful, an adaptation using grammatical relations rather than semantic roles represents a promising possibility to create iSRL annotation and iSRL annotation tools for other languages, as Universal Dependencies are becoming increasingly available for major and low-resourced languages and can be projected to others.

Footnote 7: [http://universaldependencies.github.io](http://universaldependencies.github.io)

The protofillers involved in this study are available at: [http://acoli.cs.uni-frankfurt.de/resources](http://acoli.cs.uni-frankfurt.de/resources).

 



## Acknowledgments

The authors would like to thank Joyce Chai and Hendrik De Smet for providing us access to their resources and corpora. We are grateful to Egoitz Laparra for sending us their evaluation script and also thank the anonymous reviewers for their valuable feedback and insightful comments.

## References

* [Baker et al.1998] Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet Project. In _Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (ACL-COLIG 1998)_, pages 86-90, Montreal, Quebec, Canada.
* [Bjorkelund et al.2009] Anders Bjorkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual Semantic Role Labeling. In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009): Shared Task_, pages 43-48, Boulder, Colorado, June.
* [Chen et al.2010] Desai Chen, Nathan Schneider, Dipanjan Das, and Noah A. Smith. 2010. SEMAFOR: Frame Argument Resolution with Log-linear Models. In _Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval-2010)_, pages 264-267, Los Angeles.
* [Collobert et al.2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* [Das et al.2014] Dipanjan Das, Desai Chen, Andre F. T. Martins, Nathan Schneider, and Noah A. Smith. 2014. Frame-semantic Parsing. _Computational Linguistics_, 40(1):9-56.
* [Feizabadi and Pado2015] Parvin S. Feizabadi and Sebastian Pado. 2015. Combining Seemingly Incompatible Corpora for Implicit Semantic Role Labeling. In _Proceedings of the 4th Joint Conference on Lexical and Computational Semantics (*SEM 2015)_, pages 40-50, Denver, CO.
* [Fillmore1986] Charles J. Fillmore. 1986. Pragrmatically Controlled Zero Anaphora. In _Proceedings of Berkeley Linguistics Society_, pages 95-107, Berkeley, CA.
* [Gerber and Chai2010] Matthew Gerber and Joyce Chai. 2010. Beyond NomBank: A Study of Implicit Arguments for Nominal Predicates. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL-2010)_, pages 1583-1592, Uppsala, Sweden.
* [Greber and Chai2012] Matthew Gerber and Joyce Chai. 2012. Semantic Role Labeling of Implicit Arguments for Nominal Predicates. _Computational Linguistics_, 38(4):755-798.
* [Gildea and Jurafsky2002] Daniel Gildea and Daniel Jurafsky. 2002. Automatic Labeling of Semantic Roles. _Computational Linguistics_, 28(3):245-288.
* [Gorinski et al.2013] Philip Gorinski, Josef Ruppenhofer, and Caroline Sporleder. 2013. Towards Weakly Supervised Resolution of Null Instantiations. In _Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)_, pages 119-130, Potsdam, Germany.
* [Graff and Cieri2003] David Graff and Christopher Cieri. 2003. English Gigaword. Linguistic Data Consortium, Philadelphia, LDC2003T05. Web Download.
* [Hermann et al.2014] Karl Moritz Hermann, Dipanjan Das, Jason Weston, and Kuzman Ganchev. 2014. Semantic Frame Identification with Distributed Word Representations. In _Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics (ACL-2014)_, Baltimore, Maryland.
* [Lagarra and Rigau2012] Egoitz Laparra and German Rigau. 2012. Exploiting Explicit Annotations and Semantic Types for Implicit Argument Resolution. In _Proceedings of the 6th International Conference on Semantic Computing (ICSC-2012)_, pages 75-78.
* [Laparra and Rigau2013] Egoitz Laparra and German Rigau. 2013. ImpAr: A Deterministic Algorithm for Implicit Semantic Role Labelling. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL-2013)_, pages 1180-1189, Sofia, Bulgaria.
* [Levy and Goldberg2014] Omer Levy and Yoav Goldberg. 2014. Dependency-Based Word Embeddings. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL-2014)_, pages 302-308, Baltimore, MD.
* [Li et al.2015] Ru Li, Juan Wu, Zhiqiang Wang, and Qinghua Chai. 2015. Implicit Role Linking on Chinese Discourse: Exploiting Explicit Roles and Frame-to-Frame Relations. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015)_, pages 1263-1271, Beijing, China.
* [Meyers et al.2004] Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The NomBank Project: An Interim Report. In _Proceedings of the HLT-NAACL 2004 Workshop on Frontiers in Corpus Annotation_, pages 24-31, Boston, Massachusetts.
* [Mikolov et al.2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. _CoRR_, abs/1301.3781.
* [Moreda et al.2011] Paloma Moreda, Hector Llorens, Estela Saquete, and Manuel Palomar. 2011. Combining Semantic Information in Question Answering Systems. _Journal of Information Processing and Management_, 47(6):870-885

Joaikim Nivre. 2015. Towards a Universal Grammar for Natural Language Processing. In _Proceedings of the 16th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing-2015)_, pages 3-16, Cairo, Egypt. LNCS 9041, Springer.
* Palmer et al. [2005] Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. _Computational Linguistics_, 31(1):71-106.
* Ruppenhofer et al. [2010] Josef Ruppenhofer, Caroline Sporleder, Roser Morante, Collin Baker, and Martha Palmer. 2010. SemEval-2010 Task 10: Linking Events and Their Participants in Discourse. In _Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval-2010)_, pages 45-50, Los Angeles, CA.
* Ruppenhofer [2005] Josef Ruppenhofer. 2005. Regularities in Null Instantiation. Ms, University of Colorado.
* Sammons et al. [2012] Mark Sammons, V.G.Vinod Vydiswaran, and Dan Roth. 2012. Recognizing Textual Entailment. In Daniel M. Bikel and Imed Zitouni, editors, _Multilingual Natural Language Applications: From Theory to Practice_, chapter 6, pages 209-258. IBM Press.
* Shen and Lapata [2007] Dan Shen and Mirella Lapata. 2007. Using Semantic Roles to Improve Question Answering. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007)_, pages 12-21, Prague, Czech Republic.
* Silberer and Frank [2012] Carina Silberer and Anette Frank. 2012. Casting Implicit Role Linking as an Anaphora Resolution Task. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM-2012)_, pages 1-10, Montreal, Quebec, Canada.
* De Smet [2005] Hendrik De Smet. 2005. A Corpus of Late Modern English Texts. _International Computer Archive of Modern and Medieval English (ICALME)_, 29:69-82.
* Sukhareva and Chiarcos [2014] Maria Sukhareva and Christian Chiarcos. 2014. Diachronic Proximity vs. Data Sparsity in Cross-lingual Parser Projection. A Case Study on Germanic. In _COLING-2014 Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial-2014)_, Dublin, Ireland.
* Sukhareva and Chiarcos [2016] Maria Sukhareva and Christian Chiarcos. 2016. Combining Ontologies and Neural Networks for Analyzing Historical Language Varieties. A Case Study in Middle Low German. In _Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC-2016)_, Portoroz, Slovenia.
* Trandabat [2011] Diana Trandabat. 2011. Using Semantic Roles to Improve Summaries. In _Proceedings of the 13th European Workshop on Natural Language Generation (ENLG-2011)_, pages 164-169, Nancy, France.