Venue: NAACL 2021

# Template Filling with Generative Transformers

Xinya Du  Alexander M. Rush   Claire Cardiie

Department of Computer Science

Cornell University

{xdu, cardiie}@cs.cornell.edu

arush@cornell.edu

###### Abstract

Template filling is generally tackled by a pipeline of two separate supervised systems - one for role-filler extraction and another for template/event recognition. Since pipelines consider events in isolation, they can suffer from error propagation. We introduce a framework based on end-to-end generative transformers for this task (i.e., Gtt). It naturally models the dependence between entities both within a single event and across the multiple events described in a document. Experiments demonstrate that this framework substantially outperforms pipeline-based approaches, and other neural end-to-end baselines that do not model between-event dependencies. We further show that our framework specifically improves performance on documents containing multiple events.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 909–914

## 1 Introduction

The classic template-filling task in information extraction involves extracting event-based templates from documents Grishman and Sundheim (1996); Jurafsky and Martin (2009); Grishman (2019). It is usually tackled by a pipeline of two separate systems, one for _role-filler entity extraction_ - extracting event-relevant entities (e.g., noun phrases) from the document; another for _template/event recognition_ - assigning each of the candidate role-fillers to the event(s)/template(s) that it participates in and identifying the type of each event/template.

Simplifications of the task Patwardhan and Riloff (2009); Huang and Riloff (2011, 2012); Du et al. (2020) assume that there is one generic template and focus only on role-filled entity extraction. However, real documents often describe multiple events (Figure 1). From the example, we can observe that between-event dependencies are important (e.g., a single organization can participate in multiple events) and can span the entire document (e.g., event-specific targets can be distant from their shared perpetrator organization). Alternative end-to-end event extraction models, even those incorporating pretrained LM representations, only model events in isolation Wadden et al. (2019); Du and Cardiie (2020), and are mainly evaluated on ACE-style Doddington et al. (2004) event extraction from single sentences Yang and Mitchell (2016); Lin et al. (2020).

To naturally model between-event dependencies across a document for template filling, we propose a framework called "Gtt" based on generative transformers (Figure 2). To our best knowledge, this is the first attempt to build an end-to-end learning framework for this task. We build our framework upon GfitDu et al. (2020), which tackles role-filler entity extraction (REE), but not template/event recognition. Gfit performs REE by "generating" a sequence of role-filler entities, one role at a time in a prescribed manner. For the template-filling setting, we first extend the Gfit approach to include tokens representing event types

Figure 1: The template-filling task. Role-filler entity extraction is shown on the left, and template recognition is shown on the right. Our system performs both of these document-level tasks with a single end-to-end model.



(e.g., "attack", "bombing") as part of the input sequence. We further modify the decoder to attend to the event type tokens, allowing it to distinguish among events and associate event types to each role-filler entity that it generates.

We evaluate our model on the MUC-4 (1992) template filling task. Empirically, our model substantially outperforms both pipeline-based and end-to-end baseline models. In our analysis, we demonstrate that our model is better at capturing between-event dependencies, which are critical for documents that describe multiple events. Code and evaluation scripts for the project is open-sourced at [https://github.com/xinyadu/gtt](https://github.com/xinyadu/gtt).

## 2 Task Definition: Template Filling

Assume we are given a set of \(m\) event types (\(T_{1}\), ..., \(T_{m}\)). Each event template contains a set of \(k\) roles (\(r_{1}\), ..., \(r_{k}\)). For a document consisting \(n\) words \(x_{1}\), \(x_{2}\), ..., \(x_{n}\), the system is required to extract \(d\) templates, where \(d\geq 0\) (\(d\) is not given as input). Each template consists of \(k+1\) slots: the first slot represents the event type (one of \(T_{1}\), ..., \(T_{m}\)). The rest of the \(k\) slots correspond to an event role (one of \(r_{1}\), ..., \(r_{k}\)). The system is required to fill in entities for the corresponding role, which may be filled in as null.

## 3 Methodology

Our framework is illustrated in Figure 2. First we transform the template filling task into a sequence generation problem. Then, we train the base model on the source-target sequence pairs, and apply the model to generate the sequence; finally the sequence is transformed back to structured templates.

### Template Filling as Sequence Generation

We first transform the task's input and output data into specialized source and target sequence pair encodings. As shown in Figure 2 and below, the _source sequence_ consists of the words of the document (\(x_{1}\), \(x_{2}\), ..., \(x_{n}\)) prepended with the general set of tokens representing all event/template types (\(T_{1}\), ..., \(T_{m}\)); as well as a separator token denoting the boundary between event templates ([SEP_T]). We also add a classification token ([CLS]) and another separator token ([SEP]) at the beginning and end of this source sequence. [CLS] works as the start token, [SEP] denotes the boundary between REEs.

[CLS] \(T_{1},...,T_{m}\) [SEP_T] \(x_{1},x_{2},...,x_{n}\) [SEP]

The _target sequence_ consists of the concatenation of template extractions, separated by the separator token ([SEP_T]). For template \(i\), the subsequence consists of its event type \(T^{(i)}\) and its role

Figure 2: Our generative framework for end-to-end template filling.

 

filler entity extractions \(<\) Role-filled Entities \(>^{(i)}\):

[CLS] \(T^{(1)},<\) Role-filler Entities \(>^{(1)}\)

[SEP_T] \(T^{(2)},<\) Role-fuller Entities \(>^{(2)}\)

...

[SEP_T] \(T^{(i)},<\) Role-filler Entities \(>^{(i)}\)

...

For the \(<\) Role-filler Entities \(>\) of template \(i\), following Du et al. (2020), we use the concatenation of target entity extractions for each role, separated by the separator token ([SEP]). Each entity is represented with its first mention's beginning (\(b\)) and end (\(e\)) tokens:

\[e^{1}_{1_{b}},e^{1}_{1_{e}},..\text{ [SEP] $e^{2}_{1_{b}},e^{2}_{1_{e}},..$ [SEP] $e^{3}_{1_{b}},e^{3}_{1_{e}},..$}\]

### Base Model and Decoding Constraints

Next we describe the base model as well as special decoding constraints for template filling.

BERT as Encoder and DecoderOur model extends upon the Grit model for REE (Du et al., 2020). The base setup utilizes one BERT (Devlin et al., 2019) model for processing both the source and target tokens embeddings. To distinguish the encoder / decoder representations, it uses partial causal attention mask on the decoder side (Du et al., 2020). The joint sequence of source tokens' embeddings (\(\mathbf{a}_{0},\mathbf{a}_{1},...,\mathbf{a}_{m}\)) and target tokens' embeddings (\(\mathbf{b}_{0},\mathbf{b}_{1},...,\mathbf{b}_{n}\)) are passed through BERT to obtain their contextualized representations,

\[\hat{\mathbf{a}}_{0},\hat{\mathbf{a}}_{1},...,\hat{\mathbf{a}}_{ l_{src}},\hat{\mathbf{b}}_{0}...,\hat{\mathbf{b}}_{l_{tgt}}\] \[=\texttt{BERT}(\mathbf{a}_{0},\mathbf{b}_{1},...,\mathbf{a}_{l_ {src}},\mathbf{b}_{0},...,\mathbf{b}_{l_{tgt}})\]

Pointer DecodingFor the final decoder layer, we replace word prediction with a simple pointer selection mechanism. For target time step \(t\), we first calculate the dot-product between \(\hat{\mathbf{b}}_{t}\) and \(\hat{\mathbf{a}}_{0},\hat{\mathbf{a}}_{1},...,\hat{\mathbf{a}}_{m}\),

\[c_{0},c_{1},...,c_{l_{src}}=\hat{\mathbf{b}}_{t}\cdot\hat{\mathbf{a}}_{0}, \hat{\mathbf{b}}_{t}\cdot\hat{\mathbf{a}}_{1},...,\hat{\mathbf{b}}_{t}\cdot \hat{\mathbf{a}}_{l_{src}}\]

Then we apply softmax to \(c_{0},c_{1},...,c_{l_{src}}\) to obtain the probabilities of pointing to each source token (which may be a word or an event type), test prediction is done with greedy decoding. At each time step, argmax is applied to find the source token which has the highest probability. The decoding stops when a stop token is predicted.

\[p_{0},p_{1},...,p_{l_{src}}=\text{softmax}(c_{0},c_{1},...,c_{l_{src}})\]

We also add several special decoding constraints for template filling: (1) downweighting factor (0.01) to the probability of generating [SEP] and [SEP_T], in order to calibrate recall; (2) decoding cutoff stop when it ends the \(k^{th}\) template (\(k=\)maximum number of events in one document); (3) a constraint to ensure that the pointers for the start and end token for one entity are in order.

## 4 Experiments

We conduct evaluations on the MUC-4 dataset (1992). MUC-4 consists of 1,700 documents with associated templates. We follow prior work in split: 1,300 documents for training, 200 documents (IST1+TST2) as the development set and 200 documents (TST3+TST4) as the test set. We use the metric for template filling (Chinchor, 1992) and, as in previous work, map predicted templates to gold templates during evaluation so as to optimize scores. We follow content-based mapping restrictions, i.e., the event type of the template is considered essential for the mapping to occur.1 Missing template's slots are scored as missing, spurious template's slots are scored as spurious. Note that in our work, since we do not extract the set fillers other than the event/template type, they do not affect the performance.

Footnote 1: The content-based mapping restrictions were added to MUC-4 to prevent fortuitous mappings which occurred in MUC-3 (Chinchor, 1992).

Baselines and Additional Related WorkAs an ablation baseline, we employ a pipeline, Griptpieline, that first uses the Grit model for role-filler entity extraction, and then assigns event types to each of the entities as a multi-label classification problem. We assign types by transforming the problem to multi-class classification (MCC) (Spolaor et al., 2013). As there are 6 event types (i.e., _kindapping_, _attack_, _bombing_, _robbery_, _arson_, _forced work stoppage_) in MUC-4, we use \(2^{6}\) labels for the MCC problem.

We also compare to end-to-end baselines without modeling betw 

DyGIE++ (Wadden et al., 2019)2 is a span-enumeration based extractive model for information extraction. The model enumerates all the possible spans in the document and passes each representation through a classifier layer to predict whether the span represents certain role-filler entity and what the role is. seqTagging is a BERT-based sequence tagging model for extracting the role-fillers entities. A role-filler entity can appear in templates of different event types (e.g., "Zarate armed force" appear in both attack and bombing event). For both baselines, the prediction goal is multi-class classification. More specially, we _adapt_ the DyGIE++ output layer implementation to first predict the role-filler entity's role class, and then predicts its event classes conditioned on the entity's role.

Footnote 2: Our own re-implementation.

Note that Chambers (2013) and Cheung et al. (2013) propose to do event schema induction with unsupervised learning. Given their unsupervised nature, empirically the performance is worse than supervised models Patwardhan and Riloff (2009). Thus we do not add these as comparisons.

## 5 Results and Analysis

Results on the full test set are shown in Table 2. We report the micro-average performance (precision, recall and F1). We see that our framework substantially outperforms the baseline extraction models in precision, recall and F1, with approximately a 4% F1 increase over the end-to-end baselines. It outperforms the Grit-pipeline system by around 3% F1 (\({}^{*}\) denotes \(p<0.05\)).

Per-slot F1 score is reported in Table 1. The results demonstrate that our framework more often predicts the correct event type, performs better on PerPind and PerOrg, and achieves slightly worse performance with Grit-pipeline on roles that appear later in the template (i.e., Target and Victim). We also found that DytGE++ performs better on Target, mainly due to its high precision in role assignment for spans.

Between-Event DependenciesWe also show results (Table 3) on the subset of documents that contains more than one gold event. We see the F1 score for all systems drops substantially, proving the difficulty of the task, as compared to the single/no event case. When compared to the Full Test setting in Table 2, the baselines all increase in precision and drop substantially in recall, while our approach's precision and recall drop a little. This change is understandable, as the baseline systems are more conservative and tend to predict fewer templates. As the number of gold templates increases, the fewer templates predictions have a better chance of getting matched, but their recall drops as well.

How performance changes when \(E\) increasesIn Figure 3, we see that when the number of gold events in the document is smaller (\(E=1,2\)), our approach performs on par with the pipeline-based and DyGIE++ baselines. However, as \(E\) grows larger, the baselines' F1 drop significantly (e.g., over -10% as \(E\) grows from 2 to 3).

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline Models & Event Type & PerPind & PerOrg & Target & Victim & Weapon \\ \hline Grit-pipeline & 62.28 & 38.40 & 35.36 & 36.30 & **54.97** & 53.45 \\ DyGIE++ Wadden et al. (2019) & 61.95 & 32.44 & 25.73 & **45.04** & 49.48 & 51.60 \\ seqTagging Du and Cardie (2020) & 60.22 & 30.59 & 26.79 & 36.60 & 43.62 & 51.70 \\ \hline Gtt & **67.44** & **44.04** & **41.79** & 32.39 & 54.12 & **59.71** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Per-slot F1 score.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline Models & P & R & F1 \\ \hline Grit-pipeline & **63.88** & 37.56 & 47.31 \\ DyGIE++ & 61.90 & 36.33 & 45.79 \\ seqTagging & 46.80 & 38.30 & 42.13 \\ \hline Gtt & 61.69 & **42.36** & **50.23* 

Qualitative Case AnalysisConsider the input document (doc id TST3-MUC4-0080)3, which contains an attack and a bombing template. In the gold annotations, "Farabundo Marti National Liberation Front" acts as PerpOrg in both events. Our model correctly extracts the two events and the PerpOrg in each while DyGIE++ only predicts the attack event with its PerpOrg role entity correctly. Although Grit-pipeline gets both events correct, it failed to extract this PerpOrg entity for the second event.

Footnote 3: For the full document, please refer to Appendix A.

## 6 Conclusion

We revisit the classic NLP problem of template filling and propose an end-to-end learning framework called GTT. Through modeling events relation, our approach better captures dependencies across the document and performs substantially better on multi-event documents.

## Acknowledgments

We thank the anonymous reviewers for helpful feedback and suggestions. The work of XD and AMR was supported by NSF CAREER 2037519; that of XD and CC was supported in part by DARPA LwLL Grant FA8750-19-2-0039.

## References

* T. Chambers (2013)Event schema induction with a probabilistic entity-driven model. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807. Seattle, Washington, USA. External Links: ISBN 978-1-4503-3 Cited by: SS1.
* J. Chi Kit Cheung, H. Poon, and L. Vanderwende (2013)Probabilistic frame induction. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, pp. 837-846. External Links: Link Cited by: SS1.
* N. Chinchor (1992)Muc-4 evaluation metrics. In MUC, M., K. Lee, and K. Toutanova (Eds.), Vol. 2019, Berlin, Heidelberg, Germany, pp. 1137-1147. External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1.
* G. R. Doddington, A. Mitchell, M. A. Przybocki, L. A. Ramshaw, S. M. Strassel, and R. W. Weischedel (2004)The automatic content extraction (ace) program-tasks, data, and evaluation. In Lrec, Vol. 2, pp. 1. Lisbon. External Links: Document Cited by: SS1.
* X. Du and C. Cardie (2020)Document-level event role filler extraction using multi-granularity contextualized encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8010-8020. External Links: Link Cited by: SS1.
* X. Du, A. M. Rush, and C. Cardie (2020)GriT: generative role-filler transformers for document-level event entity extraction. In EACL, Cited by: SS1.
* R. Grishman (2019)Twenty-five years of information extraction. Natural Language Engineering25 (6), pp. 677-692. External Links: ISSN 0037-692X, Link, Document Cited by: SS1.
* R. Grishman and B. Sundheim (1996)Design of the MUC-6 evaluation. In TIPSTER TEXT PROGRAM PHASE II: Proceedings of a Workshop held at Vienna, Virginia, May 6-8, 1996, pp. 413-422. External Links: Link, Document Cited by: SS1.
* R. Huang and Ellen Ridoff (2011)Peeling back the layers: detecting event role fillers in secondary contexts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA, pp. 1137-1147. External Links: Link, Document Cited by: SS1.
* R. Huang and E. Riloff (2012)Modeling textual cohesion for event extraction. In Twenty-Sixth AAAI Conference on Artificial Intelligence, Cited by: SS1.
* D. Jurafsky and J. H. Martin (2009)Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2nd edition, Prentice Hall series in artificial intelligence. Prentice Hall, Pearson Education International.

Figure 3: F1 on subset of documents with \(E\) events.

 

* Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfeu Wu. 2020. A joint neural model for information extraction with global features. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7999-8009, Online. Association for Computational Linguistics.
* MUC-4 (1992) MUC-4. 1992. Fourth message understanding conference (MUC-4). In _Proceedings of FOWTH MESSAGE UNDERSTANDING CONFERENCE (MUCS-4)_, McLean, Virginia.
* Patwardhan and Riloff (2009) Siddharth Patwardhan and Ellen Riloff. 2009. A unified model of phrasal and sentential evidence for information extraction. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_, pages 151-160, Singapore. Association for Computational Linguistics.
* Spolaor et al. (2013) Newton Spolaor, Everton Alvares Cherman, Maria Carolina Monard, and Huei Diana Lee. 2013. A comparison of multi-label feature selection methods using the problem transformation approach. _Electronic Notes in Theoretical Computer Science_, 292:135-151.
* Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)_, pages 5784-5789, Hong Kong, China. Association for Computational Linguistics.
* Yang and Mitchell (2016) Bishan Yang and Tom M. Mitchell. 2016. Joint extraction of events and entities within a document context. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 289-299, San Diego, California. Association for Computational Linguistics.

 

Example document for qualitative analysis

Official sources today reported that at least eight people, including soldiers, rebels, and civilians, were killed during clashes between the army and guerrillas over the past weekend in various points of the country.

Military spokesmen for the 6th infantry brigade, headquartered in the eastern usulutan department, told acanefe that two rebels were killed and one wounded during a clash with government troops in San Agustin.

Meanwhile, the armed forces press committee (Coprefa) reported that the bodies of two guerrillas, who were presumably killed during clashes with the army, were found by soldiers in the outskirts of Santa Tecla, in the central la libertad department.

Coprefa reported that two soldiers were killed during a clash with members of the **Farabundo Marti National Liberation Front** (FMLN) in Comasagua, about 28 km to the southwest of (San) Salvador, where a rebel attack on a coffee processing plant was successfully repelled.

It reported that a civilian was killed in the crossfire and that a soldier was also killed during clashes in Zaragoza, south of San Salvador, where two guerrillas were wounded.

...

Salvadoran (red) cross sources today reported that a 48-year-old woman identified as Maria Luz Lopez was wounded last night when a powerful bomb, which damaged several businesses in (San) Salvador, exploded.

The bomb was planted in a heavily commercial area of downtown (San) Salvador causing heavy property loses, according to the owners who provided no specific figures.

This is the fourth dynamic attack on businesses in (San) Salvador so far in 1990.

 



## Appendix B Hyper-Parameters

\begin{tabular}{l|c} \hline hparam name & value \\ \hline BERT model type & bert-base-uncased \\ train batch size & 1 \\ eval batch size & 1 \\ num train epochs & 18 \\ seed & 1 \\ number of GPU & 1 \\ learning rate & 5e-5 \\ ADAM epsilon & 1e-8 \\ warmup steps & 0 \\ downweigh factor & 0.01 \\ \hline \end{tabular}

## Appendix C Implementations

We build our model upon the HuggingFace's NER models' implementation (rb.gy/nryu2q).

Dependencies
* Python 3.6.10
* Pytorch 1.4.0
* Pytorch-Lightning 0.7.1
* Transformers: transformers 2.4.1 installed from source.

Link to CorpusWe obtain the raw corpus from [https://github.com/brendano/muc4_proc](https://github.com/brendano/muc4_proc)