Probing Representations for Document-level Event Extraction
Authors: Barry Wang
Xinya Du
Claire Cardie
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction
Interpretability
Embedding
Probing
Document-Level Information Extraction
TL;DR: Probing event information and more from the encodings of documents used in IE.

Abstract: The probing classiﬁers framework has been employed for interpreting deep neural network models for a variety of natural language processing (NLP) applications. Studies, however, have largely focused on sentencelevel NLP tasks. This work is the ﬁrst to apply the probing paradigm to representations learned for document-level information extraction (IE). We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We apply them to the representations acquired by learning models from three different LLM-based document-level IE approaches on a standard dataset. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks, albeit trade-offs in information helpful for coherence and event-type prediction. We further found that encoder models struggle with document length and cross-sentence discourse.

Probing Representations for Document-level Event Extraction
Barry Wang's and Xinya Du and Claire Cardie'
'Department of Computer Science, Cornell University Department of Computer Science, University of Texas Dallas zw545@corne11 edu, xinya. du@ut daleeuddlllll edu, cardie@cs. cornell edu edu
Abstract
REE is considered an essential and popular task, encompassing various variations. One particu- larly general approach is template filling', which can subsume certain other IE tasks by format- ting. In this regard, our focus lies on template- extraction methods with an end-to-end training scheme, where texts serve as the sole input.
The probing classifiers framework has been employed for interpreting deep neural net- work models for variety natural lan- guage processing (NLP) applications. Studies, however, have largely focused on sentence- level NLP tasks. This work is the first to apply the probing paradigm to represen- tations learned for document-level informa- tion extraction (IE). We designed eight em- bedding probes to analyze surface, semantic, and wrent-undesstanding capabilities relevant document-level event extraction. We apply them to the representations acquired by learn- ing models from three different I.I.Mbasedd a standard
Multi-task NLP models often support and are evaluated on the task. As result, we have seen frameworks of diverse underlying assump- tions and architectures for the task. Neverthe- less, high-performing modern models all leverage and fine-tune on pre-trained neural contextual em- bedding models, like variations of BERT (Devlin et al., 2019), due to the generalized performance lcap introduced by transformers and pretraining.
document-lever level IE approaches dataset. We found that trained encoders from these models yield embeddings that can mod- estly improve argument detections and label- ing but only slightly enhance event-level tasks, albeit trade-off's in information helpful for co- herence and event-type prediction. We further found that encoder models struggle with doe- ument length and cross-sentence discourse.
It is crucial to understand these representa- tions of the IE frameworks, as doing so reveals model strengths and weaknesses. However, un- like lookup style embeddings such as GloVe (Pen- nington et al., 2014), these neural contextualized representations are inherently difficult to inter- pret, leading to ongoing research efforts focused on analyzing their encoded information (Tenney et al., 2019h; Zhou and Srikumar, 2021; Belinkov, 2022). This work is inspired by various sentence- level embedding interpretability works, including Conneau et al. (2018) and Alt et al. (2020).
1 Introduction
Relation and event extraction (REE) focuses on identifying clusters of entities participating in a shared relation or event from unstructured text, that frequently contains fluctuating number of such instances. While the field of information extraction (IE) started out building training and evaluation REE datasets primarily concerned with documents, researchers have been overwhelm- ingly focusing on sentence-level datasets (Li al., 2013; Du and Cardie, 2020). Nevertheless, many IE tasks require a more comprehensive under- standing that often extends to the entire input doc- ument, leading to challenges such length and multiple events when embedding full documents. Consequently, document-level datasets continue pose challenges for even the most advanced models today (Das et al., 2022).
Yet, to the best our knowledge, no prior work has been done to understand the embedding of fea- tures that exist only at the document-leve scale. Hence, our work aims to fill this gap by investi- gating the factors contributing to the model per- formance. Specifically, we analyze the impact of three key elements: contextualization of encoding, fine-tuning, and encoder and post-eneoding archi- tectures. Our contributions can be summarized as follows:
'Defined in Appendix A. Template filling might not sub- sume certain relation extraction like n-ary relation extraction. Hence we will prefer "event extraction" in the following text.

---

Event type sempltte
Template #2
Event type
[...] FMLN detachments have conducted largest malitary operation in the entire histury of country's capital. An oftensive was launched I...] According to Reutes, atrempas
Arass bing//
Summing Kidnap
Perpetrator:
Perpetrator:
the Salvadoran conflict
Model- specific
Targel:
10 FM dettachmenns
Target:
made Alfiedo Cristian's official and personal residences; however, reported that the president was not hurt. [..]] The third round of these talks should have been held recently in Caracas, but opposition representatives refused to take part in them after left-seing trade unlon's headquarters was subjected artillry bombardment resulting in the deaths of at least 10 people. According to the insurgents, [...]
President
Fine-
Extraction
Castiani official and
10 Ercsideen Alloedo
anions heacquasters
"eleftwving tracle
tuned
Encoder
Expected Output Templates
Surface Info? Number of words :00
Semantic Info?
Probing Classifier
Event Info?
Ideal document embedding template extraction
Number of internsted events In: X
Input
Probing Performances
Figure 1: Overview of an ideal event extraction example and probing, Ideally, after contextualization by a fine- tuned encoder on the IE task, the per-token embedding can capture richer semantic and related event information, thereby facilitating easier model-specific extraction process. Our probing tasks test how different frameworks and conditions (c.g. IE training, coherence information access) affect information captured by the embeddings.
We identified the necessary document-level IE understanding capabilities and created suite of probing tasks² corresponding to each. We present fine- -ne-ggai analysis of how these capabilities relate to encoder layers, full-text contextualization, and fine-tuning.
Category
Illustration
Task
Task Full Name
aWords
WordCt
Word Count
Surface
#Bent-
SentCt
Sentence Count
a.k.a.
Coref
Are Coreferent
We compare IE frameworks of different input and training schemes and discuss how archi- tectural choices affect model performances.
Semantic
in
IsArg
Is an Argument
ArgTyp
Argument Type
Outio
Boncingg Atack...
EnvtTyps
Event Type
2 Probing and Probing Tasks
Event
in
CoEnvt
Co-Event
The ideal learned embedding for spans should include features and patterns (or generally, "in- formation") that independently show similarity to other span embeddings of the same entity men- tions, in the same event, etc., and we set out to test if that happens for trained encoders.
EnvtCt
Event Count
Figure 2: Probing Task Illustrations. Each refers to a span embedding (which is an embedding of a token in our experiment), and non con-gray means embeddings are known to be a role filler. See Section 2 for full descriptions.
Probing uses simplified tasks and classifiers to understand what information is encoded in the embedding of the input texts. We train a given document-level IE model (which finetunes its en- coder in training), and at test time capture the out- put of its encoder (the document representations) before they are further used in the model-specific extraction process. We then train and run our prob- ing tasks, each assessing an encoding capability of the encoder.
(muc, 1991) as our base dataset (details in Section 3.3) to develop evaluation probing tasks.
We present our probing tasks in Figure 2, This section outlines the probing tasks used for assess- ing the effectiveness of the learned document rep- resentations.
When designing these probing tasks, our goal is to ensure that each task accurately measures a spe- cific and narrow capability, which was often a sub- task in traditional pipelined models. Additionally, want to ensure faimess and generalizability in these tasks. Therefore, we avoid using event trig- gers in our probing tasks, especially considering that not all models use them during training.
Drawing inspiration from many sentence-leve probing works, we adopt some established setups and tasks, but with an emphasis on probing tasks pertaining to document and event understanding. We use the MUC document-level IE dataset
"Our model and probing codca are publicly available http://gihhh.coo///iiaaaaa//////
We divide our probing tasks into three catc-

---

gories: surface information, generic semantic un- derstanding, and event understanding.
classical event typing subtask, which often uses triggers as inputs. By performing this task, we can assess whether fine-tuning makes event-type information explicit.
Surface information These tasks assess if text embeddings encode the basic surface characteris- tics of the document they represent. Similar to the sentence length task proposed by Adi et al. (2017), employed word count (WordCt) task and sentence count (SentCt) task, each predicts the number of words and sentences in the text re- spectively, Labels are grouped into 10 count-based buckets and ensured uniform distribution.
Although syntactic information is commonly used in probing tasks, document-level datasets have limited syntactic annotations due to the chal- lenges of accurately annotating details like tree- depth data at scale. While the absence of these tasks is not ideal, we believe it would not signifi- cantly impact our overall analysis.
Semantic information These tasks go beyond surface and syntax, capturing the conveyed mean- ing between sentences for higher-level under- standing. Coreference (Coref) is the binary- classification task to determine the embeddings two spans of tokens ("mentions") refer to the same entity. Due to the annotation of MUC, all used embeddings are all known role-fillers, which are strictly necessary for the downstream decument-level IE to avoid duplicates. To han- dle varying mention span lengths in MUC, we uti- lize the first token's embedding for effective prob- ing classifier training, avoiding insufficient probe training at later positions. A similar setup applies to role-filler detection (IsArg), which predicts a span embedding is an argument of any template. This task parallels Argument Detection in classical models. Furthermore, the role classification task (ArgTyp) involves predicting the argument type of the role-filler span embedding. This task corre- sponds the argument extraction (argumentation) step classical pipelines.
Experiment Setup
3.1 IE Frameworks
We train the following document-level TE frame- works for 5, 10, 15, 20 epochs on MUC, and we observe the lowest validation loss or highest event FI score at epoch 20 for all these models.
DyGIE++ (Wadden et al., 2019) is frame- work capable of named entity recognition, rela- tion extraction, and event extraction tasks. It achieves all tasks by enumerating and scoring sec- tions (spans) of encoded text and using the rela- tions of different spans to detect triggers and con- struct event outputs.
GTT (Du et al., 2021) is sequence-to- sequence event-extraction model that perform the task end-to-end, without the need of labeled trig- gers. trained to decode a serialized template, with tuned decoder constraints.
TANL (Paolini et al., 2021) is multi-task model that fine-tunes T5 model (Raffel et al., 2020) translate text input augmented natural languages, with the in-text aug- mented parts extracted to be triggers and roles. It uses a two stage approach for event extraction, by first decoding (translating) the input text to extract trigger detection, then decoding related arguments for each trigger predicted.
Event understanding The highest level of document-level understanding is event under- standing. To test the model's capability in detecting events, we used an event count task (Evn(Ct) where the probing classifier is given the full-text embedding and asked predict the number of events that occurred in the text. We split all count labels into three buckets for class balancing. To understand how word embeddings are helpful to event deduplication or in argument linking, our Co-event task (CoEvnt) takes two argument span embeddings and predicts whether they are arguments the same event dif- ferent ones. Additionally, the event type task (EvntTyp..) involves predicting the type of the event template based on the embeddings of n role filler first tokens. This task is similar to the
3.2 Probing Model
We use similar setup to SentEval (Conneau et al., 2018) with extra layer. While sentence-level probing can use all dimensions of embeddings input, we added an attention weighted layer right after the input layer, as to simulate a response to a trained query and to reduce dimensions. The 768- -i8ddimnnsoon layer-output is then trained using

---

Model (IE-FI)
Input
WordCt
SentCt
IsArg
ArgTyp
Coref
Evntlypa
CoEvnt
EvntCt
Avg
DyGIE++
FullText
58.6
47.0
87.1
83.8
64.7
60.5
73.6
67.2
67.8
(41.9)
SentCat
57.4
58.9
87.5
85.6
69.2
56.7
67.9
67.0
68.8
GTT
FullText
58.6
46.3
88.3
88.5
66.7
60.4
66.4
68.3
67.9
(49.0)
SentCat
55.8
58.9
88.6
88.0
69.5
57.5
65.07
67.5
68.8
TANL
Full
54.2
43.3
88.2
86.8
66.6
57.8
60.0
65.8
65.33
(33.2)
SentCat
34.3
40.8
88.2
87.0
65.6
53.5
59.8
67.0
62.0
BERT base
FullTexx
65.5
87.8
86.1
75.7
60.4
74.0
63.5
69.7
Probing Task Test Average Accuracy. IE frameworks trained for 20 epochs on MUC, and we run probing tasks on the imput representations. We compare the 5-trial averaged test accuracy on full-text embeddings and concatenation of sentence embeddings from the same encoder to the untrained BERT baseline. IE-FI refers to the model's score on MUC test. Underlined data are the best in same embedding method, while bold, overall. We further report data over more epochs in Table and results on WikiEvents in Table 8 in Appendix
Table 1:
the same structure as SentEval. Specific training detail can be found in Appendix D.
(ArgTypt), albeit less consistently.
Model
Full Best
FullText Avg
Sent Best
Sent Avg
3.3 Dataset
We use MUC-3 and MUC-4 as our document- level data source create probing tasks, thanks to its rich coreference information. The dataset has 1300/200/200 saininggvaiddatiionnestngg doc- uments. any dataset with a similar format can be used to create probing tasks as well, and we ad- ditionally report results on the smaller WikiEvent (Li et al., 2021) Dataset in table in Appendix E. More MUC descriptions available in Appendix B.
WordCount:
209
DyGlE++
68.5
67.1
69.7
68.8
GTT
70.3
68.7
72.1
68.0
TANL
71.8
70.2
66.3
64.2
WordCount: 210-420
DyGIE++
67.0
65.7
67.6
64.7
GTT
67.6
67.0
66.4
64.7
TANL
64.8
62.0
63.6
60.8
WordCount: 2 431
DyGIE++ 70.6
70.2
74.2
72.1
GTT 69.1
68.7
71.5
70.2
4 Result and Analysis
TANL. 67.3
65.2
69.7
68.3
We present our data in Table 1, with results in more epochs available Table in Appendix E.
Table 2: EvntCt Probing Test Accuracy (%) ran- dom seed averaged. When WordCount 2431, both FullText and SentCat embeddings are truncated to the same length (e.g.. BERT-base has limit of 512) for comparison fairness. Concatenated sentence embed- dings show an advantage on medium or long texts.
Document-level IE Training and Embeddings Figure shows that embedded semantic and event information fluctuate during IE training, but steadily differ from the untrained BERT-base baseline. For the document representation, trained encoders significantly enhance embeddings for event detection as suggested by the higher accu- racy in event count predictions (EvntCtt). At the span level, embeddings lose information crucial for event type prediction and coreference, ev- idenced by decreased event typing performance (EvntTyp2L) and coreference accuracy (Coref)) over IE training epochs. Note again that coref- erence data pairs used are role-fillers and hence crucial for avoiding duplicated role-extractions, and future frameworks could seek to lower this knowledge loss. Nevertheless, IE training does aid argument detection (IsArg") and role labeling
Probing Performance of Different Models Ta- ble highlights the strengths and weaknesses of encoders trained using different IE frameworks. In addition to above observations. we see that DyGlE++ and GTT document embeddings cap- ture event information (EvntCt)) only marginally better than the baseline, whereas the TANL- finetuned encoder often has subpar performance across tasks. This discrepancy may be attributed to TANL's usage of T5 instead of BERT, which might be more suitable for the task, and that TANL em- ploys the encoder only once but the decoder mul- tiple times, resulting in less direct weight updates for the encoder and consequently lower its perfor-

---

EvniCt, Evntlyp2. CoEvnt Performance Epochs
Performance
0.75
0.50
0.70
0.85
SENTibusd
Reformere
0.05
TANL
GTT
DydiE
Bert-cowe
3.75
0.60
Taoks Pumitt
0.55
lsAg
Arglys
0.05
InGome
10
15
20
spochs
Epechs
Figure 3: Probing accuracy event (left) and semantic (right) information over document-level IE training epoch. 5 random seed results averaged (with standard deviation error bars). Color-coded by probing tasks. Trained encoder gain and lose information their generated embeddings as they trained for the tasks.
EnvcTyp2, CoErwt, EnvtCt Purformanci over Layers
discourse information, and a scheme that can do so remains an open problem. However, contex- tualized embedding with access to the full text does encode more event information in its output representation for spans (CoEvntt)..
Erxtlyo2
0.725
9EAT base EIwTyy22
0.700
GTT CeEnet
9EI laase CoEnel
2.875
QTT
EsweC1
3,600
Encoding layers Lastly, we experiment lo- cate the encoding IE information in different layers of the encoders, a common topic in pre- vious works (Tenney et al., 2019a). Using GTT with the same hyperparameter in its publication, its finetuned encoder shows semantic information encoding mostly (0-indexed) up layer 7 (IsArgt, ArgTyp1), meanwhile, event detection capabil- ity increases throughout the encoder (CoEvnt), EvntCr)). Surface information (Figure in Ap- pendix E) generally remains the same.
3.575
0.550
Layers
IsArg, Arg by, isCoref Performance over Layers
GTT GiTieetrg
0.70
36It-bauee
9TT Arplep
AFRTihase AngTyp
GTT IsCanel
SEAT base isOvee
0.00
Layars
Conclusion
Figure 4: Prohing accuracy event (upper) and semantic (lower) information over encoder layers from GTT trained over 18 epoch BERT-base.
Our work pioneers the application of probing the representation used at the document level, specifically in event extraction. We observed semantic and event-related information embed- ded representations varied throughout IE train- ing. While encoding improves on capabilities like event detection and argument labeling, training of- ten compromises embedded coreference and event typing information. Comparisons of IE frame- works uncovered that current models marginally outperformed the baseline in capturing event in- formation at best. Our analysis also suggested a potential shortcoming of encoders like BERT in utilizing cross-sentence discourse information effectively. In summary, our work provides the first insights into document-level representations, suggesting research directions for optimizing these representations for event extraction tasks.
mance in probing tasks (and the document-level IE task itself). Surface information encoding (Figure 6 in Appendix E) differ significantly by models.
Sentence and Full Text Embedding As demonstrated in Table 1, embedding sentences individually and then concatenating them can be more effective for IE tasks than using embeddings directly from fine-tuned encoder designed for entire documents. Notably, contextually encoding the full text often results in diminished perfor- mance in argument detection (IsArgJ), labeling (ArgTyp(), and particularly in Event detection (EvntCt)) for shorter texts, as highlighted in Table 2. These results suggest that encoders like BERT might not effectively utilize cross-sentence

---

Acknowledgements
References
We would like to express our gratitude to the fol- lowing undergraduate contributors who played vi- tal roles in this research:
1991. Third Message Uunderstanding Conference (MUC-3): Proceedings of Conference Held in San Diego, California, May 21-23, 1991.
Maitreyi Chatterjee, for her diligent efforts in exploring the MUC dataset, experimenting with contextual word embeddings, and making valu- able contributions to the appendix.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained anal- ysis of sentence embeddings using auxiliary predic- tion tasks.
Wayne Chen, whose contributions were in- dispensible in adapting TANL for the generic document-level IE task.
Christoph Alt, Aleksandra Gabryszak, and Leon- hard Hennig. 2020. Probing linguistic features of sentence-level representations in neural relation ex- traction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
Limitations
Online. Association for Compu- tational Linguistics.
Dataset While other document-level IE datasets are possible, none of them offer rich de- tails like MUC. For example, document-level n_ary_relations datasets like SciREX(Jain et al., 2020) can only cover three out of the six seman- tic and event knowledge probing tasks, and the dataset has issues with missing data.
Yonatan Belinkov. 2022. Prohing classifiers: Promises, shortcomings, and advances. Computational Lin- guistics, 48(1):207-219
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single S&!#" vector: Probing sentence embeddings for linguistic proper- ties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- I: Long Pupers), pages 2126-2136. Melhourne, Australia. Association for Computational Linguis- tics.
Additionally, we focus on template-filinng- capable IE frameworks they show more gen- erality in applications (and is supported by more available models like GTT), barring classical rela- tion extraction task dataset like the DocRED et al., 2019).
Aliva Das, Xinya Du, Barry Wang, Kejian Shi, Jiayuan Gu, Thomas Porter, and Claire Cardie. 2022. Auto- matic error analysis for document-leve information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 3960-3975, Dublin, Ireland. Association for Computational Linguistics.
Scoping While we observe ways to improve deeument-level IE frameworks, creating new frameworks and testing them are beyond the scope of this probing work.
Embedding length and tokenizer All models we investigated use an encoder that has an input cap of 512 tokens, leaving many entities inacces- sible. In addition, some models use tokenizers that tokenize words into fewer tokens and result, may access more content in full-text embedding probing tasks. Note that also because of tokenizer difference, despite our effort to make sure all prob- ing tasks are fair, some models might not see up to 2.1% training data while others do.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computationall Linguistics: Human Language Technologies. Volume I (Long and Shori Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 67 671.683, Online. Association for Computational Linguistics.
Xinya Du, Alexander Rush, and Claire Cardie. 2021. Template filling with generative transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 909-914, Online. Association for Computa- tional Linguistics.

---

Sarthak Jain, Madeleine van Zuylen. Hannanch Ha- jishirzi, and lz Beltagy. 2020. SciREX: A chal- lenge dataset for document-level information extrac- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7506-7516, Online. Association for Compu- tational Linguistics.
Yuan Yao, Deming Ye. Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang. Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777, Florence, Italy. Association for Computational Lin- guistics.
Qi Li, Heng Ji. and Liang Huang. 2013. Joint event extraction via structured prediction with global fea- tures. In Proceedings of the s1 Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Papers), pages 73-82, Sofia, Bulgaria. Association for Computational Linguistics.
Yichu Zhou and Vivek Srikumar. 2021. DirectProbe: Studying representations without classifiers. Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistiex: Human Language Technologies, pages 5070-5083, Online. Association for Computational Linguisties.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-
level event argument extraction by conditional gen- eration. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 894-908, Online. Association for Computational Linguistics.
A Definition of Template Filling
Assume a predefined set of event types, Ti...., Tma where m represents the total number of template types. Every event template comprises set of k roles, depicted as ri. P/- For document made up of n words, represented by æ1, 22, ... Fna the template filling task extract zero or more tem- plates. The number of templates are not given as an input, and each template can represent a n-ary relation or an event.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Each extracted template contains k + 1 slots: the first slot is dedicated to the event type, which is one of the event types from Ti.... ..... Tm. The sub- sequent k slots represent event role, which will be one of the roles rl, ... rg. The system's job is to assign zero or more entities (role-fillers) the corresponding role in each slot.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
B MUC dataset
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computa- tional Linguistics.
The MUC dataset (1991) comprises news arti- cles and documents manually annotated for coref- erence resolution and for resolving ambiguous ref- erences in the text. The MUC 4 dataset( 1992), on the other hand, expanded the scope to include named entity recognition and template-basee in- formation extraction.
Jan Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim. Benjamin Van Durme, Samuel R. Bowman, Dipan- jan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations.
We used a portion of the MUC and 4 datasets for template filling and labeled the dataset with triggers based on event types for our probing tasks. The triggers were added to make the dataset compatible with TANL so that we could perform multi-template prediction.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLL pages 5784- 5789, Hong Kong, China. Association for Computa- tional Linguistics.
The event schema includes 5 incident types namely 'kidnapping', "attack', 'bombing', 'rob- bery', 'forced work stoppage', and "arson'. The coreference information for each event includes fields like 'PerpInd', 'PerpOrg', 'Target', 'Vic- tim', and 'Weapon'.

---

C IE Framework Parameters
See Table 3, 5,5
Parameter
Value
Parameter
Value
nhid
400, (100, 200, 800)
num_epochs
[5, 10, 15, 20]
tenacity
10
patience
8
max__ppan_ward
span_width width
batch_site size
8
MaxEpoch
1000
optimizer
Ir: 5c-4
bert_mooe
bert-base-uncasee
optim
adam
dropout
0, (0.1)
target_
events
attention-head
1, (11, 22)
Table 3: DyGIE++ Model Parameters
Table 6: Probing model parameters values in the parenthesis are tested by not used, often due to lower performances.
Parameter
Value
max__ee__engaee www_sslggghhhheeeeeeeee src
435
-max_seq_lenge le__llegth__gg
75
-num_ quum_train_ee train_eppchss
[5. 10, 15, 18, 20] bert-base-uneased
-bert_model
-thresh
80
batch_size
WordCr SentCt Performance
6.70
Table 4: GTT Model Parameters
0.50
0:35
Parameter
Value
5.50
multitask
True t5-base
@ocel__ame_rr___
REAT Semt
num num_train se__trin__eeeee
[5, 10, 15, 20]
tokenizer_nann name
15-base
max_seq_length
512
Figure 5: Probing accuracy on semantic surface in- formation over encoder layers from GTT trained 18 epochs and BERT base.
max_seq_length_oee
512
per_device_traid
didevice_train_bbatth_iiii batch_sirze
4
per_device_eval sasl__bach_stte size
1
num_ beams
1
Table 5: TANL Model Training Parameters
D Probing Model Details
Epacts
See Table 6.
0.00
E
Additional Results
TAL
See Figure and Figure 6 for more probing results on MUC.
See Table 8 for more results on WikiEvents. WikilEvents a smaller (246-example) dataset.
6:5
Figure 6: Probing accuracy surface information over document-level IE training epoch.

---

Model
Epoch
Embedding
WordCi
SenCC
IsArg
ArgTyp
Coref
CoEvnt
EvarTypa
EvniCI
GTT
5
FullText
59,50=4.a
45. 45.10
89.3741.00
87. 87.588
59.54 71.05
67.564284
68.50=2.21
SentCat
53 =3,9
59. 39.60g5.38
89 42
88. 1 40.3
70.53
55. 15.16
66.6942.0
29.60=a.25
10
FullText
59, 39.30-2.57
47.1 1..0...00
88 18.0
87. 37.56.0.4
67.06+3.6
61.41=201
65. 144..0
68.40 68.40=2.19
SentCat
55 2.2
60. 1.54.
88. 11.00
88.31 +
+1.51
56, 36.74
65, 65.20
70.20=a.35
15
FullTcxt
59.60-1.80
44.90
88. 88.4440.42 87.99
67.75
61.32
66.46g1.59
68.40g2.01
SemtCat
56.60
58.70
88.3310.8
88. 28 70
69.05 13.05
57.17
64.65
66.60
20
FullText
58.60
26.30
88.31
89.51
66.68
60.43
66,40,3.25
68.3021.42
SemtCat
_5.80_2.77
18.90
88.5610.34
87.96109 96
69.45
57.48
12.07
67,50
TANL
5
FullText 55,70_..25
44.50
89.65 10.32
87.39
65.49
57,65
18.81
67,30_2.wz
SentCat
34.90_144
40.101268
89,47
87.20 1
64.86
55.01
56.85
65,80_2.80
10
FullText
54.20_1.52
41.30
88.8910.22
86 20.90
63.631317
56.88
62.06.1.45
66.50_1.77
SentCat
33.40_1.92
41.90 41.90
88.88
87.12
64.25
54.33
59.94
65,40_2.86
15
FullText
52.10=1.47
23.30
28.79
87.08
67.32.1.97
56.
20
60.32.4.15
64.80=1.04
SentCar
35. 35.00-2.76
40, 40.40
88 88.93+1.36
87. 67.20
64.36+1156
$4.30 $4.30
59.84=2.69
64.20=2.75
20
FullText
54.20__1.48
1..30
88.15 15
86 40.00
16.
57.77
60.03
65.80.2.15
SeatCat
34.30_1.0s
88.17 17.68
86 66.95
65.57+2264
53.50
59.84=1.94
67.00=1.60
DyGlE++
FullText
38.10=243
$1.80
89 89.06+n.50
87.43
64.26+5598
57,90
73.43
68.70=1.06
SentCar
57.88 10.9
59 404-2.07
88 #0.92
86.57
70.56
53,93
70.00
70.50=2.15
10
FullText
55.80=0.02
47. 47.10+2..41
88.18 18.18
84.87+0.00
63.64-+6.50
+3.51
72.15
67.80=1.05
SeatCar
58.80-2.02
38.20
87. 80
85 10.00000
29.04++2.00
54,60 24.69=2.84
70.00
70.30
FullText
60.20
48.70
87.93
84.07
66.27 +3.65
60.68 60.68.2230
72.55
68.30
SeatCar
55, 60=0.82
57.50+n.30
86.22+aan 87.74
70.49+1.45
55,00 =5.08
67.57
69.30=1.35
FullText
58 60
00.000
87.13
83 $3.83=121
64.66 9416
60.50
73.58
67.20=1.64
ScatCat
17.40-2.10 10
58. 58,90
87.53
85.63
69.20+2.09
56.69
67.88
67.00=1.91
BERT
FullText 65,50
45.00
87.76
86.05
75.72
60.37
73.99
63,50
Table 7: Probing test accuracy on more epoch. Note that underlined data, unlike those presented in Table 1, are the best performance in the model family, while bold data the best performer for the task for both embeddings. 5 random seed results averaged, with data after + indicating standard deviations.
Model
Epoch
Embedding
WonCCt
SenCC
IsAng
Argllyp
Coref
CoEvat
Evatlyp2
EvenCt
Average
TANL
PullText
26.0022415
13.00ga71
85.0 85.0920.08
28. 28.662
78.15
65 65.50_117
31.89
25.00-10.30
44.16
10
PulfText
12.00.2.4 19
85.47
29.472001
77.1
63.50_230
32.70
25.00_7.:
44.57
15
FullTest
13.00 m0
28.95 :1.5
77.28
66.06-7m
26.49
16.00
42.17
20
FullTeai 27.00
15.00 18
83.92
28.95
77.62
65.50_..29
31.89
22.00.cc47
DyGlE++
FullText 18.00
g4.00
81.12
50.27
73.40
60.35_5a
32.1
18.00_aas
40.90
10
FullText 21
19.00
80.33
30.91
73.93
61.40_95m
34.21
21.00_5.4s
42.72
15
PulfTest
22.00
22.00
81 81.07
29.97
74.57
61.40_296
37.6842.88
23.00=s.a7
43.96
20
FullText
72
24.00
80.19
30.53
74.9042.52
61.05=a.7
40.00
18.00_67:
BERT
FullText
25.00
20.00
31.05
71.07
63.30
27.40
20.00
42.33
Table 8: Average Accuracy WikiEvents Probing Task. IE frameworks were trained over varying epochs on WikiEvents and evaluated via probing tasks on their input representations. The results, averaged over trials, compare full-text embeddings against an untrained BERT baseline. Underlined figures represent the best within their model group. while bold denotes the best overall. Standard deviations are shown after +. The performance GTT on WikiEventss which requires modifications for varying roles based on event type, a topic for potential future exploration by the research community.


Reviews:
[{"review_id": "CK0Mpm0bAm", "reviewer": "Reviewer_mJYE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply thankful for Reviewer mJYE’s helpful review. \n\n**Weakness1: Novelty is not obvious as a probing work**\n\nWe acknowledge that probing embedding is not new, and designing new tasks for a new problem space seems rather standard. Nevertheless, this work \n\n- focuses on understanding document-level features encoded in neural representations, which is an important topic but has never been studied in NLP.\n  \n- showcases a new suite of probing tasks designed to mimic traditional pipeline approaches, with argument-level and event-level ones essential for the **document-level IE** task.\n  \n- shows quantitatively and in detail how IE training improves or deteriorates the trained document encoding’s capability for the first time.\n  \n- involves significant behind-the-scene engineering that could be considered novel\n  \n\n  -    We annotated the MUC dataset with event triggers for compatibility, which can be utilized for document-level IE tasks later on as well [1, 2]. \n  \n  -   Two out of the three models are carefully enhanced for the first time to fully support generic template filling from their original (single-event-per-input, sentence-level) framework.\n  \n  -   All these will be included in our codebase to be available online.\n  \n\n**Weakness2: Interpretation seems tough**\n\nWe acknowledge that the unaveraged result in our work seemed less straightforward. While previous probing works never explicitly mention using multiple trial results [3], we will include the averaged results of embeddings from 5 random states in our final paper (and are happy to show any part of it during our discussion). We observed very similar but smoother trends and low standard deviations across random states. We will summarize these findings in the more clear textual form in the final version and improve our graphing.\n\n**References:**\n\n[1] Document-Level Event Argument Extraction by Conditional Generation (Li et al., NAACL 2021)\n\n[2] GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction (Du et al., EACL 2021)\n\n[3] SentEval: An Evaluation Toolkit for Universal Sentence Representations (Conneau & Kiela, LREC 2018)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a thorough analysis about the quality of the embeddings learnt. It is one of the only works providing explainability to IE models at a document-level. The paper is well written and will also be supported with code and data, so it is a reproducible work.", "reasons_to_reject": "While the main contribution of the paper is to explore probing in the domain of document-level IE, there is no novelty in the work as such. The results need to be presented more lucidly and interpretation is tough in some cases.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses using probing techniques to analyse the quality of the representations learnt for multiple document-level IE tasks. They claim that they are the first to explore probing in the domain of document-level IE. The main contributions of this paper include providing explainability to the performance of different document level IE models along with fine-grained analysis of the embeddings learnt for the different tasks in that domain.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WPqd2PRajp", "reviewer": "Reviewer_QFh8", "summary": "", "questions": "a) Line 202: suggest adding average sentence and document length, and their distributions to make the paper more self-explanatory. Because apparently document length can be a key factor in this study, I also suggest bucketizing  the test set by lengths to conduct the analysis. Expect to see in what length full-text contextualization cannot compete with the sentence embedding concatenation.\n\nb)Line 218: this suggests that higher layers embed more abstract event knowledge, which is clean and nice. However, the paper does not discuss the difference between GTT and BERT and what results in the difference are not clearly discussed. Can the authors elaborate about your aspect about this and  include the discussion in the paper/Appendix?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate Reviewer QFh8’s helpful review, insightful questions, and grammar advice. \n\n**Weakness (a) Noisy results without clean trend**\n\nWe acknowledge that the unaveraged result in our work seemed noisy. We have the averaged results ready, which looks only more flattened (for Fig 3) (happy to present any other part of the data during discussion). Nevertheless, the gap between the performance of BERT-base (untrained) embeddings and fine-tuned encoder’s embedding on different tasks is and stays consistent. In fact, IE performance improvements between epoch 5 and epoch20 are often only modest, which means the gap between epoch0 (BERT-base) and epoch5 in Fig 3 is the most salient and crucial to understand, and we draw most of our conclusions from this area. We will summarize these findings in the more clear textual form in the final version.\n\n**Weakness (b) Limited Dataset (MUC)**\n\nWe acknowledge our observations come from limited data from a single dataset. Regrettably, document-level event datasets are scarce, and the high-quality MUC is often the one dataset people resort to or even evaluate exclusively on (See doc-level work like [1-2]). We will report limited data from WikiEvents in the appendix, but given MUC data’s higher quality (and quantity) we expect MUC results would be the most reliable.\n\n**Question (a) Text Length and Bucketizing Analysis**\n\nThank you for your great idea. We will add a more detailed analysis to our text and appendix regarding this. As some quick facts, the average (±standard deviation) word count (entire dataset) is 362.83±220, the 25 and 75 percentile is 187 and 493, and the sentence count average is 14.15±10.\n\n**Question (b) Difference between GTT and BERT**\n\nSorry for the possible confusion. GTT here uses BERT as its base model for encoding, and we are comparing encodings from a GTT-Finetuned BERT that was trained during the IE task training, and from the out-of-shelf BERT. We will make clear this and the interpretation of the camera-ready version.\n\n**Improvement Idea (a) Cite Fig4 on line 218 (b) Add average score per row in Table1**\n\nGreat catch! We will add them. \n\n**Improvement Idea (c) Use more dataset**\n\nWe will report WikiEvent results in the appendix due to the limited pages, but we prefer presenting MUC in general due to its much bigger size and richer annotation and coreference.\n\n**References**\n\n[1] Iterative Document-level Information Extraction via Imitation Learning (Chen et al., EACL 2023)\n\n[2] Template Filling with Generative Transformers (Du et al., NAACL 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) Valuable evaluation result for understanding event knowledge in pre-trained LLM, which is informative to the research community.\n\nb) Concise but clean write-up which is easy to follow.\n", "reasons_to_reject": "a) The result is sort of noisy without a clean trend, though proper score aggregation could sort it out.\n\nb) The observations can be applied to a specific dataset (MUC), but we don’t know whether it is general. \n", "questions_to_authors": "a) Line 202: suggest adding average sentence and document length, and their distributions to make the paper more self-explanatory. Because apparently document length can be a key factor in this study, I also suggest bucketizing  the test set by lengths to conduct the analysis. Expect to see in what length full-text contextualization cannot compete with the sentence embedding concatenation.\n\nb)Line 218: this suggests that higher layers embed more abstract event knowledge, which is clean and nice. However, the paper does not discuss the difference between GTT and BERT and what results in the difference are not clearly discussed. Can the authors elaborate about your aspect about this and  include the discussion in the paper/Appendix?\n", "paper_topic_and_main_contributions": "This work aims to understand event knowledge encoded in the embeddings of LLM models by employing a probing framework based on document level Information Extraction (IE) templates. The framework probes information at 3 levels, including surface-level, argument-level, and event-level. The empirical result shows that fine-tuned embeddings benefit low-abstraction-level IE tasks (e.g., surface and argument) more than high-abstraction-level tasks (e.g., event).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "a) Line 218: Figure 4 should be mentioned in the paragraph.\n\nb) Table 1: suggest adding average results for FullText and SentCat on the three levels of IE.\n\nc) The observations can be applied to a specific dataset (MUC), but we don’t know whether it is general. Suggest strengthening this aspect for the paper.\n"}}, {"review_id": "0LJihNpPWB", "reviewer": "Reviewer_FqJd", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for Reviewer FqJd’s helpful review. Firstly, we would like to add that **NLP engineering experiment** and **Reproduction study**, which our work belongs to, is also an integral part of this year’s program as indicated in the call for paper [1].  \n\nWe focus on understanding document-level features [2, 3] encoded in neural representations, which is an important topic but has never been studied in NLP.\n\n**Weakness 1: Mixed Results. Difficult to draw conclusions.**\n\n- We acknowledge that our empirical results seem mixed. Nevertheless, we have seen a clear differentiation between untrained encoder and IE-tuned encoder capabilities (Fig3, notice the difference between epoch 0 and 5 stays consistent as epoch number increases further). We will report averaging scores using embeddings from 5 different random states for less noisy graphs in our final version (which shows similar trends), but our conclusions mostly focus on IE-finetune v.s. Untrained. Moreover, mixed results, with claims or trends that can often be substantiated only partially or in specific parts, are often expected in explainability works [4-5].\n\n**Weakness 2: Limited Inspirations for Future Work**\n\n- We have shown a few interesting phenomena, including deteriorating coreference performance, inefficient encoding of discourse information, limited event information encoding in span encoding, etc.\n  \n- Our work is the first few on probing document-level NLP tasks. Future probing work can [to-add]. Future modeling work can take insights from our work to improve on these particular metrics, and future explainability work can build on our probing framework and task design.\n  \n\n**References**\n\n[1] [Call for Main Conference Papers - EMNLP 2023](https://2023.emnlp.org/calls/main_conference_papers/#contributions)\n\n[2] Document-Level Event Argument Extraction by Conditional Generation (Li et al., NAACL 2021)\n\n[3] GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction (Du et al., EACL 2021)\n\n[4] BERT Rediscovers the Classical NLP Pipeline (Tenney et al., ACL 2019)\n\n[5] Automatic Error Analysis for Document-level Information Extraction (Das et al., ACL 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors introduce probing tasks to the document-level IE tasks, and design different probing tasks for explanations. \n- The probing tasks are intuitive to understand.\n- The authors provide some empirical results on the correlation between probing tasks and the target task.", "reasons_to_reject": "- The empirical results are quite mixed and hard to draw clear conclusions on the findings of explainability.\n- Both the method and empirical results provide limited inspirations for future work. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes using probing classifiers to understand embeddings of the encoder in the document-level information extraction tasks (IE). The authors design eight probing tasks in three categories including surface, semantic and event (task) levels. They compare four specific models in the MUC document-level IE dataset with probing classifiers, and analyze the correlations between the accuracies in probing tasks and the accuracies in the target (IE) task. The results show that the probing tasks' accuracies change during the IE task training compared to the untrained BERT-base model. They also experimented with constructing the encoder differently including different aggregation methods and different layers.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors conduct probing on pre-trained models for document-level event extraction, covering different neural layers and different elements of structured events. Several fine-grained conclusions are drawn. This is the first attempt to conduct probing on the document level task, and novel probes that require engineering work have been designed. On the negative side, there has been some criticism on the results and the insight for guiding further research.
Decision: Accept-Findings