Three Stream Based Multi-level Event Contrastive Learning for Text-Video Event Extraction
Authors: Jiaqi Li
Chuanyi Zhang
Miaozeng Du
Dehai Min
Yongrui Chen
Guilin Qi
Venue: EMNLP 2023 Main (2023)
Keywords: Event extraction; Multimodal

Abstract: Text-video based multimodal event extraction refers to identifying event information from the given text-video pairs. Existing methods predominantly utilize video appearance features (VAF) and text sequence features (TSF) as input information. Some of them employ contrastive learning to align VAF with the event types extracted from TSF. However, they disregard the motion representations in videos and the optimization of contrastive objective could be misguided by the background noise from RGB frames. We observe that the same event triggers correspond to similar motion trajectories, which are hardly affected by the background noise. Moviated by this, we propose a Three Stream Multimodal Event Extraction framework (TSEE) that simultaneously utilizes the features of text sequence and video appearance, as well as the motion representations to enhance the event extraction capacity. Firstly, we extract the optical flow features (OFF) as motion representations from videos to incorporate with VAF and TSF. Then we introduce a Multi-level Event Contrastive Learning module to align the embedding space between OFF and event triggers, as well as between event triggers and types. Finally, a Dual Querying Text module is proposed to enhance the interaction between modalities. Experimental results show that TSEE outperforms the state-of-the-art methods, which demonstrates its superiority.

Three Stream Based Multi-level Event Contrastive Learning for Video
Event Extraction
Jiaqi Li
Chuanyi Zhang?; Miaozeng Du Dehai Min Yongrui Chen Guilin Qi
School of Computer Science and Engineering, Southeast University, Nanjing, China
College of Artificial Intelligence and Automation, Hohai University, Nanjing, China
3 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China jql gqli@seu. edu. en 20231 1 104@hhu. edu. cn miaozengduêseu. edu. cn edu.cn cn
zhi shanqêseu. edu. cn yrchenêseu. cnu
Abstract
Trump his own elevanth rally airfield.
meeting came ahead of plannad with Trump.
Text-video based multimodal event extraction refers to identifying event information from the given text-video pairs. Existing methods pre- dominantly utilize video appearance features (VAF) and text sequence features (TSF) as in- put information. Some of them employ con- trastive learning align VAF with the event types extracted from TSF. However, they disre- gard the motion representations in videos and the optimization of contrastive objective could be misguided by the background noise from RGB frames. We observe that the same event triggers correspond to similar motion trajec- tories, which are hardly affected by the back- ground noise. motivated by this, propose a Three Stream Multimodal Event Extraction framework (TSEE) that simultaneously utilizes the features of text sequence and video appear- ance, as well as the motion representations to enhance the event extraction capacity. Firstly, we extract the optical flow features (OFF) as motion representations from vidcos to incorpo- rate with VAF and TSF. Then we introduce Multi-level Event Contrastive Learning mod- ule to align the embedding space between OFF and event triggers, as well as between event triggers and types. Finally, a Dual Querying Text module proposed to enhance the interac- tion between modalities. Experimental results show that TSEE outperforms the statc-of-the- art methods, which demonstrates its superiority.
Disimmlar
After Trump used the song his recent rally in Tulsa.
The visit maating with Korean president
Trigger[rally]
Trigger: [meeting]
Disamiia
Trigger[arreseed [arrested]
Similer
Four men have been arrested in Australia over a plot to bring airplane.
Several people were arrested, and many reporters were injured
Figure I: An illustration of the relationship between motion representations and event triggers. To capture the motion information, we extract optical flow features from each video. The optical flow features tend to be similar if the event triggers are the same. Otherwise. they are likely to become dissimilar.
by merging complementary information from mul- tiple modalities such as texts, images or videos. Specifically, texts provide abstract semantics while visual data supplies concrete instances(Liu et al.. 2023a,b; Yang et al., 2023a,b). Compared with text-image based MEE (TIMEE) (Li et al., 2022; Liu et al., 2022: Li al., 2020; Zhang et al., 2017; Tong et al., 2020), text-video based MEE (TVMEB) (Chen et al., 2021; Wang et al., 2023) contains more context and scene information. Moreover, TYMEE presents temporal data that could capture the dy- namic evolution of events, making it an area of significant interest.
1 Introduction
Event extraction (EE) is fundamental task which aims to recognize the event structure from texts (Nguyen al., 2016; Nguyen and Grishman, 2015; Wadden et al., 2019; et al., 2022). Recent years have witnessed the booming of the multimodal event extraction (MEE). MEE (Pratt et al., 2020; Sadhu et al., 2021; Li et al., 2017) extends EE
Existing methods in TVMEE (Chen et al., 2021: Wang al., 2023) extract text sequence features (TSF) and video appearance features (VAF) from texts and RGB frames by adopting pre-trained lan- guage and video models respectively. However,
J. and C. Zhang contributed equally to this work should considered co-first authors. Corresponding author.

---

they neglect the motion representations in videos. In TVMEE, motion representations may play an important role, as they furnish details on the mo- tion and behavior of objects in videos. Furthermore, we observe that identical event triggers correspond to analogous motion representations. To explore the relationship between motion representations and event triggers, we introduce the optical flow features (OFF) (Dosovitskiy al., 2015) ob- ject motion representations. OFF (Ilg et al., 2017;
triggers, as well as event types and triggers. We align each pair in the embedding space by introduc- ing multi-level contrastive objective. Lastly, we propose a Dual Querying Text (DQT) module to increase the interaction between modalities. In this module, VAF and OFF retrieve the cross-modality attention of each token in TSF respectively.
The contributions of our work could be summa- rized follows:
We propose a novel framework called TSEE that leverages the motion representations in videos. To the best of our knowledge, we are the first to introduce optical flow features into TVMEE.
Sun al., 2018; Jiang and Learned-Miller, 2023; Marsal et al., 2023; Liu et al., 2021) represents the movement of objects a sequence between consecutive frames and is extensively applied in video fields, such as video understanding (Teed and Deng, 2020; Luo et al., 2022), video super- resolution (Zheng al., 2022; Chan et al., 2022), etc. As shown in Figure gers 'rally', 'meeting' and 'arrested'. For each sample we visualize the text, frames and OFF ex- tracted from the corresponding frames. could be observed that OFF is similar if it refers to the same trigger. In contrast, heterogeneous triggers usually point to dissimilar OFF.
Our proposed modules, MECL and DQT, sig- nificantly improve the model performance. MECL aligns the embedding space of OFF, event triggers and types. DQT enhances the interaction among text, video and optical flow modalities.
compare three trig-
The experimental results on two benchmark datasets demonstrate the superiority of our frame- work over the state-of-the-ard methods.
Related Work
2.1 Event Extraction
A previous work (Wang et al., 2023) applics con- trastive learning to reduce the distance between VAF and event types. Although VAF extracted from continuous frames may provide useful mo- tion information, it also contains misguiding back- ground noise. To specific, the background noise is various scenes heterogeneous videos. does not provide any event semantics and can suppress the alignment between visual cues and event types. However, this issue could be properly alleviated by utilizing OFF because OFF solely exploits the object motion representations and filters out the scene information.
In the field of event extraction research, the initial work primarily focused on sentence-level studies in the text. Some works have explored the use of convolutional neural networks (Nguyen and Grish- man, 2015; Nguyen et al., 2016), recurrent neu- ral networks (Nguyen and Grishman, 2015; Liu et al., 2019, 2020), graph neural networks (Li al., 2017), and later emerging pre-trained language models (Wadden et al., 2019; Wang et al., 2022; Lu et al., 2022) for handling the extraction trig- gers and arguments. In the field of computer vi- sion, event extraction is operationalized as situation recognition (Pratt et al., 2020: Sadhu et al.. 2021; Yatskar al., 2016; Li et al., 2017), with tasks pri- marily involving the classification and extraction of frames containing entities and roles (arguments) from images with actions (visual events)(Zhang et al., 2021; Chen et al., 2022a,b). In recent years, there has been an emergence of using multimodal information for event extraction et al., 2022c, 2023). (Zhang et al., 2017) demonstrated the ef- fectiveness of using visually based entity data to extract events. Previous multimodal event extrac- tion models (Li al., 2020; Liu al., 2022) mostly dealt with visual data in the form of images, (Chen et al., 2021) pioneered a model that can jointly ex- tract events from text and video data. They used
In this work, design novel frame- work. Three Stream Multimodal Event Extraction (TSEE), which simultaneously leverages three modality features (text sequence, video appearance and motion representations) to improve the event extraction capability. To begin with, we employ pre-trained I3D (Carreira and Zisserman, 2017) 1) and PWC (Sun et al., 2018) models to extract VAF and OFF from each video respectively. For the input text, adopt a pre-trained language model (Devlin al., 2018; Raffel al., 2020) to obtain TSF. Then we propose a Multi-level Event Con- trastive Learning (MECL) module, aiming to align the feature representations between OFF and event

---

pre-trained text-video retrieval model to find the most relevant text-video pairs. Based on (Chen et al., 2021)'s approach, (Wang et al., 2023) intro- duced supervised contrastive learning to enhance the representation of the two modalities for further event extraction.
3
Approach
3.1 Task Definition
Given text-video pair (T, V), denote the quence of input text tokens as T {4, l2, ..., Lm). We sample from the video every 16 frames get the clip sequence V = (2, "i.,e........). Cx). In TVMEE, each sample is annotated set event Our goal is to jointly ex- tract event triggers and event arguments. An event trigger is word or phrase in a sentence that indi- cates the occurrence of an event. For example, in the sentence 'John bought new car yesterday', the word 'bought' is the event trigger, indicating the occurrence of buying event. Event argument extraction to identify relevant pieces of informa- tion or arguments from texts. The picces commonly involve an event, such as subject, object, verb, and other modifiers. Then these roles are mapped to their semantic roles such as agent, patient, location, time, and on. Take the above sentence as ample, 'John', 'car' and 'yesterday' are the event arguments referring to the buying event and the roles are 'buyer', 'product' and 'time" respectively.
2.2 Supervised Contrastive Learning
types E =
Contrast learning is technique that trains models to distinguish between similar and different exam- ples. Self-supervised representation learning meth- ods such as (Kim et al., 2020; Yue et al.. 2022; Kim et al., 2021; Kaku et al., 2021; Iter et al., 2020) divide each sample into positive and negative sam- ples, learning feature representations by compar- ing the similarity between sample pairs. Works such as (Gunel et al., 2020; Wu et al., 2022; Gunel et al., 2020; Song et al., 2022) optimize the su- pervised contrastive objective for supervised con- trastive learning. For event extraction tasks, (Wang et al., 2021) proposes contrastive pre-training framework that uses semantic structures. (Yao et al., 2022) introduces an efficient event extraction model with a contrastive objective to distinguish triggers and arguments. (Zolfaghari et al., 2021) presents more effective mrosl- contrastive learning loss function, compared to directly using loss functions designed for visual data.
3.2 Feature Extraction
Our framework utilizes information from both text and video features as shown in Figure 2. In par- ticular, the video incorporates features from two perspectives. The first is the video appearance fea- tures, which represents color, texture, shape, and other visual cues. Secondly, motion features pro- vide information about dynamics of objects within the scene. We employ corresponding pre-tree pre-trained models to extract these features respectively.
2.3 Optical Flow
Most of the existing methods for extracting op- tical flow rely on pixel-by-pixel prediction using neural networks. Optical flow extraction models have various model structures, including encoder- decoder architecture (Dosovitskiy et al., 2015), it- erative refinement of multiple FlowNet modules using cumulative superposition (Ilg et al.. 2017), feature processing and extraction through pyra- mid structure (Sun et al., 2018), and construction of cost volumes with different expansion coefficients (Jiang and Learned-Miller, 2023). (Marsal et al., 2023) trains optical flow estimation networks by training two networks that jointly estimate optical flow and brightness changes. (Liu et al., 2021) )) ad- dresses the optical flow problem of occluded pixels. (Chan al., 2022) utilizes temporal information and proposes a novel approach to combine optical flow and deformable alignment in videos. (Huang et al., 2019) employs optical flow to solve motion detection issues related to dynamic background and changing foreground appearance.
Text feature extraction. The input text tokens are encoded using pre-trained T5-base (Raffel et al., 2020) with d, hidden dimensions. Thus each input sequence is represented where ny is the length of sequence.
matrix Fy- E R
Video feature extraction. We input each clip se- quence into the 13D network pretrained on Kinetics dataset and the PWC network pretrained on Sintel dataset. Then we obtain sequence of VAF and OFF. To represent video-level feature, we sum up all the features within the sequence. VAF and OFF denoted as Fy E Id and Fo E Rdd
3.3 Multi-level Event Contrastive Learning
We observe that identical event triggers usually in- volve similar motion representations, which are not affected by background noise. Additionally,

---

Multi level Event
Queyyang Test
The comes ahead Kim's meeting with
Ungger Veeture
Learning
Teal Sequenee Fielur
Tonadomer
Lveett Feelare
Seqpancce
Event
PWC
Optiaal flow Festure
Torndorzer
Postbire Fair
Negative Pair
Model
Tyee
Vilao Apperinioo Feature
Triggee Redium Optici Elow Fedurr
Viduo A Apuasezns Fudure
Figure 2: The overview of TSEE framework. We leverage T5 model to extract Text Sequence Feature (TSF). For each video we adopt PWC Net and an I3D model obtain Optical Flow Feature (OFF) and Video Appearance Feature (VAF) respectively. Event Trigger Feature and Event Type Feature are extracted from TSF. Then the two features along with OFF are entered into a Multi-level Event Contrastive Learning module to align the feature representations between them. After that, Dual Querying Text module enhances the interaction among TSF, VAF and OFF. Finally, the event decoder takes the aggregated feature as input to predict events.
the event extraction, event type is correlated to various triggers. Motivated by the above observa- tions, we propose a Multi-level Event Contrastive Learning (MECL) module. This module aligns the feature representations between OFF and triggers. The embedding spaces of event types and triggers are also aligned using this module. We apply su- pervised contrastive learning (Khosla et al., 2020) in this module and define multi-level labels for dif- ferent event levels.
well the corresponding event types Eg. For the Event type level, positive pairs of each event type consist of all referring trigger words and the event type itself. In contrast, the negative pairs comprise irrelevant trigger words and the event type itself. For Event trigger level, each trigger's 's positive pairs are composed of optical flow features that point to the trigger and the trigger. Conversely, the negative pairs are made up of optical flow features that are unrelated to the trigger and the trigger itself.
Considering the i-th sample in this smaller batch, we first enter w' and ex into a pre- pre-trained T5 base model to obtain respective feature representations:
Event type level. Since an event type corre- sponds to various triggers, we use event types the anchors for triggers. Our purpose is to push the triggers referring to the identical event type close. In this level, we define all the event types of the dataset event schema the label set {ei, 62, .... cp}, where p is the number of event types in the dataset event schema.
T5(w"), x² = =
(1)
Then we adopt the supervised contrastive learning to optimize contrastive loss of the Event type level and Event trigger level Ltype and Lirigi
Event trigger level. Considering the same event triggers correspond to similar motion trajectories in videos, we regard the triggers the anchors for OFF. The label set in this level is all the triggers W {w1, u2, in the dataset. For each trigger we could obtain the embedding index from pre- trained language model as the label index.
explx' 2//T)
Ltype
log
Σ Exews coppl' 22/7)'
i=1
B
cap(e Fo/TT
Lirig
log
Σ Fggeoo Aig explz
(z F
1=1
Given a batch of N samples, we first select the samples annotated with one event type for com- puting contrastive loss. It is for the reason that if sample has more than one event, OFF may con- tain multiple motion trajectories. Thus OFF could not be directly assigned the certain single label event. After filtering the samples, we obtain a smaller batch OFF Fo., the trigger words W.. as
where B is the number of samples after filtering, and 7 is the temperature parameter of supervised contrastive learning. Finally the multi-level loss Emaits is defined as
E - Ligpe + Lirig.
(3)

---

Formally, the Multi-level Event Contrastive Learning algorithm is shown as Algorithm
Algorithm 1 Multi-level Event Contrastive Learn- ing
Require:
OFF Fo, event types {el, 62 e2. ..... event triggers W {wi, W2, "2......evem type positive pairs Spy 0, event type negative pairs Say = 0, event trigger positive pairs Spg - 0, event trigger negative pairs Sng - 0, filtering batch S = 0, supervised contrastive learning function CON.
3.4 Dual Querying Text
We design a Dual Querying Text (DQT) module to enhance the interaction among three modalities. The intuition is to query TSF which token responds to VAF or OFF. For example, if the input text has the word Police describing the argument of the event, this token would respond to VAF. It be- cause VAF may contain visual cues whose seman- tics are close to this argument. To encode the dual queries of TSF, We utilize two transformer archi- tectures. The attention scores of each token reflect the degree of response to VAF or OFF.
for (Fo, EY¹ in batch do
if Len then
< T5ww")
x¹ + T5(ee)
5:
Sy.pppend(2 x², Fo)
6:
else
For TSF, VAF, OFF denoted as Fr E Rexxde FV E Rd and Fo RR in Section 3.2, VAF and OFF are projected into queries respectively. In both transformer architectures, TSF is projected to obtain keys and values. Then we adopt a softmax function to calculate the dual attention weights:
7:
CONTINUE
end if
end for
10: for
(2, Fo) in S do
11:
if 2d refers x then
12:
Spy.appen((2,, x²)
13:
else
Fv Hn H FT Vdl Fo Hm H F.T Fr
A. = softmax
Fr
14:
append(22)
15:
end if
16: if Fy refers to 2 then
A. s
(4)
ddt
17:
Spp.appen F.)
18:
else
Where Hg: Hs, He are three projection matrices for query, key and value respectively. The output attention scores are aggregated as follows:
19: Sng append( (2), F
20: end if
21: end for
22: Liype CON(S py +CON(S
F = Ag F Ao Fr
(5)
23: Lirig CON(S (ONN(Sp))COO((5,,) +CON(Sng))
24: return Ltype + Ltrig
4 Experiment
Datasets. We evaluate our approach on two open- ended TVMEE datasets: TVEE (Wang et al., 2023) and VM2E2 (Chen et al., 2021). TVEE dataset contains 7598 text-video pairs. The international news videos with captions are collected from the On Demand News channel. The event schema is from the ACE2005 (Walker, 2006) benchmark that consists 8 superior event types and 33 event types. Contact.Speeckh cident.Accident are added to the event schema be- cause the schema in ACE2005 could not cover all the event types in vidcos. The TVEE dataset is randomly divided into train, valid, and test sets in ratio of 8:1:1. VM2E2 is collection of text and video data that includes 13,239 sentences and 860 videos. Within the dataset, there are 562 pairs sentences and videos that share the same event
type, with each pair containing only one event. The dataset defines 16 multimodal event types based on the LDC ontology. Following (Chen et al., 2021), we split VM2E2 into 411 and 151 samples.
Evaluation Metries. Following (Wang et al.. 2023), we utilize the same evaluation metrics to report text, video and multimodal evaluation re- sults. The evaluation metrics include: Precision (P), Recall (R) and F-score (F1). The performance of text event extraction is evaluated by two sub- tasks: event trigger extraction and event argument extraction. The correctness of a trigger prediction is determined by whether its type and span align with the labels, while for an argument prediction, it is determined by whether its span and all the roles
Disaster.Disaster and Ac-

---

Text Evaluation
Video Evaluation
Multimodal Evaluation
Dataset
Input
Model
Trigger
Argument
P
R
F1
P
R
F1
P
R
F1
P
R
F1
DELPSTRUCT
76.4
75.2
75.8
53.1
48.9
50.9
76.4
75.2
75.8
Text
CoCoEEr
76.0
76.6
76.3
62,9
44.2
51.9
76.0
76.6
76.3
TSEE,
75.7
77.2
76.4
63.3
45.0
52.6
75.7
77.2
76.4
JSL.
48.2
51.6
49.8
48.2
51.6
49.8
TVEE
Video
CoCoEBy
49.1
60.7
54.3
49.1
60.7
54.3
TSEEy
48.7
62.1
54.6
48.7
62.1
54.6
JMMT
74.3
80.2
77.1
50.1
54.9
52.3
55.4
57.0
56.2
87.2
88.6
87.9
Multimodal
CoCoHE
80.7
76.4
78.5
65.6
45.4
53.6
56.4
57.4
56.9
92,9
92.9
92.9
TSEE (ours)
82.6
80.5
81.5
67.0
49.3
56.8
58.2
58.6
58.4
94.4
93.7
94.0
DEEPSTRUCT
44.7
43.1
43.9
19.8
13.2
15.9
44.7
43.1
43.9
CoCoFFr
41.5
45.6
43.5
20.5
15.3
17.5
41.5
45.6
43.5
TSEEr
45.2
41.8
43.4
21.2
17.1
18.9
45.2
41.8
43.4
ISL
21.2
18.6
19.8
21.2
18.6
19.8
VM2E2
Video
CoCoEBy
27.3
31.2
29.1
27.3
31.2
29.1
TSEEy
26.5
30.7
28.4
26.5
30.7
28.4
JMMT
39.7
56.3
46.6
17.9
24.3
20.6
32.4
37.5
34.8
76.1
69.5
72.7
Multimodal
CoCoFR
47.3
47.7
47.5
26.7
18.5
21.8
33.2
37.2
35.1
78.2
75.6
76.9
TSEE (ours)
49.2
53.5
51.6
24.5
27.4
25.9
35.1
38.0
36.5
78.9
77.2
78.0
Table I: Comparison with the state-of-the-art methods. The evaluation metrics are introduced in Section 4.1. The best performed methods each metric are highlighted in bold.
align with the labels.
It proposes structure pretraining to let language model understand the structure in the text.
Implementation Details. We Pytorch and 2080 Ti GPU implement our framework and conduct experiments. We apply a pre-trained T5- base (Raffel et al., 2020), as the TSF encoder. For the video input, we separately adopt pre-trained 13d (Carreira and Zisserman, 2017) and PWC (Sun et al., 2018) to extract VAF and OFF For the event extraction decoder, we use CRF decoder following (Wang et al., 2023). The dimension of TSF, VAF and OFF 768, 1024 and 1024 respectively. We utilize linear function project the dimension VAF and OFF to 768. Following (Wang al., 2023), we train our model for epochs and the batchsize is set 16. The optimizer is Adam and the learning rate is 10e-5. Following (Yao et al., 2022), we utilize 0.3 for the parameter in MECL.
-CoCoEET (Wang et al., 2023): It uses the text encoder and CRF decoder of CoCoEE without CoLearner module.
-TSEET utilizes the T5-base encoder and a CRF encoder to extract events with text modality. without MECL module and DQT module. Video Event Extraction. We only use video input as the video event extraction. We compare the models follows:
-JSL (Pratt al., 2020) :We follow (Wang et al., 2023) to use a sota model JSL in video event extraction. Key frames are utilized to detect events. -CoCoEEy (Wang et al., 2023) It utilizes the video encoder of CoCoEE and video event decoder without CoLearner module.
-SSEEy It utilizes pre-trained I3D model extract video features and the decoder is set the same as (Wang et al., 2023). It also without MECL module and DQT module.
4.1 Baselines
Following (Wang et al., 2023), we compare our model with other methods in three settings, which are Text Event Extraction, Video Event Extrac- tion, Multimodal Event Extraction.
Multimodal Event Extraction. This is our full task setting. We compare the models as follows: -JMMT (Chen et al., 2021) :lt utilizes trans- former encoder jointly encode the text and video inputs. The visual features include video-level fea- tures and image-level features.
Text Event Extraction. For text event extraction, we only utilize text input. We compare the follow- ing models in this setting:
DEEPSTRUCT (Wang et al., 2022) It is the state-of-tt art method in text event extraction.
-CoCoEE (Wang et al., 2023) :It is the state-of- the-art model text-video based event extraction.

---

Units
Trigger
Argument
Dataset
76.2
76.9
76.5
62.8
46.1
53.2
76.8
77.3
77.0
63.9
45.7
53.3
TVEE
80.5
79.2
79.8
64.5
47.3
54.6
82.6
80.5
81.5
67.0
49.3
56.8
21.3
18.7
44.0
47.2
45.5
20.8
18.1
19.4
VM2E2
47.9
49.2
22.7
25.3
23.9
49.2
53,5
51.6
24,5
27.4
25,9
w/o MECL
MECL
Ablation study on three units in TSEE 'O" represents OFF (Optical Flow Features). 'H' means MECL (Multi-level Event Contrastive Learning) mod- ule. 'D' denotes DQT (Dual Querying Text) module. ' represents our framework is equipped with the unit.
Table 2:
Figure 3: T-SNE visualization for MECL module. w/o MECL (a) denotes that MECL is removed from TSEE and w/ MECL (b) means that TSEE is trained with MECL module. Each dot represents one OFF and cach color denotes specific event trigger.
It contrasts the event types and video features.
notice that JMMT performs well in the recall met- ric, such 54.9% argument extraction recall on TVEE dataset and 56.3% trigger extraction recall on VM2E2 dataset. This may be that JMMT uti- lizes the additional object detection model to inject proposal features of key frames to the transformer encoder, improving the recall of triggers and argu- ments in samples.
4.2 Main Results
The experiment results on TVEE and VM2E2 datasets are presented in Table 1. We could find that when the input data only consists of text, DEEPSTRUCT, CoCoBET and TSEE, achieve similar performance. Specifically, on TVEE dataset TSEET performs 0.1% and 0.6% better than CoCoEE, on Fl of trigger extraction. However, DEEPSTRUCT reaches a higher F1 than TSEET and CoCoEE on VM2E2 dataset. This could be because the ability to extract event information from text of the three models is comparable.
We also observe that the results show the similar trends from single modality to multimodal input. which verifies that injecting multimodal input to TSEE and CoCoBE both boosts the performance in all metrics. Specifically, the incorporation of video to TSEEy boosts the F1 performance from 76.4% to 81.5% on TVEE dataset. For CoCoEEy, the F1 score improved from 76.3% to 78.5%.
When there is only video data in input, CoCoEEy and TSEEv show the comparable perfor- datasets.
mance and are both better that JSL It because JSL applies static frames while CoCoFFy and TSEEy adopt pre-trained models of videos to capture dynamic information.
4.3 Ablation Study
To validate the effectiveness of different innova- tions and parts in TSEE, conduct ablation stud- ies the two datasets. We investigated three main units of TSEE: (1) Integration of optical flow fea- ture; (2) Multi-level Event Contrastive Learning module; (3) Dual Querying Text module. The base- line in the first line applies sum function to VAF and TSF, The results are summarized in Table 2. Effectiveness of OFF. In this part extract OFF from video data and sum up VAF, TSF and OFF, From Table 2, we observe that the integration of OFF improves all evaluation metrics over baseline on the two datasets, verifying that OFF provides beneficial information for event extraction.
The results of multimodal data input show that TSEE achieve the best performance on most of the evaluation metrics compared with existing state- of-the-art methods. On TVEE dataset, our model reaches 81 81.5% Fl of trigger extraction compared with 77.1% and 78.5% achieved by JMMT and Co- CoEE respectively. This result demonstrates that the integration motion representations videos is helpful for the multimodal event extraction task. On VM2E2 dataset, the F1 score of trigger extrac- tion is improved from 47.5% (CoCoEE) to 31.7%, where the improvement is larger than that of TVEE dataset. This may be the reason that in VM2E2 dataset, each sample is annotated with only one event. The MECL module would not filter any sample every batch when computing contrastive loss, thus obtaining better feature representations and boosting the performance the model. We
Effectiveness of MECL. To evaluate the influence of MECL module, we utilize MECL module based on the second line of each part. shown Table 2, MECL module brings the most improvement to our framework, such as 2.8% trigger extraction Fl

---

0.14
0.62
0.11
0.39
0.22
0.04
0.07
0.15
0.01
0.08
0.04
0.12
0.09
0.10
0.05
008
0.04
0.01
0.02
0.07
0.54
0.18
0.01
0.04
0.11
0.04
0.06
0.02
0.08
0.03
0.03
0.01
police
and
protesters
clasted
macch
against
Covid
passes
8
Parts
the
weeland
05
0.08
0.09
0.02
0.04
0.01
0.01
0.05
0.04
0.03
0.01
0.08
0.04
0.12
0.06
0.02
0.07
0.02
0.15
0.01
0.02
0.04
0.03
0.04
0.91
0.06
0.03
0.01
0.04
0.01
0.02
0.05
0.03
02
Trmp
Sumpped
Into
North
Konua
during
meeting
with
km
Jang
un
the
demiitarised
zo1e
Figure 4: Case study on DQT module. Attention heatmaps of two randomly sampled examples are visualized. each example, the first line is VAF and the second line is OFF.
score on TVEE dataset and 3.7% trigger extraction Fl score on VM2E2 dataset. This demonstrates that MECL module could refine the feature repre- sentations and align the features between heteroge- neous modalities, thus boosting the performance.
4.5 Case Study on DQT
In order intuitively show the effectiveness of DQT module, conduct case studies on TVEE dataset. shown Figure 4, we visualize the attention heatmaps based on the attention scores output by DQT. As DQT utilizes VAF and OFF to query each token in TSF respectively, each sample corresponds to two lists of attention scores. From Figure 4, we could observe that for each sample, the frame appearance motion related tokens paid more attention by VAF or OFF. In the first example, When VAF queries police, protesters and march, gives more attention scores than other to- kens. We could also observe that OFF attends to the clashed most. In the second example, each token is allocated similar attention score by VAF. This may be the reason that the pre-trained I3D model does not have the knowledge of instances such Trump and Kim. OFF gives higher attention score to meeting because this token could provide motion information. From the above analysis we can see that DQT module does well understanding the relationship between multimodal semantics.
Effectiveness of DQT. We also evaluate the im- pact of DQT module. From Table 2, we could find that DQT module improves the performance of all evaluation metrics significantly the two datasets. is worth noting that the recall metric of trigger extraction is boosted from 50.6% to 53.5% on VM2E2 dataset and so is argument extraction recall metric from 25.3% to 27.4%. The reason is perhaps that in the DQT module, each text token is queried by the VAF and OFF, thus enhancing the ability of searching instances contained in videos.
4.4 Visualization of T-SNE for MECL
To verify the impact of MECL module, we use t-SNE (Van der Maaten and Hinton, 2008) to vi- sualize the manifold of TSEE with and without MECL module. Our MECL module is designed to reduce the distance between OFF and event triggers. We randomly sampled 1500 OFF trained with or without MECL module on TVEE dataset. The visu- alization results are shown in Figure 3, where OFF belonging to the same trigger is marked in the same color. It could be clearly seen that OFF trained with MECL module in subfigure is obviously separated into various compact clusters. However, when OFF is trained with MECL module, there is dis- tinctiveness between OFF belonging different triggers. This result demonstrates that our MECL. module does well in aligning the semantics of dif- ferent modalities.
Conclusion
In this paper, we propose a Three Stream Multi- modal Event Extraction framework that explores the utilization of motion representations in text- video based multimodal event extraction (TVMEF) tasks. Optical flow features extracted from videos as motion representations to incorporate with other modalities. To improve alignment among feature representations, we propose Multi- level Event Contrastive Learning module. Dual Querying Text module also designed to help en-

---

hance the interaction between different modalities. TSEE achieves the state-of-the-ar results on two datasets, demonstrating the effectiveness of our framework. future work, we will explore the uti- lization of large language model (LLM) in fusing modality features to boost TIMEE performance.
Wenting Song, and Huajun Chen. 2022c. Meaformer: Multi-modal entity aligument transformer for meta modality hybrid. arXiv preprint arXiv:2212. 14454.
Zhuo Chen, Lingbing Guo. Yin Fang, Yichi Zhang. Jiaoyan Chen, Jeff Z Pan, Yangning Li, Huajun Chen, and Wen Zhang. 2023, Rethinking uncertainly miss- ing and ambiguous visual modality in multi-mode entity alignment. arXiv preprint arXiv:2307. 16210.
Limitations
The main limitation of our work is the offline train- ing, As the insufficiency of GPU resources, we need to extract the VAF and OFF in advance and could not optimize the video pre-trained model online. The other limitation is the inapplicabil- ity of open-domain event extraction. As both two datasets are annotated in a close-domain event set, our framework can not deal with open-domain event extraction.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understand- ing, arXiv preprint 170.v21810.04805.
Alexey Dosovitskiy, Philipp Fischer, Eddy flg Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. 2015. Flownet: Learning optical flow with convolu- tional networks. In ICCV.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoy- anov. 2020. Supervised contrastive learning for pre- trained language model fine-tune fine-tuning, arXiv preprint arXiv:2011.01400
Acknowledgements
This work is partially supported by National Nature Science Foundation of China under No. U21 A20488, 62302149 and 62372155. We thank the Big Data Computing Center of Southeast Uni- versity for providing the facility support on the numerical calculations in this paper.
Junjie Huang, Wei Zou, Zheng Zhu, and Jiagang Zhu. 2019. An efficient optical flow based motion detec- tion method for non-stationary scenes. In CCDC. IEEE.
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2017, Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR.
References
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR.
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky, 2020. Pretraining with contrastive sentence objec- tives improves discourse performance language models. arXiv preprini 2035:
Kelvin CK Chan, Shangehen Zhou, Xiangyu Xu, and Chen Change Loy. 2022. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In CVPR.
Huaizu Jiang and Erik Learned- 2023. Devnet: Dilated cost volume networks for fast optical flow. In WACV.
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, and Shih- Fu Chang. 2021. Joint multimedia event extraction from video and article. In EMNLP Findings.
Aakash Kaku, Sahana Upadhya, and Narges Razavian. 2021. Intermediate layers matter in momentum con- trastive self supervised learning. NIPS, 34:24063- 24074.
Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng. Chuanqi Tan. Changliang Xu, Fei Huang. Luo Si. and Huajun Chen. 2022a. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In SIGIR.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samn. Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Super- vised contrastive learning. NIPS
Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao. Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Good visual guidance makes better extractor: Hierarchical visual prefix for multi- modal entity and relation extraction. arXiv preprint arXiv:2205.03521.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. 2020. Adversarial self-supervised contrastive learn- ing. NIPS
Seonhoon Kim, Seohyeong Jeong, Funbyul Kim, Inho Kang, and Nojun Kwak. 2021. Self-supervised pre- training and contrastive representation learning for multiple-choice video qa. In AAAI.
Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yuxia Geng, Jeff Z Pan,

---

Manling Li, Ruochen Xu, Shuohang Wang. Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022. Clip-event: Con- necting text and images with event structures. In CVPR.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. JMLR.
Manling Li, Alireza Zareian, Qi Zeng, Spencer White- head, Di Lu. Heng Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multime- dia event extraction. In ACL.
Arka Sadhu. Tanmay Gupta, Mark Yatskar, Ram Neva- tia, and Aniruddha Kembhavi. 2021. Visual semantic role labeling for video understanding. In CVPR.
Xiaohui Song, Longtao Huang, Hui Xue, and Songlin Hu. 2022. Supervised prototypical contrastive learn- ing for emotion recognition in conversation. arXiv preprini urXiv:2210.087/3
Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, and Sanja Fidler. 2017. Situation recognition with graph neural networks. In ICCV.
Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. 2018. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR.
Fan Liu, Delong Chen, Xiaoyu Du, Ruizhuo Gao, and Feng Xu. 2023a. Mep-3m: A large-scale multi- modal e-commerce product dataset. Pattern Recogni- tion.
Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV. Springer.
Fan Liu, Delong Chen, Fei Wang. Zewen Li, and Feng Xu. 2023b. Deep learning based single sample face recognition: a survey, Artificial Intelligence Review.
Meihan Tong, Shuai Wang, Yixin Cao, Bin Xu, Juanzi Li, Lei Hou, and Tat-Seng Chua. 2020. Image en- hanced event detection in news articles. In AAAI.
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction machine reading com- prehension. In EMNLP.
Laurens Van der Maaten and Geoffrey Hinton, 2008.
Visualizing data using t-sne. JMLR.
Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2019, Neural cross-lingual event detection with minimal parallel resources. In EMNLP.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. arXiv preprint 139.03566
Jian Liu, Yufeng Chen, and Jinan Xu. 2022. Multimedia event extraction from news with a unified contrastive learning framework. In MM.
Strassel S. Medero J. Maeda K. Walker, C. 2006. Ace 2005 multilingual training corpus. LDC.
Shuaicheng Liu, Kunming Luo, Nianjin Ye, Chuan Wang, Jue Wang, and Bing Zeng. 2021. Oiflow: Occlusion- Declusion-inpainting optical flow estimation by un- supervised learning. TIP.
Chenguang Wang. Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022. Deepstruct: Pre- training of language models for structure prediction. In ACT.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In ACL.
Shuo Wang, Meizhi Ju, Yunyan Zhang, Yefeng Zheng, Meng Wang. and Guilin Qi. 2023. Cross-mode con- trastive learning for event extraction. DASFAA. Springer.
Ao Luo, Fan Yang. Kunming Luo, Xin Li, Haoqiang Fan, and Shuaicheng Liu. 2022. Learning optical flow with adaptive graph reasoning. In AAAL.
Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. 2021. Cleve:
Contrastive pre pre-traie training for event traction. arXiv preprint 1xxi::2005.1448.
Rémi Marsal, Florian Chabot, Angélique Loesch, and Hichem Sahbi. 2023. Brightflow: sesabness-change aware unsupervised learning of optical flow. In WACV.
Patrick Y Wu, Richard Bonneau, Joshua A Tucker, and Jonathan Nagler. 2022. Dictionary-assisted supervised contrastive learning. arXiv preprint arXiv:2210.15177
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Gr- ishman. 2016. Joint event extraction via recurrent neural networks. In NAACL.
Sai Yang, Fan Liu, Delong Chen, and Jun Zhou. 2023a. Few-shot classification via ensemble learn- ing with multi-order statistics. arXiv preprint arXiv:2305.0045:
Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In ACL, pages 365-371.
Sai Yang, Fan Liu, Shaoqiu Zheng, and Ying Tan. 2023b. Jlcsr: Joint learning of compactness and separabil- ity representations for few-shot classification. IEEE Transactions on Cognitive and Developmental Sys- lems.
Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded situation recognition. In ECCV. Springer:

---

Shunyu Yao, Jian Yang, Xiangqun Lu, and Kai Shuang. 2022. Contrastive learning for event extraction. In ICMI, pages 167-172.
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In CVPR.
Zhenrui Yue, Huimin Zeng, Bernhard Kratzwald, Ste- fan Feuerriegel, and Dong Wang. 2022. Qa domain adaptation using hidden space augmentation and self- supervised contrastive adaptation. arXiv preprint arXiv:2210.108 10861.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Huang, Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2 11..2108.111.. 13161.
Tongtao Zhang, Spencer Whitehead, Hanwang Zhang, Hongzhi Li, Joseph Ellis, Lifu Huang, Wei Liu, Heng Ji, and Shih-Fu Chang. 2017. Improving event traction via multimodal integration. MM.
Zihua Zheng, Nie, Zhi Ling, Pengfei Xiong, Jiangyu Liu, Hao Wang, and Jiankun Li. 2022. Dip: Deep inverse patchmatch for high-resolution optical flow.
CVPR.
Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. 2021. Crosselr: Cross-modal con- trastive learning for multi-modal video representa- tions. ICCV.


Reviews:
[{"review_id": "fAkMUMCxtJ", "reviewer": "Reviewer_necu", "summary": "", "questions": "1. Most datasets have been set up with no attention to motion. Have you considered creating a dataset that would better show off your algorithm's motion characterization? What would be the challenges there? Note that I am NOT asking you to create a new dataset, only express your educated opinion on a possible new dataset.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are greatly encouraged by your positive comments. Thanks a lot for all the appreciation. We will revise the typos in our final version.\n\n\n>**Q1**: Most datasets have been set up with no attention to motion. Have you considered creating a dataset that would better show off your algorithm's motion characterization? What would be the challenges there?    \n\n**A1**: Thank you for your professional and rigorous thinking. We think it would be promising to set up a dataset with more attention to motion. The new dataset could be constructed jointly with a Computer Vision task, named spatio-temporal action localization. The downstream task of the dataset is to locate the start time and end time of a motion trigger in the text caption from the long video. We suppose the challenge in this task for a model is the capacity of completely understanding the features of text and video. Identifying the start and end time is a high-level ability of recognition motion representation, let alone understanding heterogeneous motion triggers. The dataset needs more refined motion feature extraction, which could better show off our algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough literature survey.\n2. Innovative incorporation of optical flow features through multi-level event similarity and contrast in a contrastive learning framework.\n3. Clear advancement of the state of the art.\n4. Insightful ablation studies.", "reasons_to_reject": "1. I do not see any signficant weaknesses in the paper. The authors do describe their limitations which I don't think count as significant weaknesses. ", "questions_to_authors": "1. Most datasets have been set up with no attention to motion. Have you considered creating a dataset that would better show off your algorithm's motion characterization? What would be the challenges there? Note that I am NOT asking you to create a new dataset, only express your educated opinion on a possible new dataset.", "paper_topic_and_main_contributions": "This paper presents a novel three stream multimodal event extraction approach that in addition to text and visual appearance features also incorporates video motion features. The Motion feature extraction is combined with a multi level event contrastive module to establish similarity and contrast using motion features. In my view, that is the main innovation of the proposed work. The proposed technique achieves significant improvement over the state of the art and its various aspects are thorough clarified through well designed ablation studies.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references to my knowledge.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 015 typo please replace “moviated” by “motivated” (not good to have typos in the abstract)\nLIne 555 please replace “provides” by “provide”\n"}, "extra_text": {}}, {"review_id": "xoe3OCPNR0", "reviewer": "Reviewer_aGn9", "summary": "", "questions": "contrastive learning usually requires a lot of data for the negative and positive pairs formation, since the existing dataset is small it may or may not affect the result as you scale up. Is there any way you can think about any idea borrowed from score based/diffusion models for this framework? It might be helpful in a dual way from image to event trigger and also event trigger to image generation as it may learn the joint distribution of the embeddings.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your attentive comments and providing thoughtful feedback on our work. We will provide our insights point by point below.\n\n> **Q1**: Is there any related work that supports the claim of affecting event trigger extraction from videos due to the other noise in the frames? \n\n**A1**: Thank you for your thoughtful question. Works [1], [2] and [3] could all support the claim that the background noise in the frames affects event trigger extraction. In their works, they directly propose to obviate the influence of background noise of the frames or images for better information extraction. We will add the related work to support the claim in our final revised version.\n\n> **Q2.1**: Specific reason to sample 16 frames.\n\n**A2.1**: Thank you for your valuable question.  We sample from the video every 16 frames because the input for backbones ( I3D Model and PWC Net ) are restricted to 16 frames. In fact, most backbones ( [4], [5] ) for video features extraction are required 16 frames input such as C3D, R3D and S3D.\n\n> **Q2.2**: Other sampling algorithm such as importance sampling and random sampling. \n\n**A2.2**: Thank you for your pertinent suggestion. We suppose that importance sampling could provide another insight in extracting video appearance features. For video appearance features which mainly provide the scene information, it does not really need to sample continuous frames. To validate the idea, we conduct several additional experiments. We utilize the key frame sampling algorithm of OpenCV as the importance sampling strategy.  Besides, we conduct the experiment of random sampling. The results are summarized as follows: \n\n\n\n|        Model        | Trigger F1 | Argument F1 | Video F1 | Multimodal F1 |\n| :-----------------: | :--------: | :---------: | :------: | :-----------: |\n| Importance sampling |    80.2    |    55.1     |   56.2   |     92.4      |\n|   Random sampling   |    79.3    |    53.7     |   55.9   |     90.7      |\n|     TSEE (ours)     |    81.5    |    56.8     |   58.4   |     94.0      |\n\nThe results show that changing sampling strategy does not help improve the performance for our framework.\n\n\n\n> **Q3**: replace the optical flow feature and use the saliency feature from the video frames\n\n**A3**: Thank you for your critical consideration. Both saliency feature and optical flow feature could alleviate the background noise. We utilize ITSD [6] to extract salience feature. The experiment results are shown as follows:\n\n\n\n|       Model       | Trigger F1 | Argument F1 | Video F1 | Multimodal F1 |\n| :---------------: | :--------: | :---------: | :------: | :-----------: |\n| TSEE (two stream) |    76.5    |    53.2     |   52.5   |     89.9      |\n| Salience feature  |    79.8    |    55.4     |   56.7   |     91.3      |\n|    TSEE (ours)    |    81.5    |    56.8     |   58.4   |     94.0      |\n\nTSEE (Two stream) means only VAF (video appearance feature) and TSF (text sequence feature) are fed into the framework. It could be observed that incorporating salience feature achieves higher performance than TSEE (two stream). This result demonstrates the effectiveness of sailence feature. However, salience feature less considers the motion representations in videos, which is inferior to optical flow feature.\n\n\n\n\n\n> **Q4**: Any idea borrowed from score based/diffusion models for this framework.\n\n**A4**: Thank you for your insightful comment on our framework. We think you provide a novel idea. Trigger-guided generation model may promote the performance of our framework. The generation model could first encode the motion trajectories of each event trigger. Then we could generate any image/video from given event trigger. During the encode/decoder process, the framework may learn more discriminative motion representations.\n\n\n\n\n\n\n\n\n\n\n\n# References\n\n[1] Liu, Liu, et al. \"Activity image-to-video retrieval by disentangling appearance and motion.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 35. No. 3. 2021.\n\n\n\n[2] Rajnoha, Martin, Radim Burget, and Lukas Povoda. \"Image background noise impact on convolutional neural network training.\" *2018 10th international congress on ultra modern telecommunications and control systems and workshops (ICUMT)*. IEEE, 2018.\n\n\n\n[3] Zhang, Jingke, et al. \"Improved background noise suppression in ultrasound localization microscopy using spatial coherence beamforming.\" *2021 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2021.\n\n\n\n[4] Abdessaied, Adnen, Ekta Sood, and Andreas Bulling. \"Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA.\" *Proceedings of the 7th Workshop on Representation Learning for NLP*. 2022.\n\n\n\n[5] Yu, Tiezheng, et al. \"Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization.\" *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. 2021.\n\n\n\n[6] Zhou, Huajun, et al. \"Interactive two-stream decoder for accurate and fast saliency detection.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. observation is good. Using extra visual features and fusing them into the learning is a good way of utilizing the visual feature more to understand the event.\n2. T-SNE visualization looks promising in the ablation study part, the clusters support the claimed results and observation.\n3. The proposed method outperforms almost all the parameters in the evaluation section.", "reasons_to_reject": "1. Is there any related work that supports the claim of affecting event trigger extraction from videos due to the other noise in the frames?\n\n2. \"We sample from the video every 16 frames to get the clip sequence\"- is there any specific reason to sample 16 frames? did you experiment with any other sampling algorithm i.e. importance sampling? what if for each epoch you randomly sample frames and feed it?\n\n3.  is it possible to replace the optical flow feature and use the saliency feature from the video frames, Although saliency does not represent motion information it can help with suppressing the background noise and only focus on the relevant information required for the event trigger.\n\n", "questions_to_authors": "contrastive learning usually requires a lot of data for the negative and positive pairs formation, since the existing dataset is small it may or may not affect the result as you scale up. Is there any way you can think about any idea borrowed from score based/diffusion models for this framework? It might be helpful in a dual way from image to event trigger and also event trigger to image generation as it may learn the joint distribution of the embeddings.", "paper_topic_and_main_contributions": "In this frame, The authors leverage three modality features text sequence, video, and motion representations. The observation and motivation for the problem is clear. The authors utilize contrastive loss between optical flow features and even triggers as well as event triggers and event types to make sure similar motion representations correspond to similar event triggers. This way the framework extracts the relevant information from the video required to generate a suitable event trigger and type. Further studies show this framework generates state-of-the-art results in various evaluation metrics. Their claims are also supported by ablation studies. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "in Fig 3. I believe there is a typo! it should be MECL, not HECL!"}, "extra_text": {}}, {"review_id": "CqnLZKNiCc", "reviewer": "Reviewer_ZJRe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical feedback and suggestions. We address your thoughts point by point below.\n\n> **Q1**: Optical flow feature is commonly used in vision and calling using this as first novelty lacks basis.\n\n**A1**: Thank you for your thoughtful question. Our statement on the contribution may not be appropriate and it may cause misunderstanding. In TVMEE subtask, we are the first to associate the optical flow feature with the event triggers. This correlation is hardly influenced by background noise. Moreover, the performance is improved by our proposed contrastive learning module which aligns the embedding space of two elements. Thus, we regard introducing the optical flow feature as an important contribution and a good novelty. We agree with you that just mentioning optical flow feature may not be the first novelty and contribution. We will restate our first novelty in the final revised version. \n\n> **Q2**: Section 3.1 of task definition is not well explained.\n\n**A2**: Thank you for your rigorous review. Our task is to identify the triggers and arguments in text. For video modality, there is no trigger and argument annotation. We represent events in videos using event type as the definition of line 212 $E=(e_{1},e_{2},...) $.  We will add the explanation in our final version.\n\n> **Q3**: The authors mention in the abstract that previous research disregard motion representation in videos. This is not correct. \n\n**A3**: Thank you for your pertinent comment. What we claim in the abstract may cause ambiguity. There are indeed many works in other related fields of video that utilize motion information. However, previous research we mentioned refers to works on Text-Video event extraction.  They ignored the motion representations as illustrated in the introduction section. Nevertheless, we observed the correlation between motion representations and event triggers and then introduced this correlation to our framework.  We will make our statement clearer in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach is logical. Given the event extraction task, utilizing flow based feature is intuitive and utilizing a query module to map tokens to this as well as using contrastive learning based on event types and triggers are well founded.\n2. The experimental results are solid with clarity on the effect of each component.\n3. The paper is well written with clear explanation of the approach and the experimental setup making the results reproducible.", "reasons_to_reject": "1. Optical flow feature is commonly used in vision and calling using this as first novelty lacks basis.\n2. Section 3.1 of task definition is not well explained. Authors start by mention the setting of multimodal event extraction but only give an example of text based event.\n3. The authors mention in the abstract that previous research disregard motion representation in videos. This is not correct. Video transformers do capture motion.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new approach in text-video based multimodal event extraction. They introduce three components - 1) utilizing the optical features for this task, 2) multi-level event contrastive learning to align embedding spaces, and 3) dual querying text which maps the tokens to visual appearance feature (VAF) or optical flow feature (OFF). The authors show that the approach outperforms unimodal and multimodal baselines and show that majority of the improvement comes from components 2 and 3.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method for text-video event extraction based on a three-stream multi-level event contrastive learning approach. In addition to text and visual appearance features, this work incorporates video motion features based on optical flows, significantly improving the proposed system from previous studies.

The reviewers' evaluations are primarily high. I don't find any critical issues in the reviews. "Reasons to reject" from reviewers look limited to minor issues and questions for clarification. The authors responded carefully to such reviewers' concerns and questions in the rebuttal.
 
One reviewer asked about related works on noise-robust event trigger extraction, the appropriateness of every 16-frame sampling, and the effectiveness of optical flow features against saliency-based features. The authors answered the questions with additional experimental results, showing that the proposed method outperforms different sampling approaches and saliency features.

Another reviewer was concerned about the novelty of optical flow features and motion representation. For these concerns, the authors explained that associating the optical flow features with the event triggers is novel, and the use of motion representation is also novel in text-video event extraction. The reviewer acknowledged the authors' response and raised the soundness score to 4.

Considering the reviews and discussions, the paper is well-written, showing that the proposed method is technically sound and effective for text-video event extraction.
Decision: Accept-Main