Transitioning Representations between Languages for Cross-lingual Event Detection via Langevin Dynamics
Authors: Chien Van Nguyen
Huy Huu Nguyen
Franck Dernoncourt
Thien Huu Nguyen
Venue: EMNLP 2023 Findings (2023)
Keywords: Event Detection
Information Extraction
Cross-lingual Transfer Learning
Langevin Dynamics
TL;DR: We introduce a novel method for cross-lingual transfer learning for event detection that utilizes Langevin Dynamics to transition target language representations into the source language space.

Abstract: Cross-lingual transfer learning (CLTL) for event detection (ED) aims to develop models in high-resource source languages that can be directly applied to produce effective performance for lower-resource target languages. Previous research in this area has focused on representation matching methods to develop a language-universal representation space into which source- and target-language example representations can be mapped to achieve cross-lingual transfer. However, as this approach modifies the representations for the source-language examples, the models might lose discriminative features for ED that are learned over training data of the source language to prevent effective predictions. To this end, our work introduces a novel approach for cross-lingual ED where we only aim to transition the representations for the target-language examples into the source-language space, thus preserving the representations in the source language and their discriminative information. Our method introduces Langevin Dynamics to perform representation transition and a semantic preservation framework to retain event type features during the transition process. Extensive experiments over three languages demonstrate the state-of-the-art performance for ED in CLTL.

Transitioning Representations between Languages for Cross-lingual Event Detection via Langevin Dynamics
Chien Van Nguyen', Huy Huu Nguyen',
Franck Dernoncourt', and Thien Huu Nguyen
DDpparmmen of Computer Science, University of Oregon, Eugene, OR, USA
VinAI Research, Vietnam Adobe Research, USA
{chienn, thien@cs edu v. v.huynh16évi io, franck. www.onncourt@dddoeee.oo com
Abstract
However, despite such progress, existing datasets for ED are limited to a small set of popular languages (e.g., English and Chinese) due to the high cost of data annotation for this task. The lack of annotated data has thus hindered the develop- ment of ED systems for larger set of languages and the benefits of the technology for broader soci- ety. To enable ED for more languages while avoid- ing expensive annotation efforts, recent research has focused on zero-shot cross-lingual transfer learning (CLTL) for ED where models are trained on annotated data a high-resource source lan- guage and directly applied to extract events for text in another low-resource target language (Nguyen et al., 2021; Guzman Nateras et al., 2023; Nguyen et al., 2023). In this setting, labeled data is not avail- able for the target language; however, unlabeled data can be used facilitate knowledge transfer between languages.
Cross-lingual transfer learning (CLTL) for event detection (ED) aims to develop mod- els high-resource souree languages that can be directly applied produce effective per- formance for lower-resource target languages. Previous research in this area has focused representation matching methods to develop a language-univent representation space into which source- and larget-language example rep- resentations can be mapped to achieve cross- lingual transfer. However, as this approach modifies the representations for the source- language examples, the models might lose dis- criminative features for ED that are learned over training data of the source language to pre- vent effective predictions. To this end, our work introduces novel approach for cross-lingual ED where we only aim to transition the rep- resentations for the target-language examples into the Nource-languse space, thus preserving the representations in the source language and their discriminative information. Our method introduces Langevin Dynamics perform rep- resentation transition and semantic preserva- tion framework to retain event type features during the transition process. Extensive exper- iments over three languages demonstrate state-of-the-ard performance for ED in CLTLL
The key challenging in CLTL for ED bridge the gap between representation spaces for the source and target languages to enable knowl- edge transfer. As such, previous work in this area has focused on the representation matching ap- proaches, aiming to transform the representations for the source and target languages into com- mon space train language-general ED models over source language data, e.g., via similarity reg- ularization (Nguyen et al., 2021) or adversarial lcarning (Guzman-Naterae et al., 2022). However, to achieve a common space, the source-lang source-language representations might need to sacrifice language- specific information, which can involve discrimi- native features for event prediction due to the mix of information in the representations for input text. Consequently, losing discriminative features might lead to impaired performance for ED.
1 Introduction
Event Detection (ED) is one of the fundamental problems Information Extraction (IE). This task aims to identify and classify the words or phrases that most clearly evoke events in text (called event triggers). For instance, given a sentence "He was fired from the corporation yesterday.", an ED sys- event
tem needs to recognize the word "fired" trigger of the type Attack. With the advances in deep learning, recent research has produced impres- sive performance for ED (Nguyen and Grishman, 2015; Liu et al., 2017; Yang et al., 2019; Liu et al., 2020; Tran Phu and Nguyen, 2021).
To this end, our work explores a new direction for CLTL for ED using representation transition. Instead of transforming the source-language repre- sentations, which might eliminate discriminative

---

features for ED, we only seek to adapt the target- language representations into the source space whose results can be directly fed into the source- language models for event prediction. In this way, we can preserve the source-language representa- tions learned over training data for ED to maximize prediction ability. To achieve this idea, we propose to employ an energy function to model the represen- tations for the source-language training examples where low-energy regions correspond to high like- lihood of belonging to the source language (LeCun et al., 2006). Afterward, target-language repre- sentations can be adapted to the source- languag space by transitioning them into low-energy re- gions (i.e., minimizing the energy function). As such, our method leverages Langevin Dynamics (Welling and Teh, 2011; Du and Mordatch, 2019) perform efficient representation transition for the target language for cross-lingus linsull ED where gra- dient descent utilized to iteratively update the initial target representations into the source lan- guage with low-energy regions. To our knowledge, this is the first work to explore energy functions and Langevin Dynamics for CLTL in IE and ED. Finally, we introduce a mechanism to regulate the adapted representations for the target language examples from Langevin Dynamics so they can maintain semantic similarity of event types with the original representations. This regularization is necessary to ensure that the predictions of the source language model over the adapted representa- tions correct for the target examples. Extensive evaluations over three languages demonstrate our state-of-the-un art performance for cross cross-lingual ED,
the parameters of XLM-R. Afterward, the represen- tation h; is fed into feed-forward network (FFN) with parameters O to produce a label distribution for w; for event prediction. For convenience, in the following, we will use the subscripts src and tgt to specify words and representations from the source and target languages (respectively).
Energy Model and Langevin Dynamies: As dis- cussed above, to perform CLTL for ED, we pro- pose to learn an energy function (LeCun et al., 2006) to model the representation distribution of the source- languae examples. The representa- tions of the target-language examples can then be transitioned toward this source distribution to perform ED. Formally, in our method, an energy function Egg (h) B + R is implemented as a feed-for forward network with parameters f5 to map representation h to a scalar. This function can then be used to define a distribution over the rep- resentation space: pag (h) exp( Eg (h)) where Zog
30 = =f c'cpp( Fe (h))dh denotes the normal- izing term. As such, to draw sample from this distribution, we can employ the Langevin Dynam- ics process (Du and Mordatch, 2019): h 1 V E(h*) + w, where w NO, σ) is a Gaussian noise and denotes the step size. Here, the initial representation ho" will be sampled from an uniform distribution, and then transitioned along the direction of the gradient of Egg as in gradient descent. After K steps, the resulting representation h. is expected to have low energy, thus having higher chance to be distributed from pep (h).
Training: Given the training data in the source language, our method trains an ED model and an energy function for the source examples in two steps. In the first step, the base ED model is trained using the negative log-likelihood loss, optimizing the parameters 0x and Og for XLM-R and FFN: Liase (0x,0c) = Σ log(v""(lu"") By, fc). In the second step, we freeze the parameters 0x for XLMMR and train the energy function Fog by optimizing the negative log- ligeeiilddood for the in- duced representations h; for wi E W in the source language: Leng = - EjEL log Peg (Igro), where L involves indexes for the words in the event trigger spans in W. Here, although computing the normal- izing term Zog for E is intractable, the gradient of log Pog (h) can still be estimated perform train- ing in our second step (Du and Mordatch, 2019; Song and Kingma, 2021): Veg log Po (h) = E (P)) [- Vog Eag (h) + EP (A) (Vog Fog (h)).
2 Model
Base Model: Following prior work (Guzman- Nateras et al., 2022), we formulate ED as a se- quence labeling problem to facilitate cross-lingual transfer learning (CLTL). Given an input sentence of tokens W [Wi, W2..-... Wn), our ED model needs to predict its corresponding sequence of labels
{y) . )- Here, y/ is the label for the token we using the BIO annotation schema to capture event mentions and types in W. first use the multilingual
In our model, pre-trained language model XLM-R (Conneau al., 2019) as the feature extractor for the input sentence W, leading to a representation vector h; for each word wy: h1, he.....,u, fnn = XLM-RA, Rox (w), W2,.... w.). Here, 0x captures

---

As such, the data for the first term is drawn from the representations for the source language examples Paata(M) (h) while those for the second term can be sampled from the model representation distribution Pog (h) using our Langevin Dynamics process.
the Kullback-Leibler (KL) divergences serve as the regularization. This loss can be obtained from un- labeled data for both source and target languages. For the language-invarian representation, we expect 2 to encode information on event types to allow transferability across languages. As such, we utilize to predict the event type y using training examples in the source language. In particular, a feed-forward network is introduced to compute the distribution P(y]2) from and the log-likelihood gg (=]k) [log P(y|=) =]] is used to train the network. For the representation 1, its language-specifee in- formation is achieved by computing the distribu- tion P(d[t) with a feed-forward network FFL to predict the language identity d for h (i.e., d e (source, targer)). FF can then be trained with the objective Eg,, (4]b) [log P(d[t)] using unlabeled data in both source and target languages. Conse- quently, to promote the expected features for and I, we minimize the combined function: Ldec =
At inference time for CLTL, given an input sen- tence Wiet in the target language, the representa- tion h for each word t E Wist from XLM-R is iteratively adapted to the source space using K- step Langevin Dynamics, resulting in the adapted representation 1.91 Afterward, due to their compat- ibility, we can apply the source-trained ED model 19t tC over h, to predict the event type for w
Semantic Preservation: Although Langevin Dy- namics can adapt the representation h! to achieve features in the source language, it cannot guarantee that the adapted representation Fig can preserve se- mantic information of event types from h/t to pro- duce correct prediction based on h To facilitate semantic maintenance in the adaption, we propose decompose a representation h into language- invariant and taaguage--pecifii components 2 and (respectively). As such, our intuition for semantic maintenance is that the target-to soeet-to-souree adaptation process should retain aagguag--inarrian informa- tion while eliminating target-specific features and absorbing source-language features along the way. We employ both labeled data in the source language and unlabeled data in the target language achieve semantic preservation in this step.
90z (s k) flog P(ul =] ge (11A) [log P(d[t)]
Model
Langange
Source
EN
EN
ZH
ZH
AR
AR
Target
2H
AR
EN
AR
EN
ZH
BERFCRE BERT-CRF-LAT
68.5
30,9
37.3
20.1
20.1
58.8
70.0
33.5
41.2
20.3
37.2
55.6
BERT-CRE-FTUT
69.4
33.4
42.9
20.0
36.5
56.3
BEKT-CRF-CCCAR
72.1
42.7
45.8
20.7
40.7
59.8
XLMR-CRF
70.5
43.5
41.7
32.8
45.4
XLMR-CRF-LAT
70.2
43.4
42.3
33.2
45.2
60.9
FLMR-CRF-FTUT
71.1
43.7
42.1
32.9
45.9
62.1
XLMR-CRF CRF-COCAR
74.4
44.1
49.5
34.3
46.3
62.9
XLMR-OACLAS
74.6
44.9
45.8
35.1
48.0
63.1
RepTran (Durs)
77.7
46.6
50..6
39.3
50.8
66.2
LLas
74.1
43.7
46.2
36.6
45.4
63.8
Representation Decomposition: To disentan- gle the representations, we employ the Varia- tionl Auto-Encoder (VAE) framework (Kingma and Welling, 2014) introduce two encoders Io. (z(h) and 9o: (4) to transform a represen- tation h into stochastic dimensional spaces for language-invariant and language-specific represen- tations= = and respectively. In addition, a de- coder Po.. (hl 2, 1) is introduced to infer the repre- sentation h from z and 1. Here, the distributions go. (2]h), (4(h), and Pei. (hlz,1) are assumed be Gaussian and feed-forward neural networks with parameters ds. Φr, and on are leveraged to compute the means and variances for the distri- butions from corresponding representations. To this end, VAEs learn the parameters by minimiz- ing the negation of the variational lower bound: LVAR
76.5
45.1
48.4
37.2
-C sem. preservation
75.2
43.9
47.3
36.1
46.5
63.4
74.3
42.9
46.6
35.7
Table 1: Cross-lingual performance (Fl scores) on test data. Each column corresponds to one language pair where source languages are shown above the target languages. The proposed model is significantly better than other models with p < 0.01.
Representation Constraints: Given the represen- tation ha for word in the target language, we expect that its language-invariant features for event types in el will be preserved in the adapted repre- sentation 19 from Langevin Dynamics to enable successful ED with the source language model. To implement this idea, we first compute the language- invariant components za and æla! from h!st and its adapted version htat via: gtat ~ Lo. (2 (z(htot) and glst ~ q0. (2)h/90). Afterward, the maintenance of language-invariant information is enforced by min- imizing the L2 difference between 2/9t and 199 leading to the objective: L. = ||=4ot =/o0
Egg, (=lk)as; (1]k) [logPou (h],,1) + KL(go, (=[A)]]p())) + (1, (lp))),,))) where the first term is the reconstruction loss, p(2) and p(l) are the standard Gaussian distribution, and
In addition, to realize the target-to- tran- sition of g9 to lig we expect that the language-

---

specific component of Teat i.e. Pot 9á, (1 (19t). should be predictive for the source language. To this end, we update our model so the language pre- diction network FF can predict the source lan- guage from plot_ As such, we obtain the distribution P( from FF and minimize the likelihood loss: Li - -B 20, (((//ut) [P (source)(
lingual language models, i... mBERT-CRE and XLMR-CRF (Nguyen al., 2021), fine-tuning with unlabeled target data (FTUT), i.e., BERT- CRF-FTUT and XLMM- (Nguyen al., 2021), representation matching, i.e., BERT-CRF- CCCAR and SLMR-CRF-CCCAR (Nguyen et al., 2021), and adversarial training, i.e., BERT-CRE- LAT, XLMR-CRF- (Nguyen et al., 2021), and XLMR-OACLED (Guzman-NNatras al., 2022). As such,
Finally, the overall loss to train our model in the second step is: L = Leny LV Ldec C2 Lt.
and XLMR- OACLED represent the current state-of-the methods for CLTL for ED over ACE0S.
3 Experiments
Datasets and Hyper-parameters: Following pre- vious work on CLTL for ED M'hamdi et al., 2019; Nguyen et al., 2021), we evaluate our proposed method (called RepTTran) on the dataset ACE05 (Walker et al., 2006). This dataset contains docu- ments in three languages: English (EN), Chinese (ZH) and Arabic (AR) with annotations for 33 event types. The same data split and preprocessing as in previous work (Nguyen et al., 2021) are em- ployed for a fair comparison. For each language, the data split provides training, development, and test portions for experiments. Given the three lan- guages, consider six possible pairs of languages form the source and target languages in our cross- lingual transfer learning experiments. For each language pair, we train the models on the training data of the source language and evaluate the per- formance on the test data of the target language. Unlabeled data for the target language is obtained by removing labels from its training data portion done in previous work (Nguyen et al., 2021).
Evaluation: Table presents the cross- pooss-lngual per- formance of the models. The most important obser- vation from the table is that the proposed method RepTran is significantly better than the baselines across all six language pairs, thus clearly demon- strating the benefits of the Langevin sannevvi--baed repre- sentation transition for cross-lingual ED. In addi- tion, evaluate the contribution of the proposed se- mantic preservation, we perform an ablation study where each introduced loss (i.e., Ldees Lg and Li) is excluded from the training process. We also ex- plore the complete removal of the semantic preser- vation component. It is clear from the table that removing any of the components would hurt the performance of RepTran significantly, thus con- firming the importance of the semantic preservation techniques in our model.
Representation Visualization: To illustrate the Lengevin-based representation adaptation process from the target to the source language space in Rep- Tran, we visualize the representations of the event triggers in the target language along its adaptation process. In particular, using English and Chinese as the source and target languages, Figure 1 shows the t-SNE representation visualization for sample of target-language event triggers for the most fre- quent event types at different steps in the adaptation process. It is clear from the figure that the target language representations are gradually shifted to- ward the source language space based on Langevin Dynamics and the energy function (i.e., the trian- gles become closer and closer to the circles). Im- portantly, due to the introduction of our semantic preservation mechanism, the final states of the tar- get language representations (i.e., K 128) can also preserve their information about event types as demonstrated by the proximity of the circles and triangles of the same colors for event types.
We utilize the XLM-R base model with 768 dimensions in the hidden vectors be compa- rable with previous work (Nguyen et al., 2021; Guzman-Nateras et al., 2022). We tune the hyper- parameters for our model over the development data using the EN >ZH language pair. The same typer-parameters employed to train models for other language pairs in our experiments. In par- ticular, the following values are selected for our model: layers for the feed-forward networks with 768 dimensions for the hidden layers, 3e-5 for the AdamW optimizer, 16 for the batch size, and K = 128 for the number of update steps, < = 5 for the step size in the update rule, and σ - 0.005 for the variance in the Gaussian noise for the target- language representation adaptation with Langevin Dynamics.
Baselines: We compare our method with a vari- ety of baselines for CLTL for ED, based on multi-

---

sheps
Contt ei Ailacc
MertadiMed: job
Oentac: Mest
Sertences
22
-23
-20
20
Albut
Snd comsoreet
-20-
Conlt Ailacc
-36-
Figure T-SNE visualization for the representation transition process from Chinese event triggers (i.c., the target) the English language space (i.e., the source) based on I Langevin Dynamics and the energy function. Circles and triangles indicate representations for the English and Chinese examples respectively while colors encode event types.
4
Related Work
Conclusion
We introduce a novel method for CLTL for ED where representations of examples in the target language transferred into the source language space test time for prediction with the source lan- guage model. Our extensive experiments confirm the benefits of the proposed method. the future, we plan to extend our method to other tasks IE.
A large body of previous ED research is dedicated to monolingual learning, i.c., training and testing over the same languages (Nguyen et al., 2016; Yang and Mitchell, 2016; Lu and Nguyen, 2018; Lai al., 2020; Lin et al., 2020). The models might consider different domains (Nguyen and Nguyen, 2018; Man Duc Trong et al., 2020). To address the language barriers, recent work has studied CLTL for ED, focusing on mapping representations for different languages into the same space for transfer learning, e.g., using bilingual resources (Muis et al., 2018; Liu et al., 2019), multilingual language mod- els (M' ha'mamii et al., 2019), and adversarial train- ing (Guzman-Nateral Nateras et al., 2022). Our work is also related to previous work on data projection for ED and IE where texts in the target language are translated and aligned into the source language for prediction at test time (Riloff et al., 2002; Yarmo- hammadi et al., 2021; Yu et al., 2023). However, data projection requires high-quality machine trans- lation and alignment systems, which might be available for different languages. Our method is different as we project representations (not actual texts) the target to the source language space to avoid translation systems.
Acknowledgement
This research has been supported by the Army Re- search Office (ARO) grant W911NF-2111-0112 12. the NSF grant CNS-1747798 the IUCRC Center for Big Learning, and the NSF grant # 2239570. This research also supported in part by the Office of the Director National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract 2022- 22072200003. The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and dis- tribute reprints for governmental purposes notwith- standing any copyright annotation therein.

---

Limitations
Nguyen. 2023. Hybrid knowledge transfer for im- proved cross- -onsu-l event detection via hierarchical sample selection. In Proceedings of the st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 5414-5427, Toronto, Canada. Association for Computational Lin- guistics.
this work, we develop a novel cross-lingual transfer learning method for event detection that achieves statc-of-the-art performance on bench- mark datasets for this problem. However, our work still has several limitations that can be further ex- plored in future work. First, our RepTran method leverages Langevin Dynamics perform represen- tation transition for target language examples at in- ference time to achieve knowledge transfer. While our experiments demonstrate the superior perfor- mance of this approach for cross-lingual ED, our method does involve an additional computational cost at inference time for representation transition. In Appendix A, we show that this additional cost is not substantial and it can be acceptable given the demonstrated benefits. Nonetheless. we believe that future work can explore this direction further to reduce computational costs at inference time for RepTran to improve its efficiency and applica- tions. Second, our method relies on the multilin- gual pre- pre-trrined language model XLMR to encode text in different languages. Although XLMR can support more than 100 languages, the world has more than 7000 languages and our method cannot be applied directly to many other languages. Ex- tending the language set of pre-trained language models is an important direction that can improve our method. Finally, to perform semantic preserva- tion, our methods need to employ unlabeled data in the target language. While unlabeled data can be more available for many languages, it might still be challenging retrieve text for extremely low-resource languages. As such, future research can explore methods relax this requirement of unlabeled data for the target language to maximize the applicability of cross-lingual transfer learning methods for ED.
Luis
Minh Van Nguyen, and Thien Nguyen. 2022. Cross-lingual event detection via optimized adversarial training. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5588-5599, Seattle, United States. Association for Computational Linguistics.
Diederik Kingma and Max Welling. 2014. Auto- encoding variational bayes. In Proceedings of the International Conference on Learning Representa- tions (ICLR).
Viet Dac Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2020. Exploiting the matching information in the support set for few shot event classification. In Proceedings of the 24th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD).
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu Jie Huang. 2006. Energy-based models. In Predicting structured data, MIT.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, pages 1909, Online. Association for Computational Linguistics.
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction as machine reading com- prehension. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1641-1651, Online. Association for Computational Linguistics.
Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2019. Neural cross-lingual event detection with minimal parallel resources. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP- IJCNLP). pages 738-748 Hong Kong, China. As- sociation for Computational Linguistics.
References
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In CoRR.
Shulin Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2017. Exploiting argument information to improve event detection via supervised attention mechanisms. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Pupers), pages 1789-1798. Vancouver, Canada. Association for Computational Linguistics.
Yilun Du and Igor Mordatch. 2019, Implicit generation and modeling with energy based models. In Proceed- ings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
Luis Guzman Nateras, Franck Dernoncourt, and Thien

---

Weiyi Lu and Thien Huu Nguyen. 2018. Similar but not the same: Word sense disambiguation improves event detection via neural representation matching. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4822 4822, Brussels, Belgium. Association for Com- putational Linguistics.
neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guisties and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 365-371, Beijing, China. Association for Computational Linguistics.
Ellen Riloff, Charles Schafer, and David Yarowsky. 2002. Inducing information extraction systems for new languages via cross-language projection. In COLING 2002: The 19th International Conference on Computational Linguistics.
Hieu Man Due Trong, Duc Trong Le, Amir Pouran Ben Veyseh, Thuat Nguyen, and Thien Huu Nguyen. 2020. Introducing a new dataset for event detec- tion in cybersecurity texts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5381-5390, Online. Association for Computational Linguisties.
Yang Song and Diederik P. Kingma. 2021. How to train your energy- -aaeddd models. ArXiv, 0tss/2011.32280
Meryem M' d'hamdi, Marjorie Freedman, and Jonathan May. 2019. Contextualized cross- lingual event trig- ger extraction with minimal resources. In Proceed- ings of the 23rd Conference on Computational Nat- ural Language Learning (CoNLL), pages 656-665, Hong Kong, China. Association for Computational Linguistics.
Minh Tran Phu and Thien Huu Nguyen. 2021. Graph convolutional networks for event causality identifi- cation with rich document-level structures. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3480-3490, Online. Association for Computational Linguistics.
Aldrian Obaja Muis, Naoki Otami, Nidhi Vyas, Ruochen Xu, Yiming Yang, Teruko Mitamura, and Eduard Hovy. 2018. Low-resource cross-lingual event. type detection via distant supervision with minimal effort. In Proceedings of the 27th International Conference on Computational Linguistics, pages 70-82, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. In Technical report, Linguistic Data Consortium.
Max Welling and Yee Whye Teh. 2011. Bayesian learn ing via stochastic gradient langevin dynamics. Proceedings of the International Conference on Ma- chine Learning (ICML).
Chien Nguyen, Linh Ngo, and Thien Nguyen. 2023. Re-
trieving relevant context to align representations for
cross-lingual event detection. In Findings of the As- sociation for Computational Linguistics: ACI. 2023, pages 2157-2170, Toronto, Canada. Association for Computational Linguistics.
Bishan Yang and Tom M. Mitchell. 2016. Joint extrac- tion of events and entities within a document context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 289-299, San Diego California. Association for Computational Linguistics.
Minh Nguyen and Thien Huu Nguyen. 2018. Who is killed by police: Introducing supervised attention for hierarchical LSTMs. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 2277-2287, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Sen Yang. Dawei Feng. Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019. Exploring pre-trained language models for event extraction and generation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5284- 5294, Florence, Italy. Association for Computational Linguistics.
Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 54 1414-54,6, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mahsa Yarnohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, and Benjamin Van Durme. 2021. Everything is all it takes: A multi- pronged strategy for zero-shot cross-lingual informa- tion extraction. In Proceedings of the 202/ Confer- ence Empirical Methods in Natural Language Pro- cessing, pages 1950-1967, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grish- man. 2016. Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300-309, San Diego, Califomia. Association for Computational Linguistics.
Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional
Pengfei Yu, Jonathan May, and Heng Ji. 2023. Bridg- ing the gap between native text and translated text

---

through adversarial learning: case study on cross- lingual event extraction. In Findings of the Associ- ation for Computational Linguistics: EACL, 2023, pages 754-769, Dubrovmik, Croatia, Association for Computational Linguistics.

---

A Speed Inference Evaluation
Our Langevein- based representation transition method needs to adapt each target language ex- ample at inference time into the source language space to allow event prediction with the source lan- guage ED model. As such, compared to previous cross-lingual ED method, in addition to the encod- ing cost for target language example at inference time, our method needs perform an additional step to transition the representations into the source language space based on the energy function. This implies an additional computational cost for pre- dicting each target language example.
Model
Inference Time
FLMR--CRF
1.00x
Rean (K = 64)
1.15x
RepTran (K = 128)
1.24x
Table 2: Latency cost for our Rep model. All results computed with single NVIDIA VI00 GPU.
To measure the impact of this additional cost for the inference time method, Table compares the inference time of the baseline XLMR-CRI and our model Rep over test data. Using the in- ference time of XLMR-CRF as the reference, we show the inference time for adaptation steps at both K = 64 and K 128 (the final step). Although the representation transition does increase the in- ference time for RepTran, the additional cost is not substantial and be acceptable to achieve greater cross-lingual performance for ED.


Reviews:
[{"review_id": "JHDCDp6oSe", "reviewer": "Reviewer_xAXc", "summary": "", "questions": "1. Whether the proposed method can also work on other cross-lingual tasks such as tasks in the XTREME benchmark?\n\n2. In Line 356, It is true that “data project requires high-quality machine translation” and have you compared your methods with any translation-test baselines?  \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"It seems that the proposed solution can not well support the motivation. The general idea is to disentangle the language-specific and task-specific features from the representations. It is not clearly stated or explained why the proposed solution can achieve this while other alignment based methods fail. It is suggested to justify some insights upon this.\n\"*\n\n**Our Response**:\n\nThanks for your comment. However, we would emphasize that our main motivation is to transition the representations from the target-language space to the source-language space using Langevin Dynamics for event detection. The decomposition of representations into language-invariant and language-specific features is just a component of our representation transition proposal to facilitate semantic presentation during the transition process. This is juts in contrast to previous cross-lingual models for event detection, e.g. adversarial training, that focus on learning a universal representation space for the source and target languages by eliminating language-specific features to align the source- and target-language representations. As discussed in our paper, the representation alignment in previous method might also remove discriminative features from the source-language representations for event prediction, thus hurting the performance. Our representation transition method does not suffer from this issue as we preserve the representation space for the source languages (i.e., we only shift the target-language representations), thus maintaining discriminative features for event detection.\n\n**Reviewer**: *\"Whether the proposed method can also work on other cross-lingual tasks such as tasks in the XTREME benchmark?\"*\n\n**Our Response**:\n\nYes, our representation transition method can also be applied to other cross-lingual tasks, including those in the xTREME benchmark. It has the potentials to preserve discriminative features for the tasks to boost cross-lingual performance. Our current work focuses on cross-lingual event detection and we will explore other tasks in future work.\n\n**Reviewer**: *\"In Line 356, It is true that “data project requires high-quality machine translation” and have you compared your methods with any translation-test baselines?\"*\n\n**Our Response**:\n\nOur intuition is that the translation-based methods are not comparable to our representation transition model as the transition-based methods would require translation and text alignment systems while our method does not need those resources. A comparison will be not fair in this case. In addition, for translation-based methods, it is expected that the translation and alignment systems should have high quality to avoid noise for the data, which will introduce another challenge for this method and limit their applications to many other languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. small and focused contribution of improving zero-shot cross-lingual event detection.\n\n2. Clear writing and good representation. \n\n3. Achieve state-of-the-art performance \n", "reasons_to_reject": "It seems that the proposed solution can not well support the motivation. The general idea is to disentangle the language-specific and task-specific features from the representations. It is not clearly stated or explained why the proposed solution can achieve this while other alignment based methods fail. It is suggested to justify some insights upon this. \n", "questions_to_authors": "1. Whether the proposed method can also work on other cross-lingual tasks such as tasks in the XTREME benchmark?\n\n2. In Line 356, It is true that “data project requires high-quality machine translation” and have you compared your methods with any translation-test baselines?  \n", "paper_topic_and_main_contributions": "This paper proposes a representation transition method for zero-shot cross-lingual event detection tasks. The goal is to align representation spaces among languages and also preserve the discriminative information. Specifically, the Langevin Dynamics is leveraged to perform representations for the target language. Experimental results demonstrate the effectiveness of three languages. Also the authors claim that this work is the first to explore energy functions in the information extraction and event detection. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UaMDEz1pKE", "reviewer": "Reviewer_dUHy", "summary": "", "questions": "A.Will the framework can also work on other CLTL problems, such as cross-lingual NER? \n\nB.Since Large Language Models (LLM) can achieve great performance in cross-lingual transfer learning, will they also outperform in CLED?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"1)... However, the presented problem and solution may be questioned. First, prior works typically employ joint optimization for both event detection and representation matching, leading to language-invariant yet event-related representations for ED. Second, sacrificing language-specific information may not be a concern, as the event types in both source and target language datasets are identical. Finally, the paper employs representation transition for Cross-Language Event Detection (CLED). This approach only modifies the representations of the target language, potentially leading to information loss. Unfortunately, the paper does not explicitly compare the pros and cons of representation matching and representation transition.\"*\n\n**Our Response**:\n\nThank you for your comment. We would like to clarify that while the event types remains consistent between the source and target languages, the linguistic discrepancies give rise to the differences in the representations for the source and target languages as we visualize in Figure 1 of the Appendix. That’s why previous representation matching methods aim to learn a shared representation space for the source and target languages by removing the language-specific features. In our method, we argue that language-specific feature removal might also lead to the **elimination of discriminative features** for event prediction learned from the source-language representations, thus hurting the performance of the systems. As such, our method proposes to avoid language-specific feature removal from the source-language representations by preserving this representation space for the model. Instead, we propose to explicitly transition the representations from the target language space into the source language space to perform event prediction. Our experiment results show that representation transition can produce state-of-the-art performance for cross-lingual event detection over different language pairs, thus testifying to the benefits of our methods and argument.\n\nRegarding the potential information loss due to representation transition, we would like to note that our method introduces a semantic preservation component to explicitly address issue (L195-275). Using representation decomposition with the Variational Auto-Encoder framework, our model seeks to preserve the event type information during the representation transition process to the source language space, thus avoiding information loss for event detection. Our ablation study and performance comparison in Section 3 demonstrates the effectiveness of this component, leading to the state-of-the-art performance for our method.\n\nFinally, as discussed in our Limitation section, compared to the representation matching methods, our representation transition method might require more time during the inference time. In Appendix C, we evaluate the inference time for our methods, showing that the inference time does increase. However, the increase is not substantial and can be acceptable given the demonstrated performance. We will clarify those details in the final version.\n\n**Reviewer**: *\"2)The method proposed in this paper lacks sufficient justification. Firstly, the paper uses an energy function, specifically Langevin Dynamics, for representation transition. However, the paper does not explicitly explain how the energy function impacts representation transition, nor does it clarify why Langevin Dynamics is chosen over other energy functions. Additionally, the techniques employed for semantic preservation are similar to representation matching, making it challenging to comprehend the paper's core motivation and leading to a perception of insufficient innovation.\"*\n\n**Our Response**:\n\nThank you for your comment. We would like to clarify the role of the energy function in our approach. First, the energy function helps estimate the probability distribution over the source language representations, which is essential to enable our representation transition method. Second, the energy function can be viewed as an implicit generator, enabling us to utilize the Langevin Dynamics algorithm for the purpose of sampling from the distribution to perform representation transition. As such, it is important to note that **Langevin Dynamics is a typical Markov Chain Monte Carlo algorithm (not an energy function)** to generate samples from an initialization. This sample geenration process is leveraged for the representation transition from the target to the source language space in our method. Exploring other data sampling methods (i.e., rather than Langevin Dynamics) can improve our performance further. However, in this short paper, we aim to demonstrate the benefit of our representation transition approach even using a typical sampling method, leaving it for future work to devise better sampling methods for the framework.\n\nFinally, we would like to note some key differences between our semantic presentation component and the representation matching methods. First, our semantic presentation is embedded in a larger framework of representation transition, which is fundamentally different from the representation matching methods. Second, our semantic representation component aims to preserve the event type information in the target language space during the representation transition to ensure correct type prediction with the source language model. The source language representation space learned from training data is preserved in our method. In contrast, the representation matching methods aim to eliminate language-specific features from both source and target spaces to achieve language-universal representations. As such, these methods modify the source language representation space learned from training data, which might cause loss of discriminative features for event detection.\n\n**Reviewer**: *\"3)The experimental evaluation in this paper is also inadequate. Firstly, the paper only evaluates the model on the ACE 2005 dataset, focusing on a limited number of language pairs. It disregards other CLED datasets such as ERE[1] and MINION[2], which involve a broader range of language pairs and etymological combinations. Moreover, the ablation study falls short, as it only explores semantic preservation while neglecting the critical aspects of the model: the energy model and Langevin Dynamics, which are essential components deserving of more thorough examination.\"*\n\n**Our Response**:\n\nThank you for your comment. We have already performed evaluation for our method over the MINION dataset that provides data for 8 languages English (EN), Spanish (ES), Portuguese (PT), Polish (PL), Turkish (TR), Hindi (HI), Japanese (JP), and Korean (KO). Following the same setting in the original paper of MINION [1], we compare our method RepTran with the state-of-the-art models for MINION in [1], i.e., BERT-CRF and XLMR-CRF, over the cross-lingual performance from English to 7 other languages. Please find our results below:\n\n| Model | EN->ES | EN->PT | EN->PL | EN->TR | EN->HI | EN->JP | EN->KO |\n| :----------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| BERT-CRF | 62.6 | 71.1 | 59.6 | 46.6 | 58.0 | 34.1 | 55.6 |\n| XLMR-CRF | 62.8 | 72.8 | 60.1 | 47.2 | 58.2 | 35.1 | 56.8 |\n|RepTran (ours) | 65.4 | 75.5 | 62.5 | 49.6 | 59.1 | 38.9 | 59.9 |\n\nAs can be seen, our method significantly outperforms the current state-of-the-art performance over MINION, thus further demonstrating its benefits. We will provide this results in the final version.\n\nFor the ablation study, we would like to note that the energy model and Langevin Dynamics are critical elements of our representation transition method, and ablating any of them would turn our model back to an simple XLMR model as we cannot obtain any intermediate representations for the transition process and semantic preservation. As the performance of the simple XLMR model (with just the XLMR encoder and a classification head) has been shown to be worse than the current state-of-the-art models, we don't include them in the ablation study. We will clarify this detail in the final version.\n\n**Reviewer**: *\"A.Will the framework can also work on other CLTL problems, such as cross-lingual NER? \"*\n\n**Our Response**:\n\nYes, as mentioned in our response to reviewer 1, our representation transition method can also be applied to other cross-lingual transfer learning problems, including the cross-lingual NER task. It has the potentials to preserve discriminative features for the tasks to boost cross-lingual performance. Our current work focuses on cross-lingual event detection and leave the other tasks for future work.\n\n**Reviewer**: *\"B.Since Large Language Models (LLM) can achieve great performance in cross-lingual transfer learning, will they also outperform in CLED?\"*\n\n**Our Response**:\n\nThank you for your question. From our understanding, it has actually been shown in some recent work that LLMs, such as ChatGPT, cannot achieve very good performance for event detection and span detection tasks (e.g., named entity recognition) in different languages, e.g., in [2] and [3]. These studies show that ChatGPT's performance is still significantly worse than the state-of-the-art supervised learning models for these tasks over different languages with large performance gaps. The main issue might stem from the misalignment between nature of the next word prediction/text generation tasks in the LLMs and the sequence labeling tasks we are deadling in event and span detection. We believe further exploration of LLMs to overcome such misalignments can lead to promising advancement for this area.\n\n[1] Veyseh, Amir Pouran Ben, et al. \"MINION: a Large-Scale and Diverse Dataset for Multilingual Event Detection.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n[2] Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, Shikun Zhang. Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness: https://arxiv.org/pdf/2304.11633.pdf\n\n[3] Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, Thien Huu Nguyen. ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning: https://arxiv.org/pdf/2304.05613.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper leverage Langevin Dynamics for cross-lingual representation transition. This is the first work to explore energy functions and Langevin Dynamics for CLTL in IE and ED.\n\n2) A semantic preservation mechanism is used for the target language examples from Langevin Dynamics to retain event features.\n\n3) Extensive evaluations over three languages demonstrate the paper’s SOTA performance for CLED.", "reasons_to_reject": "1)The motivation of this paper is not clearly stated. The purpose of this paper is to address issues in previous works where they achieve matching representations but lose discriminative features, potentially leading to impaired performance in Event Detection (ED). However, the presented problem and solution may be questioned. First, prior works typically employ joint optimization for both event detection and representation matching, leading to language-invariant yet event-related representations for ED. Second, sacrificing language-specific information may not be a concern, as the event types in both source and target language datasets are identical. Finally, the paper employs representation transition for Cross-Language Event Detection (CLED). This approach only modifies the representations of the target language, potentially leading to information loss. Unfortunately, the paper does not explicitly compare the pros and cons of representation matching and representation transition.\n\n2)The method proposed in this paper lacks sufficient justification. Firstly, the paper uses an energy function, specifically Langevin Dynamics, for representation transition. However, the paper does not explicitly explain how the energy function impacts representation transition, nor does it clarify why Langevin Dynamics is chosen over other energy functions. Additionally, the techniques employed for semantic preservation are similar to representation matching, making it challenging to comprehend the paper's core motivation and leading to a perception of insufficient innovation.\n\n3)The experimental evaluation in this paper is also inadequate. Firstly, the paper only evaluates the model on the ACE 2005 dataset, focusing on a limited number of language pairs. It disregards other CLED datasets such as ERE[1] and MINION[2], which involve a broader range of language pairs and etymological combinations. Moreover, the ablation study falls short, as it only explores semantic preservation while neglecting the critical aspects of the model: the energy model and Langevin Dynamics, which are essential components deserving of more thorough examination.\n\n[1] Song, Zhiyi, et al. \"From light to rich ere: annotation of entities, relations, and events.\" Proceedings of the the 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation. 2015.\n\n[2] Veyseh, Amir Pouran Ben, et al. \"MINION: a Large-Scale and Diverse Dataset for Multilingual Event Detection.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.", "questions_to_authors": "A.Will the framework can also work on other CLTL problems, such as cross-lingual NER? \n\nB.Since Large Language Models (LLM) can achieve great performance in cross-lingual transfer learning, will they also outperform in CLED?", "paper_topic_and_main_contributions": "The topic of this paper is to develop models for cross-lingual transfor learning for event detection. Previous works focused on representation matching, but they modified the representations for source language examples, which might lose discriminative features for ED. To address this problem, the paper utilize Langevin Dynamics for cross-lingual representation transition and a semantic preservation framework to retain event features, and achieve SOTA performance in CLED benchmaks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eKYj5cHKPt", "reviewer": "Reviewer_9TYZ", "summary": "", "questions": "1. How does the proposed method compare with existing methods in terms of computational efficiency, especially considering the additional computational cost at inference time for representation transition?\n2. Could you elaborate on the limitations of the results? Were there any surprising findings or anomalies in the results that were not discussed in the paper?\n3. Could you provide more insights into why the proposed method outperforms existing methods? What specific features or aspects of the method contribute to its superior performance?\n4. Given the complexity of the proposed method, how do you see its practical application, especially in real-world scenarios where computational resources might be limited?\n7. Are there any specific future research directions you suggest based on the findings of this paper?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"The authors do not provide a thorough comparison with existing methods. This leaves the reader unsure of how the proposed method stands against current state-of-the-art methods.\"*\n\n*\"The results section lacks a detailed analysis. The authors do not discuss any surprising findings, limitations of the results, or how their results compare to the existing state-of-the-art.\"*\n\n**Our Response**:\n\nThank you for your comments. However, we would like to emphasize that in Section 3 of the paper, we have compared our work with the current state-of-the-art methods for cross-lingual event detection by the time of work for this paper, including XLMR-CRF-CCCAR [1] and  XLMR-OACLED [2]. We hope the reviewer can reconsider this point for us.\n\nRegarding the analysis, we would like to note that we have provided a visualization analysis in Appendix B to demonstrate the transitions of the representations from the target language space to the source language space for our model for cross-lingual event detection. Our analysis shows that the target language representations are gradually shifted toward the source language space based on Langevin Dynamics and the energy function. Importantly, due to the introduction of our semantic preservation mechanism, the final states of the target language representations can also preserve their information about event types that allows the succesful application of the source language to predict event types for the target languages in our method. In addition, in Appendix C and the Limitation Section, we have discussed the limitation of our method regarding additional cost for inference time. Our experiments in Appendix C shows that although our method require additional inference time, this increase is not substantial and it can be accepted given the demonstrated performance benefits.\n\n[1] Nguyen et al., 2021. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In EMNLP 2021.\n\n[2] Guzman-Nateras, et al., 2022. Cross-lingual event detection via optimized adversarial training. In NAACL 2022.\n\n**Reviewer**: *\"The proposed method seems overly complex and less generalized compared to current methods based on large language models. The necessity and practicality of this approach in the era of large models are not convincingly demonstrated.\"*\n\n**Our Response**:\n\nThank you for your comment. However, as mentioned in our response for reviewer 2, it has actually been shown in some recent work that despite their strong text generation performance, large language models (LLMs), such as ChatGPT, cannot achieve very good performance for event detection and span detection tasks (e.g., named entity recognition) in different languages, e.g., in [3] and [4]. These evaluations show that ChatGPT's performance still significantly falls short of the state-of-the-art supervised learning models for these tasks over different languages with large performance gaps. The main issue might stem from the misalignment between nature of the next word prediction/text generation tasks in the LLMs and the sequence labeling tasks we are deadling in event and span detection. As such, we believe dedicated models for cross-lingual event detection, such as our current work, are still necessary to ensure strong performance for applications. These dedicated models are also more practical given that they don't need gigantic scale to perform well and they can be hosted internally. This will avoid data privacy and security risks as we don't need to send data to external LLMs for processing.\n\n[3] Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, Shikun Zhang. Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness: https://arxiv.org/pdf/2304.11633.pdf\n\n[4] Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, Thien Huu Nguyen. ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning: https://arxiv.org/pdf/2304.05613.pdf\n\n**Reviewer**: *\"1. How does the proposed method compare with existing methods in terms of computational efficiency, especially considering the additional computational cost at inference time for representation transition?\"*\n\n**Our Response**:\n\nWe would like to note that we have provided an analysis to compare the inference time for our method and previous cross-lingual event detection methods in Appendix C. The results show that our method needs additional time due to the representation transition, but it's not substantial and can be acceptable to obtain strong prediction performance.\n\n**Reviewer**: *\"2. Could you elaborate on the limitations of the results? Were there any surprising findings or anomalies in the results that were not discussed in the paper?\"*\n\n**Our Response**:\n\nOur extensive experiments demonstrate the benefits of our method for cross-lingual event detection, achieving state-of-the-art performance for different pairs of source and target languages compared to previous work. The ablation study further highlights the necessity and contribution of each component in our representation transition model. The performance improvement is consistent across different languages, and we do not observe any anomalies in the development and performance for our model.\n\n**Reviewer**: *\"3. Could you provide more insights into why the proposed method outperforms existing methods? What specific features or aspects of the method contribute to its superior performance?\"*\n\n**Our Response**:\n\nWe would like to emphasize that we have discussed the limitations of previous work for cross-lingual event detection to motivate the design for our method in this work in L59-87. Essentially, the main approach of the existing methods for cross-lingual event detection is to perform representation matching to transform the representations for the source and target languages into a common space to train language-general ED models. However, to achieve a common space, the source-language representations might need to sacrifice language-specific information, which can involve discriminative features, causing performance degradation for event detection. To address this issue, we introduce a new method for cross-lingual event detection based on representation transition. Instead of transforming the source-language representations, which might eliminate discriminative features for ED, we only seek to transition the target-language representations into the source space whose results can be directly fed into the source-language models for event prediction. In this way, we can preserve the source-language representations with discriminative features learned over training data for ED to maximize prediction ability.\n\n**Reviewer**: *\"Given the complexity of the proposed method, how do you see its practical application, especially in real-world scenarios where computational resources might be limited?\"*\n\n**Our Response**:\n\nAs demonstrated in Appendix C and acknowledged in our Limitation Section, although our method does require more inference time than previous methods for cross-lingual event detection, our analysis demonstrates the additional time is not substantial and it can be accepted for pratical applications to achieve state-of-the-art performance.\n\n**Reviewer**: *\"Are there any specific future research directions you suggest based on the findings of this paper?\"*\n\n**Our Response**:\n\nAs discussed in the Limitation Section of our paper (L372-348), given the additional inference time imposed by our method, future work can explore more efficient approaches for representation transition, aiming to reduce the sampling time during the transition process to improve the efficiency. In addition, future research can explore larger set of languages to obtain better understanding for the generalization of the methods over different languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper attempts to address a significant problem in the field of cross-lingual event detection.\n2. The authors provide a novel approach to the problem, which could potentially contribute to the field.", "reasons_to_reject": "1. The authors do not provide a thorough comparison with existing methods. This leaves the reader unsure of how the proposed method stands against current state-of-the-art methods.\n2. The results section lacks a detailed analysis. The authors do not discuss any surprising findings, limitations of the results, or how their results compare to the existing state-of-the-art.\n3. The proposed method seems overly complex and less generalized compared to current methods based on large language models. The necessity and practicality of this approach in the era of large models are not convincingly demonstrated.", "questions_to_authors": "1. How does the proposed method compare with existing methods in terms of computational efficiency, especially considering the additional computational cost at inference time for representation transition?\n2. Could you elaborate on the limitations of the results? Were there any surprising findings or anomalies in the results that were not discussed in the paper?\n3. Could you provide more insights into why the proposed method outperforms existing methods? What specific features or aspects of the method contribute to its superior performance?\n4. Given the complexity of the proposed method, how do you see its practical application, especially in real-world scenarios where computational resources might be limited?\n7. Are there any specific future research directions you suggest based on the findings of this paper?", "paper_topic_and_main_contributions": "The paper, \"Transitioning Representations between Languages for Cross-lingual Event Detection via Langevin Dynamics\", introduces a new method, RepTran, for cross-lingual event detection. The authors claim that their method outperforms existing ones on the ACE05 dataset. However, the paper lacks a comprehensive analysis of the methods used and the experimental results, raising questions about the validity and applicability of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vforDrWt6H", "reviewer": "Reviewer_87k9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"... the proposed model relies on pre-trained multilingual language model to generate the representations for target language and source language. The representations of different languages generated by XLM-R are already projected into a common space. Langevin Dynamics plays a role of matching the representation distributions of the target language data and the source language data. Thus some Generative Adversarial Networks related work should be mentioned and included as baseline.\"*\n\n**Our Response**:\n\nThank you for your comment. However, we would like to note that although XLM-R could project the representations of the source and target languages into a common space, there are still significant gaps between source- and target-language representations as demonstrated in Figure 1 of our Appendix B. Such representation gaps can cause significant challenges for cross-lingual knowledge transfer, necessitating additional alignment steps to bridge the gaps between the source and target representations to boost the performance. This performance gap issue from XLM-R representations for source and target languages have also been observed in previous cross-lingual work in this area, e.g., [1].\n\nTo achieve the representation alignment for the source and target languages, we propose a novel representation transition approach that aims to transition the representations from the target languages into the source language space, thus preserving the source-language representations and their discriminative features to improve event detection. As such, we employ Langevin Dynamics and an energy function whose representation sampling can be used to implement the target-to-source representation transition process. We agree with the reviewer that Generative Adversarial Networks (GANs) are relevant to our approach as they also aim to match the representation distributions. However, we would like to highlight a crucial distinction between our method and adversarial-based approaches. Our method is designed to preserve the source-language representations and their language-specific and discriminative features learned over training data, which can be valuable for the event detection task. In contrast, adversarial training, when applied to cross-lingual transfer learning for event detection, will perform elimination of language-specific features in both the source and target language representations to achieve a shared space for transfer leraning. As discussed in our introduction (L69-76), removing language-specific features might also lead to the exclusion of discriminative features for event detection, thus hindering the performance of the methods. \n\nFurthermore, we would like to clarify that we have already provided a comprehensive comparison of our model with a range of adversarial-based methods, such as the *-LAT, *-CCCAR, and *-OACLED models in Table 1. These results demonstrate strong performance for our representation transition method over adversarial learning models, thus confirming the effectiveness of our model for cross-lingual event detection.\n\n[1] Nguyen et al., 2021. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In EMNLP 2021.\n\n**Reviewer**: *\"The Semantic Preservation components proposed in the paper can also be incorporated into other models. The proposed model does not significantly outperforms baselines with the semantic preservation components removed.\"*\n\n**Our Response**:\n\nThank you for your comment. We agree with the reviewer that the Semantic Preservation component is critical for the performance of our model as demonstrated in our ablation study. However, we would like to emphasize that this Semantic Preservation component is inherent to our model as it is specially designed for the representation transition approach, which involves a sequence of representations sampled with Langevin Dynamics to transition a representation vector in the target space to the source space. As such, our representation transition method involves an initial representation computed over an example in the target language with XLMR and a final state representation from Langevin Dynamics to serve as the adapted representation to the source space. The representation contraint is then introduced to ensure that the event type information in these initial and adapted representations of the target examples are preserved in our model. To this end, we believe the Semantic Preservation component cannot be applied directly to previous cross-lingual methods for event detection as they don't involve any the representation transition process with initial and adapted representations, which are necessary to perform event type information preservation. We hope the reviewer can reconsider this point for us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The conducted experiments demonstrate the effectiveness of the proposed model.\n2. The ideas can be adapted to other cross-lingual tasks.", "reasons_to_reject": "1. The motivation is not very convincing. The paper claims that it aims to transition the representations for the target-language examples into the source-language space. However the proposed model relies on pre-trained multilingual language model to generate the representations for target language and source language. The representations of different languages generated by XLM-R are already projected into a common space. Langevin Dynamics plays a role of matching the representation distributions of the target language data and the source language data. Thus some Generative Adversarial Networks related work should be mentioned and included as baseline.\n2. The Semantic Preservation components proposed in the paper can also be incorporated into other models. The proposed model does not significantly outperforms baselines with the semantic preservation components removed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a model for cross-lingual event detection, which relies on pre-trained multilingual language model to generate representations for source and target languages, and then utilizes Langevin Dynamics to transition the representations of the target language to source language. The paper also proposes several components to restrict the transitions.\n\nThe main contribution in the paper is the proposal of using the Langevin Dynamics to transition the representations of the target language to source language. Combined with several other methods, the proposed model outperforms baselines on ACE05 datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes method for cross-lingual event detection. The method has promise for solving the cross-lingual event detection task, performing zero-shot event detection; experiments suggest improvement over the state-of-the-art models. 
It needs a refinement with respect to the following: generalizability of the method for the other datasets; comparisons with more potential techniques; and a more clearly stated motivation.
Decision: Accept-Findings