Learning Event-aware Measures for Event Coreference Resolution
Year: 2023

Abstract:
Researchers are witnessing knowledge-inspired natural language processing shifts the focus from entity-level to event-level, whereas event coreference resolution is one of the core challenges. This paper proposes a novel model for within-document event coreference resolution. On the basis of event but not entity as before, our model learns and integrates multiple representations from both event alone and event pair. For the former, we introduce multiple linguistics-motivated event alone features for more discriminative event representations. For the latter, we consider multiple similarity measures to capture the distinctions of event pairs. Our proposed model achieves new state-of-theart on the ACE 2005 benchmark, demonstrating the effectiveness of our proposed framework.

heading: 1 Introduction
text: Knowledge-inspired natural language processing (NLP) may be generally conducted on the basis of entities. However, researchers are realizing that it is the form of "event" that can more comprehensively depict the knowledge clues in NLP tasks (Lee et al., 2012), among which coreference resolution is a fundamental task for either entity or event. Within-document event coreference resolution aims at finding all event mentions that refer to the same real-world event in a document (Lu and Ng, 2018).
Figure 1 shows an example of event coreference resolution from ACE 2005 (Walker et al., 2006) dataset. In the event coreference resolution, the trigger of an event mention usually refers to the word or phrase that describes the event with the most clarity and the event mention is typically the sentence containing the given trigger. As shown
∗Corresponding author. † Equal contribution. This work was supported by the Key Projects of National Natural Science Foundation of China (U1836222 and 61733011) and the Fundamental Research Funds for the Central Universities (No. 2042023kf0133).
in the given example of Figure 1, the word taken triggers the Transport event. The words wedding and marry trigger the Marry event and these event mentions are coreferent because they both refer to the same real-world Marry event.
Compared with entity-level coreference resolution, event coreference resolution is more challenging despite their similarity (Lu and Ng, 2018). The main reasons are: (1) an event contains more complex syntactic information than an entity; (2) event coreference resolution suffers from error propagation since it has more upstream tasks; (3) there are far less triggers than entities in a document. Therefore, simply applying entity coreference resolution to event coreference resolution is unsatisfactory.
Many works utilize event linguistic features to address above challenges. In most cases, as shown in the above example, events that are coreferent have the same event features, such as type, polarity, and modality. Therefore, many studies (Lu and Ng, 2017b; Yu et al., 2020a; Lai et al., 2021; Lu et al., 2022) have suggested that incorporating event features into event coreference resolution is effective.
Despite of the developing trend of utilizing event features, the semantic similarity measures for event coreference resolution are less studied. Most of the works simply used element-wise multiplication to measure the similarity between two spans (Lee et al., 2017; Li et al., 2018, 2021) for entity-level
13542
NLP tasks. However, incorporating multiple similarity measures interestingly shows helpful in many NLP tasks other than our concerned coreference resolution. For instance, He and Lin (2016) proposed a deep pairwise word interaction model to measure semantic textual similarity between two text pieces, which directly utilized multiple similarity measures for answer selection task and semantic textual similarity measuring task. Liu et al. (2020) designed a U-shaped rewritten network to incorporate multiple similarity measures for context rewriting task.
Compared to entity-level coreference resolution, event-level coreference resolution focuses more on the cross-sentence semantic relationships, which are more complex and flexible. Therefore, to better address the event coreference resolution problem, we need to measure the relationships between events in more diverse and comprehensive ways. Inspired by this, in this paper, we propose a novel model for within-document event coreference resolution which integrates multiple measures from both event alone and event pair representations.
As far as we know, we are the first to introduce more comprehensive and distinguishing measures for event-level coreference resolution. Concretely, we introduce multiple linguistics-motivated event alone features for event alone representation and multiple similarity measures to capture the distinction of event pair. By conducting experiments on ACE-2005 benchmark, our proposed model shows a significant improvement and achieves a new stateof-the-art F1 score of 61.71% in end-to-end settings and 92.32% in all gold settings. Results demonstrate that our model can effectively models the event along features and event pair features, thus surpassing our baseline model and previous stateof-the-art models by large margins.

heading: 2 Related Work
text: Entities serve as the fundamental basis for conducting various knowledge-inspired natural language processing (NLP) tasks (Li et al., 2019, 2022; Yang et al., 2022; Stylianou and Vlahavas, 2021). Event coreference resolution, as an integral sub-task of natural language understanding, has gained increasing attention and finds applications in various domains. Most recent works of event coreference resolution can be categorized into two types: (1) joint models; (2) pipeline models.
In 2012, Lee et al. (2012) presented a joint model
for cross-document coreference resolution. The model employed linear regression to build entity and event clusters and jointly solved events and entities references by handling both nominal and verbal mentions. Araki and Mitamura (2015) jointly formulated the event trigger extraction and event coreference as a problem of structured prediction to solve the error propagation problem. The authors utilized segment-based decoding with the multiplebeam search for event trigger identification and combined it with the best-first clustering for withindocument event coreference resolution. Lu and Ng (2017a) proposed a joint model of trigger detection, event coreference resolution, and event anaphoricity determination. The model employed a structured conditional random field with two types of factors: (1) unary factors; (2) binary and ternary factors. Yu et al. (2020a) presented a Pairwise Representation Learning scheme for cross-document and within-document event coreference problems. The scheme jointly encoded text snippets pair by forwarding concatenated sentences into a transformer encoder and employed structured argument features to argument the pairwise representation.
Different from these works, in this paper, we adopt a more straightforward pipeline model conforming to the general idea of the task. Pipeline models mainly have two stages: event detection and event coreference resolution. Traditionally, pipeline models first detect event triggers, arguments, and event features and then apply coreference resolution to the predicted event mentions. Liu et al. (2016) first used bidirectional GRU to detect events and a logistic regression model to classify event features (i.e., realis). The model then employed the latent antecedent tree method to conduct coreference resolution. Choubey and Huang (2017) proposed a novel iterative approach for withindocument and cross-document event coreference resolution. The method constructs event clusters gradually by exploiting inter-dependencies among event mentions in two stages. (Lai et al., 2021) employed OneIE (Lin et al., 2020) to extract event triggers and types. For other event features, the authors trained a simple classification model based on SpanBERT (Joshi et al., 2020). The model then used a context-dependent gated module (CDGM) to incorporate event features for event coreference resolution and a noisy training method to tackle the error propagation problem. Our work is closely related to (Lai et al., 2021). However, the method-
ologies are different. In our model, we found that a simple FFNN achieves a better result than the gating mechanism which questions the necessity for the CDGM gating method in event coreference resolution. We also innovatively introduce multiple similarity measures for event pair representation. With different ablation experiments and training settings, we verified that our proposed multiple measures learning model is effective and we hope our results can offer a new insight into not only event coreference resolution but also other fundamental tasks in NLP.

heading: 3 Method

heading: 3.1 Formalization
text: Our model focuses on within-document event coreference resolution. The input of our model is a document D containing n tokens. We then use a pre-tained language model to obtain the contextual embeddings of all the tokens in document D. Let X = (x1, ..., xn) denote the token embeddings. For every word Hi, we use Sitoken and E i token to denote its subword tokens’ start and end indices. Similarly, for the event ei, Sitrigger and E i trigger denote its event trigger’s start and end indices.

heading: 3.2 Model Overview
text: Based on our formalization, we present a model flow for event coreference resolution. Our model first uses OneIE (Lin et al., 2020) to detect event triggers following the practice of previous work in a given document and classify its event type. Then a novel prompt-based event features prediction model generates every event alone features.
After obtaining the predicted event triggers and event alone features, we use a multiple measures learning model for event coreference resolution. The model overview is shown in Figure 2. Our model consists of two parts: single mention encoding and multiple measures pair encoding. Firstly, the single mention encoding is employed to construct event-level representation for the given document D. Secondly, the multiple measures pair encoding aims at building event pair representations combined with multi-similarity and event alone features. Finally, we use a scoring layer to calculate the antecedent scores for every event pair.

heading: 4 Trigger Detection and Event Alone Feature Classification
text: Our model uses OneIE (Lin et al., 2020) to detect event triggers and their types and designs a model to predict other event alone features. The overview of event alone features prediction model can be seen in Figure 3. Specifically, we first insert special tokens <t> and </t> around the event trigger to highlight the target event. By doing so, we can transfer the original event feature prediction problem into a traditional text classification problem. To be more specific, we first use BERT-large-cased (Devlin et al., 2019) to build contextualized representations hr for the input sentence X , which can be defined as:
hr = Encoder(X)
We then utilize MLP to classify each sentence. The probability distribution of event feature labels is calculated by:
M r = Pooler(hr))
P (y|X) = softmax(MLP(M r)) where y ∈ Y and Y denotes a set of pre-defined event linguistic feature labels. We choose the label with the highest probability score among all the event feature labels.

heading: 4.1 Event-aware Representation Learning
text: Our model integrates multiple linguisticsmotivated and attention-based feature learning for event alone representations and multiple similarity measures to capture the distinction of event pair.

heading: 4.1.1 Event Alone Representation
text: Event alone representation includes linguisticaware properties integration and linguistic-agnostic attention-based feature for an event. For linguisticaware properties in our consideration for this work, type, polarity, modality, genericity and tense, which can be seen in Figure 1, are included. Their properties are used as linguistic-aware event alone features in our model. For feature encoding, we employed the same embedding method used in Lai et al. (2021). Given event ei feature k, our model use a simple embedding layer to obtain its feature representation F ki .
Besides, considering data-driven feature learning, we introduce a linguistic-agnostic attentionbased feature extraction module. Inspired by Dobrovolskii (2021), we proposed an attention based
method to encode event mentions. We calculate event representations as weighted sum of their triggers’ token embeddings, instead of simply averaging them (Lai et al., 2021). To be more specific, for a given word Hi , its word-level representation hi is calculated as where Wj is a learnable weight:
hi =
Eitoken∑
j=Sitoken
Wj · xj
We then use the same attention strategy to compute event representation ei where W ′j is a learnable weight :
ei =
Eitrigger∑
j=Sitrigger
W ′j · hj
By using this attention based encoder, our model can generate more accurate event representations with a better focus on important tokens.

heading: 4.1.2 Event Pair Representation
text: After obtaining event alone representations, we design a novel multiple similarity measures for measuring the coreference relationship. Given event representation ei and ej , we use the following similarity measures to calculate their event pair features:
Linear Similarity The linear similarity between ei and ej is defined as:
Sim (ij) lin = W T lin[ei; ej ] + b
where Wlin is a vector of trainable weights, b is a bias parameter, and [;] is vector concatenation operation. Cosine Similarity The cosine similarity between ei and ej is calculated by:
Sim(ij)cos = ei · ej
‖ei‖2 · ‖ej‖2
where ‖·‖2 denotes L2 norm. Element-wise Similarity The element-wise similarity between ei and ej is calculated by:
Sim (ij) ele = ei ej
where denotes element-wise multiplication. Then, we concatenate all three measures mentioned with event representation to build event pair representation eij which is calculated by:
eij = [ei; ej ;Sim (ij) lin ;Sim (ij) cos ;Sim (ij) ele ]
Similarity Fusion Layer After obtaining the pair representation, our model then employed a similarity fusion layer to construct the multi-similarity
based event pair representation sij , which is defined as:
sij = FFNNe(eij)
where FFNNe is a feed forward neural network mapping from R3×d1+2 → Rl. By fusing event mentions and similarity measures all together, we are able to generate the event representation which is cleaner and contains richer information. Feature Filtering Layer Similar to event pair encoding, for each event linguistic feature k, we too construct multi-similarity based feature pair representation f̂kij by:
F kij = [F k i ;F k j ;Sim (ij) lin ;Sim (ij) cos ;Sim (ij) ele ]
f̂kij = FFNNf (F k ij)
where FFNNf is a feed forward neural network mapping from R3×d2+2 → Rl
Whereas predicted features can be noisy and hence cause the error propagation problem. To address this problem, we propose a feature filtering layer to clean the noisy features and focus on more important event alone features. For each feature
k, we employed a feed forward neural network to filter the features.
fkij = FFNN k c (f̂ k ij)
By doing this, we are able to obtain clean event pair feature representations.

heading: 4.1.3 Coreference Scoring Layer
text: We then concatenate feature pair representations and event pair representation to get the multisimilarity based event-feature pair representation Sij .
Sij = [sij ; f 1 ij ; f 2 ij ; ...; f m ij ]
Next, we employ a coreference scoring layer to obtain antecedent scores between event i and event j where FFNNs maps from R(m+1)×l → R

heading: 5 Experiments

heading: 5.1 Dataset
text: We evaluate our model on ACE 2005 English dataset (Walker et al., 2006). ACE 2005 dataset adopts a strict notion of event coreference (Song et al., 2015). Specifically, two event mentions are coreferential if and only if “they had the same agent(s), patient(s), time, and location. We adopt the same split as Chen et al. (2015) and the detailed dataset statistics are shown in Table 1.

heading: 5.2 Model Setup
text: In order to have a fair comparison with Lai et al. (2021), we too use SpanBERT-base-cased (Joshi
et al., 2020) as the base Transformer encoder. We use AVG and CoNLL as our evaluation metrics. The AVG is the average F-score of four metrics (Lu and Ng, 2018): MUC, B3, CEAFe and BLANC. The CoNLL score is the average of first three metrics. All models are trained for 150 epochs. The detailed training parameters can be seen in Table 2
According to Lai et al. (2021), simply training the model using predicted event alone features may hurt the performance since the prediction of test set contains more errors than that of the train set. Inspired by Lai et al. (2021), we too introduce the noisy training method. Specifically, for every predicted event alone features k, we randomly assign a new label with a probability of pk. According to Lai et al. (2021), the probability pk varies inversely with the discrepancy between the train and test accuracies. Our noisy training probability for different features can be seen in Table 3. The probability value is positively correlated with the accuracy of the feature prediction. The main idea behind this is that for event feature that is more prone to prediction errors, we need a higher replacement probability to eliminate the noise propagation problem. By using the noisy training method, our model can be more sensitive to noisy features and hence mitigate the error propagation problem.

heading: 5.3 Results and Discussion
text: To probe deeper into the effect of different stages in our pipeline model. We employ our model in three settings: end-to-end, gold triggers, and all gold. In end-to-end, we use predicted triggers and predicted features. In gold trigger setting, given gold triggers we predict event features and then apply event coreference. In all gold setting, we utilize gold triggers and gold features to have a better understanding on coreference stage.

heading: 5.3.1 Event Feature Prediction
text: The feature prediction model formulates the event feature prediction problem as a traditional text classification problem by using special tokens to high-
light event triggers. The details can be seen in the appendix. Compared with the joint classification model proposed used by Lai et al. (2021), our model achieves obvious improvement on most event features. According to Lin et al. (2020), the event detection type-F1 score of OneIE on ACE 2005 test set is 74.7. Table 4 shows the accuracy of different event features.
From Table 4, we can see that comparing to CDGM (Lai et al., 2021), our model has a significant advantage on most of the features. We speculate that by introducing special tokens, we are able to maintain more semantic information than simply using the average of the trigger’s token embeddings as the classification input.

heading: 5.3.2 End-to-end
text: The end-to-end event coreference result can be seen in Table 5. We use OneIE (Lin et al., 2020) to extract event triggers and their types. We then formulate the event feature classification problem as a traditional text classification problem.
From the Table 5, we can see that our model achieves the state of the art on ACE 2005 dataset. Although Peng et al. (2016) used cross-validation to utilize more training data, our model still shows a great improvement. Also, as a direct comparison, our model outperforms the CDGM (Lai et al., 2021) by more than 2 points on AVG, which shows the effectiveness of our proposed pipeline model.

heading: 5.3.3 Gold Triggers and Predicted Features
text: In order to have a thorough comparison with the CDGM model, we also perform our model on ground-truth triggers and predicted event alone features. The results are shown in Table 6. We achieve an AVG score of 86.63 which significantly improve the CDGM performance by more than 2.5 points. The result sufficiently proves the advantage of our proposed event alone feature classification and multi-similarity coreference model.

heading: 5.3.4 All Gold
text: To have a better focus on the event coreference stage of our model, we conduct our experiments on all gold setting in which we use ground-truth triggers and ground-truth features. Table 7 shows the overall result. In all gold setting, we use two training strategy: noisy training and clean training. In clean training strategy, all event features are not randomly replaced. We can see that by using original clean features, we improve the noisy training
strategy by near 5.5 points. As a direct comparison, our model outperforms the CDGM without noisy training method and reaches the state-of-the-art of 92.32. Such significant improvements demonstrate the effectiveness of incorporating multiple similarity measures and attention based encoding.

heading: 6 Further Exploration

heading: 6.1 Gated or Not?
text: According to Lai et al. (2021), they propose Context Dependent Gated Module (CDGM) to obtain the event features. CDGM selectively distill input event features using a gating mecha-
nism. Therefore, to have a thorough comparison between CDGM and our proposed feature filtering layer, we introduce the CDGM method into our model. More specifically, after calculating the multi-similarity based feature pair representations, we employ CDGM instead of feature filtering layer to clean the event feature pair. We conduct experiments on all three settings and Table 8 shows the comparison result.
From Table 8, we can see that feature filtering layer outperforms the CDGM in all three settings. The largest improvement is made in gold trigger setting which reaches 0.91 points. Whereas, in all gold scenario, CDGM reaches a similar result to our model with a relatively small margin of 0.27 points. We speculate that in all gold setting, the ground-truth event features are already clean and hence filtering features exerts limited influence on the overall performance. When using ground truth triggers, all event features are predicted and therefore containing errors. In this case, using feature filtering layer can evolve our model to its greatest potential. In conclusion, the result shows that, compared to CDGM, simply using a feed forward neural network can reserve more information and hence achieves obvious improvement.

heading: 6.2 Attention or Not?
text: To have a better analysis on our proposed attention based encoder, we conduct ablation experiments. In the experiments, all models simply concatenate noisy feature pair representation together without any filtering operation (e.g., CDGM; feature filtering layer). We denote this concatenate strategy as simple. To be more specific, We treat the CDGM
model (Lai et al., 2021) without using cleaned features as the baseline which treats event representations as the average of their triggers’ token embedding. To have a fair comparison, we also use noisy feature representations by stripping the feature filtering layer and multi-similarity measures off our model and simply keep the attention based encoder. The overall results are shown in Table 9.
All models use the same event linguistic features. From Table 9, we can conclude that our model yields an absolute improvement of 5.05 points in AVG score. The great performance sufficiently proves the superiority of our attention based encoder. By introducing learnable weights to calculate the word-level and event-level representations, we believe that it can help our model focus on more critical tokens instead of simply averaging them. In a nutshell, our model acquires the ability to grab the core information for the given event through using an attention based encoder and hence achieves a better result.

heading: 6.3 Multi-similarity or Not?
text: We also conduct ablation experiments for the purpose of understanding the impact of different similarity measures. In the experiment, we deprive our model of multiple similarity measures pair encoder and similarity fusion layer and return to the original method which simply uses element-wise similarity to capture the relevance between two events. We treat the deprived model as the baseline. We then introducing the similarity fusion layer to generate more similarity-aware event pair representations. On the basis of fusion layer, we respectively add cosine similarity and linear similarity to better analyse the different influence they exert on the overall results. Finally ,we train our full model on multiple similarity measures. The detailed results can be seen in Table 10.
We can see clearly that our model outperforms the baseline model significantly by 1.82 points on
the AVG score. When combined with element-wise similarity, cosine similarity achieves a relatively small improvement compared to linear similarity and even suffers a minor decrease when used alone. We speculate that the main reason is that cosine similarity is more sensitive to the angle between two vectors instead of their lengths. Whereas, with the help of trainable weights, linear similarity can better capture the relevance between events by learning different perspectives of vectors. Nevertheless, our model still shows a great improvement when using all three similarity measures indicating that different similarity measures specialist in different aspects. As a result, we believe that using multiple similarity measures can help the model better capture the similarity between two events from more diverse and comprehensive aspects.

heading: 6.4 Noisy or Not?
text: To better analyze the usefulness of the noisy training method under different circumstances, we compare the noisy trained models and clean trained models under all three settings. The overall results are shown in Table 11.
We denote the models which replace the event feature labels randomly as noisy and clean for models which do not. Compared to the end-to-end setting, the model under the gold trigger setting achieves a much more apparent improvement when using the noisy training method. We presume the
main reason is that since the event trigger detection is far from perfect, the predicted event features would be more accurate when given the groundtruth triggers. With fewer error propagation problems, our model is able to uncover the deeper connections between event features. Similar to Lai et al. (2021), when using the all gold setting, however, our model suffers a decline of 5.49 points on the AVG score. Since all event features are clean in the all gold setting, our model do not need noisy training to tackle error event features and on the contrary, simply using clean features can already boost the performance to the most degree.

heading: 6.5 Performance on Different Length
text: By introducing attention based encoder and multiple measures pair encoder, we believe that we can accredit the great improvement of our model to better performances in long documents. Compared to CDGM, our attention based encoder can better construct word-level representation which can be less influenced by the distance and multiple similarity measures pair encoder further addresses the long-document problem.
To verify our hypothesis, we evaluate our model and the CDGM model respectively on the ACE 2005 test set for different document length ranges and the results are shown in Figure 4. According to the curves in the figure, in the all gold setting, it is obvious that our model do not suffer a significant decline as the document lengths grow. Although the CDGM model achieves a similar result with our model when the document lengths range be-
tween 1050 and 1199, our model still shows a great advantage over most document length ranges. Especially, when document length range grows from 300 to 900, the CDGM suffers a drastic decline while our model’s performance remains at a high level. In the end-to-end setting, our model shows a more obvious advantage in the long documents over the CDGM model indicating that our method can effectively cope with the long documents in different settings.

heading: 7 Conclusion
text: In this work, we propose a multiple measures learning model for event coreference resolution for the first time, as far as we know. By incorporating multiple event features and similarity measures, we are capable of calculating antecedent scores more comprehensively from different aspects. We unleash the potential of employing multiple similarity measures and filtering event features through various ablation experiments and proved that our method can effectively address the long sentence dependency problem. Our model achieves an evident and significant improvement on the ACE 2005 benchmark compared to current state-of-the-art models. In the future work, we aim at exploring the potential of incorporating multiple similarity metrics into different tasks.

heading: 8 Limitation
text: The findings of this study have to be seen in light of some limitations. Compared to CDGM (Lai et al., 2021), introducing multiple metrics to capture flexible cross-sentence event relationships would cause extra computation cost and slightly slow our training. The training parameters of different models can be seen in Table 12. We deprive our model of multiple similarity metrics and denote it as the baseline model. We can see that our model increases 7% parameters compared to CDGM and 0.009% compared to the baseline model.

heading: A Appendix
text: A.1 Full Experiment Results We report the all four metrics of our experiment results in Table 13.
A.2 Ablation Study In order to show that adding extra feature filtering FFNN layer is indeed effective instead of simply using additional parameters to help the model fit, we deprive our model of the feature filtering layer and increase the number of training parameters to the same size as before (110.52M parameters). We found that the final AVG score is 90.05 which is 2.05 lower than the model with the feature filtering layer. The result shows that adding extra FFNN is necessary for alleviating the errors. In fact, cleaning the features means distilling reliable signals from noisy features by mapping and activation, and the noisy training method is used to handle the error propagation problem. We believe that by using individual FFNN for each feature, our model can filter reliable signals from noisy features and hence generate more accurate feature representations.
Secondly, to prove that calculating the similarity for the event-only representation and event feature representation separately is more effective than concatenating the original event-only representation with its feature representation and have a single similarity measures encoding step. We conduct an experiment where the representations are simply concatenated and go through a single similarity measurement. The results showed that the model will suffer a performance drop of 2.03 which sufficiently verified the importance of calculating the similarity measures separately.
A.3 Case Study To facilitate a more illustrative comparison between our model and CDGM, a case study for the ACE 2005 dataset can be seen in Figure 5. We can see that compared to our model, CDGM additionally clusters "appointed" and "took office" into an event cluster. However, despite these two events having the same event type, they refer to two different persons being appointed as different ministers, and hence they are two different event. By incorporating different similarity measures and event features, we are able to analyse the coreference relation from different aspects, leading to a more accurate result.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7 limitation

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7 limitation

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 Experiment

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 Experiment
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Due to space limitation, we do not include these.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 Experiment
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Due to space limitation, we do not include these.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 Experiment
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 Experiment
C 3 Did you run computational experiments? Section 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 7 Limitation
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2 Model Setup

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 Experiments Appendix B Full Experiment Results we report the average results of 3 different runs

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.2 Model Setup
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future work, we aim at exploring the potential of incorporating multiple similarity metrics into different tasks