RE-Matching: A Fine-Grained Semantic Matching Method for Zero-Shot Relation Extraction
Year: 2023

Abstract:
Semantic matching is a mainstream paradigm of zero-shot relation extraction, which matches a given input with a corresponding label description. The entities in the input should exactly match their hypernyms in the description, while the irrelevant contexts should be ignored when matching. However, general matching methods lack explicit modeling of the above matching pattern. In this work, we propose a fine-grained semantic matching method tailored for zero-shot relation extraction. Following the above matching pattern, we decompose the sentence-level similarity score into entity and context matching scores. Due to the lack of explicit annotations of the redundant components, we design a feature distillation module to adaptively identify the relationirrelevant features and reduce their negative impact on context matching. Experimental results show that our method achieves higher matching F1 score and has an inference speed 10 times faster, when compared with the stateof-the-art methods.

heading: 1 Introduction
text: Relation extraction (RE) is a fundamental task of natural language processing (NLP), which aims to extract the relations between entities in unstructured text. Benefitting from high-quality labeled data, neural relation extraction has achieved superior performance (Han et al., 2020; Wu and He, 2019a; Zhao et al., 2021). However, it is expensive and even impractical to endlessly label data for a fast-growing number of new relations.
In order to deal with the emerging new relations that lack labeled data, zero-shot relation extraction (ZeroRE) has recently attracted more attention. Levy et al. (2017) frame the ZeroRE as a slot-filling task solved in a question-answering way. Each relation is associated with a few question templates.
*Equal Contributions. †Corresponding authors.
However, the templates are expensive and timeconsuming to build (Chen and Li, 2021). Obamuyide and Vlachos (2018) simplify the templates to readily available relational descriptions and reformulate ZeroRE as a semantic matching task. Recently, pretrained model based ZeroRE methods have achieved great success. Siamese scheme and full encoding scheme are two mainstream methods for matching semantics. The siamese scheme separately encodes the input and description (Chen and Li, 2021). Therefore, the encoded representations of descriptions can be both stored and reused for each input, resulting in a fast inference. However, insufficient interaction during encoding also limits the matching performance. By contrast, the full encoding scheme performs selfattention over the pair to enrich interaction (Sainz et al., 2021), although the performance increase comes with a computational overhead. (For m inputs and n descriptions, the siamese scheme requires m + n encodings, while the number is m × n for full encoding). An approach that combines the advantages of both can be attractive.
Unlike ordinary sentence pairs, relational data has a unique matching pattern, which is not explicitly considered by general matching methods. As shown in fig. 1, the entities in the input should
6680
exactly match their hypernyms in the description (e.g., Apple and organization). Meanwhile, not all contextual words contribute equally to the relation semantics. For example, the clause “is a great company” is only used to modify Apple instead of expressing the relationship between Apple and California. Such redundant components should be ignored when matching. Due to the lack of explicit annotations to the redundant components, it is nontrivial for the model to learn to identify them.
In this work, we propose a fine-grained semantic matching method that improves both the accuracy and speed over the current state-of-theart. Specifically, we decouple encoding and matching into two modules. While the encoding module follows a siamese scheme for efficiency, the matching module is responsible for the finegrained interaction. Following the matching pattern of relational data, the sentence-level similarity score is decomposed into two: entity matching and context matching scores. To deal with the redundant components without explicit annotations, we design a feature distillation module. Context features that maximize a classification loss are identified as relation-irrelevant features. Then, the context representations are projected into the orthogonal space of the features to improve context matching. Experimental results show that this method outperforms state-of-the-art (SOTA) methods for ZeroRE, in terms of both accuracy and speed. Our codes are publicly available*.
The main contributions are three-fold: (1) We propose a fine-grained semantic matching method for ZeroRE, which explicitly models the matching pattern of relational data; (2) We propose a context distillation method, which can reduce the negative impact of irrelevant components on context matching; (3) Experimental results show that our method achieves SOTA matching F1 score together with an inference speed 10 times faster.

heading: 2 Related Works
text: Text semantic matching aims to predict a matching score that reflects the semantic similarity between a given pair of text sequences. The approaches used to calculate the matching score roughly fall into one of two groups. The first group uses a siamese scheme, which separately maps the pair of text sequences into a common feature space wherein a dot product, cosine, or parameterized
*https://github.com/zweny/RE-Matching
non-linearity is used to measure the similarity (Bai et al., 2009; Huang et al., 2013; Zamani et al., 2018; Wu et al., 2018; Chen and Li, 2021). The second group adopts a full encoding scheme, in which the concatenation of the text pair serves as a new input to a nonlinear matching function. Neural networks with different inductive biases are used to instantiate the matching function (Wu et al., 2017; Yang et al., 2018; Zhang et al., 2018). More recently, large-scale pretrained language models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) are introduced to yield richer interactions between the text pair, to improve semantic matching.
Unlike the general matching methods, the proposed method is designed to be specifically used for relational data. By explicitly modeling the matching pattern, our method achieves SOTA performance while decreasing the computational cost by an order of magnitude compared with a full encoding scheme (as the more candidate relations exist, the more cost decreases).

heading: 3 Approach
text: In this work, we propose a fine-grained semantic matching method tailored for relational data. To facilitate inference efficiency, we adopt a siamese scheme to separately encode the input and the candidate relation descriptions. To explicitly model the matching pattern of relational data, we decompose the matching score into entities and contexts matching scores. In addition, we design a context distillation module to reduce the negative impact of irrelevant components in the input on context matching.

heading: 3.1 Task Formulation and Method Overview
text: Task Formulation: In Zero-shot relation extraction (ZeroRE), the goal is to learn from the seen relations Rs = {rs1, rs2, ..., rsn} and generalize to the unseen relations Ru = {ru1 , ru2 , ..., rum}. Such two sets are disjoint, i.e., Rs ∩Ru = ∅ and only the samples of the seen relations Rs are available at the training phase. Following Chen and Li (2021), we formulate ZeroRE as a semantic matching problem. Specifically, given the training set D = {(xi, ei1, ei2, yi, di)|i = 1, .., N} with N samples, consisting of input instance xi, target entity pair ei1 and ei2, relation yi ∈ Rs and its corresponding relation description di, we optimize a matching model M(x, e1, e2, d) → s ∈ R, where the score s
measures the semantic similarity between the input instance x and the relation description d. When testing, we transfer the matching model M to extract unseen relations in Ru. Specifically, given a sample (xj , ej1, ej2) expressing an unseen relation in Ru, we make prediction by finding the relation ŷj whose description has the highest similarity score with the input sample.
Method Overview: We approach the problem with a fine-grained semantic matching method tailored for relational data. As illustrated in fig. 2, we decouple encoding and matching into two modules, explicitly modeling the matching pattern of relational data while ensuring efficient inference.
The encoding module is designed to extract both entity and contextual relational information from an input instance xi and a candidate description d ∈ {drsi |i = 1, .., n}, which lays the groundwork for fine-grained matching. To enrich the insufficient entity information in d, we separate the hypernyms (i.e. entity types) in d and design several methods to automatically expand them into a complete head (tail) entity description dh(dt). We adopt a fixed Sentence-BERT (Reimers and Gurevych, 2019) to encode the d, dh, dt to their representations d, dh, dt ∈ Rn, respectively. For input xi, we use a trainable BERT (Devlin et al., 2019) to encode its context and entity representation xi, xhi , xti ∈ Rn. Because descriptions and instances are separately encoded, the computational complexity is reduced from O(mn) to O(m + n), compared with the full encoding scheme (m,n represent the number of candidate descriptions and input instances, respectively).
The matching module is responsible for the interaction between input xi and description d. The score of entity matching is directly calculated
with cosine similarity cos(xhi ,d h) and cos(xti,d t). To reduce the impact of redundant information in xi, the context representation xi is fed into the distillation module, where xi is projected into the orthogonal space of irrelevant features to obtain the refined representation xpi . The score of context matching is cos(xpi ,d). Finally, the overall matching score is the sum of entity and context matching scores.

heading: 3.2 Input-Description Encoding Module
text: The encoding module aims to encode the entity and context information in the input and label description into fixed-length representations for subsequent fine-grained matching.

heading: 3.2.1 Relation Description Encoder
text: Each candidate relation corresponds to a natural language label description d ∈ {drsi |i = 1, .., n}. For example, the relation headquartered_in corresponds to the description “the headquarters of an organization is located in a place”, and its encoded representation d can be used for contextual matching. However, how an entity description is constructed (based on d), is important for a highquality entity representation. In this subsection, we explore different ways to automatically construct and enrich entity description as follows: Keyword: The entity hypernym (i.e. entity type) in d is directly used as the entity description. Take headquartered_in as an example, dh is “organization” and dt is “place”. Synonyms: To further enrich the entity information, we use the words that mean exactly or nearly the same as the original hypernym extracted using Wikidata† and Thesaurus ‡. Then, dh becomes
†https://www.wikidata.org/ ‡https://www.thesaurus.com/
“organization, institution, company”. Rule-based Template Filling: Inspired by prompt learning (Liu et al., 2021), a fluent and complete entity description may stimulate the pretrained model to output a high-quality entity representation. The synonym-extended hypernym sequence is filled into a template with some slots (i.e. the head/tail entity types including [S], [S], ...), and dh becomes “the head entity types including organization, institution, company”. Empirical results show that expanding all candidate descriptions with the above template works well. The customized template design is not the focus of this paper, so we leave it to future work.
Following (Chen and Li, 2021), we adopt a fixed sentence-BERT as the implementation of the relation description encoder f(·), which encodes the above descriptions into fixed-length representations, that is f(d) = d ∈ Rd, f(dh) = dh ∈ Rd, f(dt) = dt ∈ Rd.

heading: 3.2.2 Input Instance Encoder
text: Given an input instance xi = {w1, w2, ..., wn}, in which four reserved special tokens [Eh], [\Eh], [Et], [\Et] are inserted to mark the beginning and end of the head entity ei1 and tail entity ei2 respectively. We obtain entity representation xhi and x h i by MaxPool the corresponding hidden states of entity tokens. Following (Baldini Soares et al., 2019), the context representation xi is obtained by concatenating the hidden states of special token [Eh], [Et].
h1, ...,hn = BERT(w1, ..., wn) (1)
xhi = MaxPool(hbh , ...,heh) (2)
xti = MaxPool(hbt , ...,het) (3)
xi = ϕ( 〈 h[Eh]|h[Et] 〉 ), (4)
where ⟨·|·⟩ denotes the concatenation operator. bh, eh, bt, et denote the beginning and end position indexes of the head and tail entities respectively. h[Eh] and h[Et] represent the hidden states of [Eh] and [Et] respectively. Their corresponding position indexes are bh − 1 and bt − 1. ϕ(·) denotes a linear layer with tanh activation, converting the dimension of 〈 h[Eh]|h[Et] 〉 from 2n back to n.

heading: 3.3 Contextual Relation Feature Distillation
text: Due to the lack of explicit annotations to the relation-irrelevant components, it is non-trivial for the model to learn to identify them. This section
introduces how to reduce the negative impact of the redundant components on context matching.

heading: 3.3.1 Relation-Irrelevant Feature Aggregator
text: Given the output h1, ...,hn of the input instance encoder, we aggregate the relation-irrelevant features through a trainable query code q ∈ Rd as follows:
(α1, ..., αn) = Softmax(q · h1, ..., q · hn) (5)
x∗i = n∑
j=1
αj · hj , (6)
This leads to a immediate question, how do we make query code q select relation-irrelevant features from context? Intuitively, it is impossible for a relational classifier to discriminate the relations of input instances based on relationirrelevant features. Therefore, we introduce a Gradient Reverse Layer (GRL) (Ganin and Lempitsky, 2015) and optimize q by fooling the relational classifier.
probi = Softmax(GRL(x∗i ) ·W + b) (7) Lce,i = CrossEntropy(yi, probi), (8)
where W and b are the weights and biases of the relation classifier. x∗i goes through a GRL layer before being fed into the classifier. GRL does not affect forward propagation but changes the gradient sign during backpropagation by multiplying −λ. That is, as the training proceeds, the classifier is optimized by gradient descent to reduce Lce,i, while the query code q is optimized by gradient ascent to increase Lce,i, until no relational features are included in x∗i . The effectiveness of GRL has been verified in many literatures in the past (Ganin et al., 2016; Zhang et al., 2019).

heading: 3.3.2 Relation Feature Distillation Layer
text: The distillation module aims to reduce the negative impact of relation-irrelevant components on its representation and thus improving context matching. Given a context representation xi (refer to eq.4), as well as relation-irrelevant features x∗i (refer to eq.6), we achieve the above goal by projecting xi to the orthogonal space of x∗i . Specifically, we first project xi to the direction of x∗i to find the relationirrelevant features x̂i mixed in xi as follows:
x̂i = Proj(xi,x∗i ) (9)
Proj(a, b) = a · b |b| · b |b| , (10)
where Proj(·, ·) denotes the projection operator and a, b are the input vectors of Proj. Then, we obtain the refined context representation xpi by removing x̂i from xi as follows:
xpi = xi − x̂i. (11)
Essentially, eqs. 9-11 project xi in the orthogonal direction of x∗i . The above process is illustrated in the right side of fig. 2.

heading: 3.4 Fine-Grained Matching and Training
text: Following the matching pattern of relational data, we decompose sentence-level matching into entity matching and context matching. For an input xi and a candidate relation description d ∈ {drsi |i = 1, .., n}, the entity and context information are encoded into fixed-length representations (xhi , x t i, xpi ), and (d h, dt, d), respectively. The matching score between xi and d is the sum of entity and context matching scores as follows:
s(xi, d) =α · cos(xhi ,dh) + α · cos(xti,dt) + (1− 2 · α) · cos(xpi ,d), (12)
where α is a hyper-parameter and cos(·, ·) denotes the cosine operator. In order to optimize the above matching model and avoid over-fitting, we use margin loss as the objective function.
δi = s(xi, dyi)−max j ̸=yi (s(xi, dj)) (13)
Lm,i = max(0, γ − δi), (14)
where γ > 0 is a hyper-parameter, meaning that the matching score of a positive pair must be higher than the closest negative pair. The overall training objective is as follows:
L = 1 N
N∑
i=1
(Lce,i + Lm,i), (15)
where N is the batch size. When testing, the learned model is transferred to recognize the unseen relations in Ru, by a match between the input and the descriptions of the unseen relations.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: FewRel (Han et al., 2018) is a few-shot relation classification dataset collected from Wikipedia and further hand-annotated by crowd workers, which contains 80 relations and 700 sentences in each
type of relation. Wiki-ZSL (Chen and Li, 2021) is derived from Wikidata Knowledge Base and consists of 93,383 sentences on 113 relation types. Compared with the FewRel dataset, Wiki-ZSL has more abundant relational information but inevitably has more noise in raw data since it is generated by distant supervision.
Following Chia et al. (2022), we randomly select 5 relations for validation set, m ∈ {5, 10, 15} relations as unseen relations for testing set, and consider the remaining relations as seen relations for training set. Meanwhile, we randomly repeat the class selection process 5 times to ensure the reliability of the experiment results. We report the average results across different selections.

heading: 4.2 Compared Methods
text: To evaluate the effectiveness of our method, we make comparisons with a classic supervised method and state-of-the-art matching-based ZeroRE methods. We also compare a recent competitive seq2seq-based ZeroRE method. R-BERT (Wu and He, 2019b). A SOTA supervised RE method. Following Chen and Li (2021), we adapt it to zero-shot setting by using the sentence representation to perform nearest neighbor search and generate zero-shot prediction. ESIM (Levy et al., 2017). A classical matchingbased ZeroRE method, which uses Bi-LSTM to encode the input and label description. ZS-BERT (Chen and Li, 2021). A SOTA siamesebased ZeroRE method, which adopts BERT as the encoder to separately encode the input and relation description. In addition to classification loss, a metric-based loss is used to optimize representation space to improve nearest neighbor search. PromptMatch (Sainz et al., 2021). A SOTA full encoding-based ZeroRE method, which adopts BERT to encode the concatenation of input pairs and model their fine-grained semantic interaction. REPrompt (Chia et al., 2022). This baseline is a competitive seq2seq-based ZeroRE method. It uses GPT-2 to generate pseudo data of these relations to finetune the model. We use NoGen to denote the results without data augmentation.

heading: 4.3 Implementation Details
text: We use Bert-base-uncased as the input instance encoder, and we adopt a fixed sentence-Bert(Reimers and Gurevych, 2019) stsb-bert-base as the relation description encoder. We set AdamW(Loshchilov and Hutter, 2017) as the optimizer, and 2e− 6 as
the learning rate. Based on the validation set, we conduct hyper-parameter selection. α is selected among {0.2, 0.33, 0.4} and λ is selected among {0.1, 0.3, 0.5, 0.7}. Finally, we set α = 0.33 and λ = 0.5 for all datasets. The batch size is set to 128. All experiments are conducted using an NVIDIA GeForce RTX 3090.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: The results on Wiki-ZSL and FewRel datasets are reported in tab. 1, which shows that the proposed method consistently outperforms previous SOTA methods when targeting at a different number of unseen relations. Specifically, simple classification loss only focuses on the discrimination between known relations, so the supervised method such as R-BERT fails on ZeroRE. Although ZS-BERT is designed for ZeroRE, the siamese scheme limits the word-level interaction between the input and relation description, leading to suboptimal performance. By contrast, our method compensates for the shortcoming by explicit modeling of fine-grained matching patterns of relational
data, thereby outperforming ZS-BERT by a large margin. Although full encoding scheme such as PromptMatch can implicitly model fine-grained interactions through self-attention, the proposed method is still able to outperform it. One possible reason is that the relational matching pattern, as an inductive bias, alleviates the over-fitting to seen relations in the training set and thus our model has better generalization. Compared with REPrompt, which is a seq2seq-based method, our method achieves better results without using pseudo data of new relations to fine-tune the model, showing the superiority of the method.

heading: 5.2 Ablation Study
text: To study the contribution of each component in the proposed method, we conduct ablation experiments on the two datasets and display the results in tab. 2. The results show that the matching performance is declined if the context distillation module is removed (i.e., w/o Proj), indicating that the relationirrelevant information in the context disturbs the match of relational data, and projection in the distillation module is effective for reducing this impact. It is worth noting that entity information
plays an important role in relational data (see w/o Ent). Explicitly modeling the matching between entities and their hypernyms significantly improves the performance of the model. As two vital components, when both context distillation and entity matching are removed (i.e., w/o both), the matching degenerates into sentence-level matching and the performance will be seriously hurt.

heading: 5.3 Efficiency Advantage
text: Fig. 3 shows the time consumption and matching F1 scores on Wiki-ZSL and FewRel datasets. Take FewRel as an example, each relation contains 700 testing inputs. The siamese scheme (ZS-Bert and our RE-Matching) separately encodes input and descriptions and the encoding is run (700 ·m+m) times. By contrast, the full encoding scheme (PromptMatch) requires the concatenation of the text pair to be fed and the encoding is run (700·m2) times. Clearly, as the number of new relations m increases, ZS-Bert and our RE-Matching have a significant efficiency advantage over PromptMatch that adopts a full encoding scheme. When m = 15, the inference time can be reduced by more than 10 times. Although our method takes slightly more time than ZS-BERT, the fine-grained matching brings a significant F1 improvement. As shown
in tab. 1, when m = 15, our method improves the F1 score by 33.07% and 36.84% on two datasets respectively, compared with ZS-BERT.

heading: 5.4 Consistency over Various Encoders
text: In this section, we evaluate the effectiveness of our method by varying the selection of encoders. Tab. 3 shows the comparison results between ours and the full encoding scheme (i.e., PromptMatch) when m = 15. It can be observed that our method achieves consistent improvement. PromptMatch is able to learn the word-level interaction by selfattention on the input-description pair. However, the data-driven optimization paradigm usually learns spurious correlations in data pairs, especially in noisy data. By contrast, the matching pattern can be seen as a reasonable inductive bias. Modeling the pattern can reduce the overfitting risk. Therefore, our method consistently outperforms PromptMatch, especially on the noisy Wiki-ZSL (A distantly supervised dataset. See sec. 4.1).

heading: 5.5 Error Analysis
text: What errors of baselines our method is able to correct? (1) Our method reduces the negative impact of irrelevant components on context matching by projection. To intuitively show this, we use the attribution technique (Feng et al., 2018) to find words that are highly influential to the context matching score. A visualization case is shown in fig. 4. When using the projection, the model pays more attention to entities and relational phrases instead of irrelevant components (e.g., held, first, a) to make the prediction. (2) The entity matching score can provide more information to distinguish confusing relations. Taking P59:constellation_of_celestial_sphere as an example, its F1 is only 0.123 without entity matching score. In this example, 79.71% of the incorrect cases are recognized as P361:part_of, due to the fact that the descriptions of P361 and P59
are similar (a constellation is described as part of a celestial sphere). With entity matching, the type of head and tail entities is constrained and the F1 is improved from 0.123 to 0.950. What errors need to be addressed in further studies? Taking P937:place_of_work and P19:place_of_birth as examples, the entity pair type of both relations are person-location. Therefore, explicitly modeling entity matching does not lead to further improvement, when compared with the baselines. In addition, some abstract relations are difficult to accurately recognize. Take P460:said_to_be_the_same as an example. Such abstract relations do not have explicit entity types, and it is difficult to give a high-quality relation description. Therefore, the F1 score of P460 is only 0.03.

heading: 5.6 Hyper-parameter Analysis
text: γ is an important hyper-parameter in our optimization objective. It means that the matching score of the positive pair should be at least γ higher than that of the negative pairs. In this subsection, we conduct experiments on two datasets (single class selection process) to study the influence of the value γ on matching performance. From fig. 5 we can obtain the following observation. First, as γ increases from 0, the model learns the difference between positive and negative pairs, thus assigning higher matching scores to input and correct description. When γ increases to a critical value (i.e., 0.06), the performance begins to decline gradually. This indicates that a too-large value makes the model overfit to known relation in training set, and then
lose generalization. Finally, even if γ is increased to a very large value, the matching does not crash. This shows that our method has good robustness.

heading: 5.7 The Influence of Entity Description
text: In a description, there are usually only one or two words that identify the entity type. As shown in tab. 4, we explore how to build a high-quality entity representation based on the words for entity matching. A simple way is to directly encode the words as entity representation. However, insufficient entity information limits matching performance. We further use synonyms to enrich entity information and improve the F1 score by 3.85% in Wiki-ZSL. In order to further construct a complete and fluent entity description, we fill the synonym-extended word sequence into the template slot. Compared with the original keyword method, the two operations improve F1 score by 6.78% and 1.75% on the two datasets respectively.

heading: 6 Conclusions
text: In this work, we propose a fine-grained semantic matching method for ZeroRE. This method explicitly models the matching pattern of relational data, by decomposing the similarity score into entity and context matching scores. We explore various ways to enrich entity description and thus facilitating high-quality entity representation. The context distillation module effectively reduces the negative impact of irrelevant components on
context matching. Experimental results show that our method achieves higher matching F1 score and has an inference speed 10 times faster when compared with SOTA methods.
Limitations
Elaborated relation descriptions are the foundation of the matching-based methods to achieve superior performance. Although we have proposed some ways to enrich the entity information in the descriptions, it is still a promising direction to explore more diversified and effective ways to enrich relation description (e.g. ensemble of multiple descriptions). We leave this as our future work.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.61976056,62076069,62206057), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract and introduction sections
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3
3 B1. Did you cite the creators of artifacts you used? section 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? section 3
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 4
C 3 Did you run computational experiments? section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this as our future work