Learning Query Adaptive Anchor Representation for Inductive Relation Prediction
Year: 2023

Abstract:
Relation prediction on knowledge graphs (KGs) attempts to infer the missing links between entities. Most previous studies are limited to the transductive setting where all entities must be seen during the training, making them unable to perform reasoning on emerging entities. Recently, the inductive setting is proposed to handle the entities in the test phase to be unseen during training, However, it suffers from the inefficient reasoning under the enclosing subgraph extraction issue and the lack of effective entity-independent feature modeling. To this end, we propose a novel Query Adaptive Anchor Representation (QAAR) model for inductive relation prediction. First, we extract one opening subgraph and perform reasoning by one time for all candidate triples, which is more efficient when the number of candidate triples is large. Second, we define some query adaptive anchors which are independent on any specific entity. Based on these anchors, we take advantage of the transferable entityindependent features (relation-aware, structureaware and distance features) that can be used to produce entity embeddings for emerging unseen entities. Such entity-independent features is modeled by a query-aware graph attention network on the opening subgraph. Experimental results demonstrate that our proposed QAAR outperforms state-of-the-art baselines in inductive relation prediction task.

heading: 1 Introduction
text: Knowledge graphs (KGs) store a large amount of facts regarding the real-world knowledge in the form of (h, r, t), where h, t are head and tail entities and r is the relation between h and t. They have played important roles in various downstream applications, including but not limited to question answering (Yasunaga et al., 2021; Li and Xiong, 2022), semantic search (Xiong et al., 2017a; Zhang et al., 2021) and many more. Nowadays many
∗ Corresponding author.
large-scale KGs such as Freebase (Bollacker et al., 2008), Dbpedia (Lehmann et al., 2015), Yago (Hoffart et al., 2013; Mahdisoltani et al., 2015) and Wikidata (Vrandecic and Krötzsch, 2014) have been constructed, but they still suffer from an incompleteness problem. To make up such limitation, relation prediction or knowledge graph embedding (KGE) has been proposed to infer the missing links in KGs.
The conventional relation prediction models, such as TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), RotatE (Sun et al., 2019), TuckER (Balazevic et al., 2019), CompGCN (Vashishth et al., 2020), KBGAT (Nathani et al., 2019), ReInceptionE (Xie et al., 2020), DisenKGAT (Wu et al., 2021), PairRE (Chao et al., 2021), FFTAttH (Xiao et al., 2022), KGTuner (Zhang et al., 2022) have successively achieved satisfactory performance on relation prediction. They all follow a general strategy to represent the entities in the low-dimension vector space by using the KG structure and neighborhood information in the transductive setting, where all entities must be seen in the training phase. However, many KGs are evolving and new entities are emerging every day in the real world. These conventional relation prediction models in the transductive setings cannot effectively learn the embeddings for the new entities unless re-training the whole KGs from scrath. Recently, researchers have focused on relation prediction in the inductive setting, where new entities are emerging while testing. Compared to relation prediction in the transductive setting, the inductive setting is more challenging since the missing links can be inferred in KGs with unseen entities.
Currently, models on relation predictions in the inductive setting can be divided into two categories (Mai et al., 2021; Lin et al., 2022): rule-based methods and GCN-based methods. Rule-based methods (e.g., NeuralLP (Yang et al., 2017), DRUM (Sadeghian et al., 2019), RuleN (Meilicke et al.,
14041
2018) etc.) are able to handle the relation prediction with new/unseen entities since the rules are independent of any specific entities, their performance is usually insufficient due to the limitation of the flexible issue. GCN-based methods, such as GraIL (Teru et al., 2020), CoMPILE (Mai et al., 2021), Meta-iKG (Zheng et al., 2021), TACT (Chen et al., 2021), ConGLR (Lin et al., 2022), require to extract a k-hop enclosing subgraph1 for every candidate triple (h, r, t) and then perform reasoning on all these candidate subgraphs. However, these methods suffer from the inefficient issue since the number of candidate triples is usually large. Figure 1 shows an example of the query (Barack Obama, Nationality, ?). The subfigures (b), (c) and (d) are enclosing subgraphs extracted from graph (a) for three different candidate triples. Though the subgraphs (b) and (c) have the same structure, these previous methods have to perform reasoning two times, which will spend more reasoning time. Moreover, previous methods only use the distance as features and ignore rich local structure information in subgraphs, which leads to sub-optimal performance. As shown in Figure 1, the entity Honolulu in subfigure (b) is labeled as "(1,1)", which is the same with the entity Malik Obama in subfigure (d). This leads to similar node labeling for the nodes in (b) and (d), although these two subgraphs are totally different. Using these weak node features will result in poor performance since it is hard for the model to distinguish different subgraphs.
1Following (Teru et al., 2020), a k-hop enclosing subgraph between nodes h and t contains all nodes that appear on the graph from h to t with the maximum k-hop distance.
Based on the above analysis, we are motivated to address two research issues: (1) How to develop a model to perform inductive reasoning for all candidate triples with only one subgraph and one-time node labeling and reasoning. (2) How to extract adequate transferable entity-independent anchor2 knowledge to produce more expressive entity embeddings for emerging entities.
To address these issues, we propose a novel query adaptive anchor representation (QAAR) model. Specifically, for a given query (h, r, ?) where ? denotes a missing entity to be inferred, instead of extracting an enclosing subgraph for every candidate triple, we first extract a k-hop opening subgraph3 surrounding the query entity h for all candidates. This enables our model to perform reasoning by one time to rank all candidates, which is more efficient when the number of candidates is large. Then, we propose a dynamic anchor-based node labeling method to label each node with rich entity-independent information. Specifically, we construct three kinds of anchors for each query (h, r, ?), including relation types, the center node of the subgraph and the one-hop neighbors of the center node which is adaptively selected according to the query entity h. Based on these anchors, we can label each node in the subgraph with three kinds of features, namely relation-aware, structure-aware and distance features. These rich node features en-
2The anchors in this paper denote the transferable meta knowledge, which is independent of any specific entities and can be transferred to new graphs with unseen entities.
3Unlike the enclosing subgraph, the k-hop opening subgraph of entity h contains all neighboring nodes with the maximum k-hop distance.
able our model to learn expressive representation for subgraphs. Finally, a query-aware graph attention neural network is proposed to perform reasoning on subgraphs by gathering query-related neighborhood features. Experimental results on three benchmark datasets demonstrate that our proposed QAAR significantly outperforms previous methods, and achieves the new state-of-the-art (SOTA) performance for inductive relation prediction.

heading: 2 Our Approach
text: Figure 2 illustrates the overview of the proposed QAAR, which consists of four modules: (1) opening subgraph extraction; (2) query adaptive anchor construction; (3) anchor-based node labeling; and (4) query-aware reasoning. Next, we will describe each module in details.

heading: 2.1 Opening Subgraph Extraction
text: Multi-hop subgraphs can provide rich evidence to infer missing links due to the implicit correlation between different relations. For example, the relation (A, farther,B) can be inferred according to the two-hop facts (A,mother, C) and (C, husband,B). Thus, we first extract a multihop subgraph for inductive relation prediction. Different from GraIL (Teru et al., 2020) and its extensions (e.g., TACT (Chen et al., 2021), CoMPILE (Mai et al., 2021) and ConGLR (Lin et al., 2022)), which extract a specific enclosing subgraph for every candidate triple (h, r, t), we attempt to extract one opening subgraph for all candidate entities so that we can score all candidates with one-time reasoning on the subgraph. For a given query (h, r, ?), we first construct a k-hop opening subgraph for the query entity h by extracting the neighboring nodes surrounding h with maximum k-hop distance. We
denote the k-hop opening subgraph of h as Gk(h).

heading: 2.2 Query Adaptive Anchor Construction
text: To learn transferable entity-independent node features in a given subgraph, we first design some anchors which can be shared by different subgraphs. Ideally, the anchors should be independent of any specific entities and have the ability to express local structure information of the given subgraph. To this end, we introduce the following three kinds of anchors:
Relation types. Intuitively, the relation types are common information for subgraphs with different entities. Even on a new KG with unseen entities, the same set of relation types can be used to express the relations among entities. The set of relation types are denoted as R = {r1, r2, · · · , r|R|}, where ri is a relation in the relation set and |R| is the number of relations.
Center node. In our paper, each subgraph is extracted surrounding the given query entity h which can be considered as the center node of the subgraph and each opening subgraph has a unique center node. We use the center node as an anchor and denote it as node04. With the center node, the model can be aware of where is the start node to perform reasoning.
One-hop neighbors of center node. Besides, we also define the one-hop neighbors of the center node as anchors to better learn the local structure of subgraphs. For inductive relation prediction, we cannot directly use entity-specific embeddings to represent these one-hop neighbors. To address this problem, we leverage relationship between each
4Note that h denotes a specific query entity in KG, while node0 is a special symbol to represent the center node of a subgraph, which is transferable to any subgraph.
neighboring node and the center node to denote the neighbors. Specifically, for a neighboring entity t which is connected to the center node h with a triple (h, ri, t), the neighboring entity t can be denoted as rci which means a node connected to the center node with relation ri. Take the triple (Barack Obama, PlaceOfBirth, Honolulu) as an example, the entity Honolulu can be denoted as "the PlaceofBirth of Barack Obama". Thus, all neighboring nodes of the center node can be represented by combining the relation and center node, which can yield a set of one-hop neighborhood anchors D = {rci |(h, ri, t) ∈ Tk(h)}, where Tk(h) denotes the set of triples in Gk(h).
According to the above definition, we can obtain a set of anchor relations R and a set of anchor nodes Anc = {node0} ∪ D. These anchor relations and the nodes are independent on any specific entity, which can be acted as the transferred metaknowledge to bridge the gap between the training KG and the test KG.

heading: 2.3 Anchor-based Node Labeling
text: In this subsection, we describe how to label each node in the opening subgraph using the anchors R and Anc.
Relation-aware features. Given a node n in the subgraph Gk(h), we extract the node’s neighboring relations denoted as R(n). We can capture some important type information about the node from these neighboring relations. For example, the tail entities of relation PlaceOfBirth are entities with a place type. Therefore, we use these relations to learn embedding for node n:
vrn = ∑
r∈R(n) r (1)
where r is the embedding for the relation r. Structure-aware features. Only using the relation information cannot adequately express the structure of a given subgraph. In this paper, we propose a structure-aware labeling method to enrich the node features with local structure information. In a given subgraph, the center node and its one-hop neighboring nodes are viewed as anchor nodes Anc as defined in Section 2.2. Thus, we can leverage neighboring relationship between each node with these anchor nodes to learn local structure of each node in the subgraph. Formally, let Aj(n) = {ai|ai ∈ Nj(n) ⋂ Anc} be the j-th hop neighboring anchor nodes of node n, where
Nj(n) denotes the j-th hop neighbors of node n. The j-th hop structure feature of node n can be denoted as:
vjn = ∑
ai∈Aj(n) ai (2)
where ai is the embedding of anchor node ai. Note that by gathering neighboring anchor nodes from different hops, we can obtain multi-hop structureaware features {v0n,v1n, · · · ,vJn}, where J is the maximum hop of the structure-aware features5.
Distance features. The distance can reveal the degree of correlation between each node and the center node. Therefore, we also leverage distance features to enrich the representation of each node. Specifically, we visit the neighboring nodes from different hops of the center node and obtain the shortest distance between each node and the center node. Let dn denote the distance between node n and the center node, the embedding of distance feature is denoted as vdn.
Finally, the relation-aware, structure-aware and distance features are combined to learn the initial embedding of each node n:
e0n = v r n + v 0 n + · · ·+ vJn + vdn (3)

heading: 2.4 Query-aware Reasoning
text: In the above sections, we have described how to label the nodes for a extracted opening subgraph with relation-aware, structure-aware and distance features. Then, we can perform reasoning on the opening subgraph. In this paper, we use a queryaware graph attention neural network (GAT) to propagate neighborhood information in the opening subgraph. . Given a query (h, r, ?) and a k-hop subgraph Gk(h), we can aggregate the neighborhood features for each node n in subgraph Gk(h):
e′n = ∑
(r,p)∈N (n) αlp[e l p||rp]Wl + bl (4)
where N (n) = {(rp, p)|(n, rp, p) ∈ Tk(h)} is the set of neighbors of node n, Tk(h) is the set of triples in Gk(h). Wl and bl are trainable parameters in the l-th GAT layer, αlp is the attention weight for the neighbor (rp, p) which is computed as:
αlp = σ(r T a ([e l n||rp||elp]Wla + bla)) (5)
5Note that k in Gk(h) denotes the maximum hop of the opening subgraph surrounding the center node h, while J denotes the maximum hop of structure-aware neighboring anchors for each node n in the subgraph Gk(h).
where Wla and b l a are the trainable parameters, ra is the attention embedding of the query relation r, σ is the sigmoid activation function.
To alleviate the vanishing gradient problem, we use a gated layer to integrate the node features as well as its neighborhood features.
el+1n = g × eln + (1− g)× e′n (6)
where g is the gate to control the weight between the l-th layer node feature eln and the neighborhood feature e′n, which is defined as:
g = σ([eln||e′n]Wlg + blg) (7)
where Wlg and b l g are learnable parameters of the gate mechanism. In this paper, we stack query-aware GAT with L layers to propagate multi-hop neighborhood features and obtain the final node embeddings by concatenating representations of all layers, namely en = [e 0 n||e1n|| · · · ||eLn ]. Then, an average-pooling layer is applied to obtain the representation of the subgraph Gk(h):
eG = 1 |Ek| ∑
n∈Ek en (8)
Sequentially, we can compute the embedding for the given query (h, r, ?) by taking into account the subgraph embedding eG , the query entity embedding eh and the query relation embedding r:
equery = [eG ||eh||r]Wq + bq (9)
where Wq and bq are the trainable parameters of the query embedding layer.
Finally, the score for a candidate answer entity t can be obtained using the inner product of the
query embedding equery and the candidate answer entity embedding et:
f(h, r, t) = σ(eTqueryet) (10)
Note that all the candidate answer entities are in the same subgraph Gk(h), Therefore, we can compute the scores for all candidates by performing reasoning on the subgraph Gk(h) by one time. This is different from previous methods which need extract and process a specific subgraph for every candidate triple.
Following (Teru et al., 2020), we randomly sample a negative answer entity t′ for each query (h, r, ?) and correct answer t. The goal is to predict a higher score for each positive answer entity and lower score for the negative answer entity. Thus, we use the following cross entropy loss function to train the model:
L = ∑
(h,r,t)∈G(h) log(f(h, r, t′))− log(f(h, r, t))
(11) For the query (?, r, t), which needs to predict the missing head entity, we use a inverse form (t, r−1, ?) to convert it into a query with a missing tail entity.

heading: 3 Experiments

heading: 3.1 Datasets and Evaluate Metrics
text: To evaluate the inductive relation prediction performance of our QAAR, we conduct extensive experiments on three widely used datasets which are proposed in GraIL (Teru et al., 2020). These inductive datasets are extracted from WN18RR (Dettmers et al., 2018), FB15k-237(Toutanova et al., 2015) and NELL-995 (Xiong et al., 2017b). Each inductive dataset consists of train and test KGs and these
two KGs have no overlapping entities. And there are four versions with different sizes for each inductive dataset. The details of these datasets are shown in Table 1.
Following the prior work (Teru et al., 2020; Lin et al., 2022), we use hits@10 as the evaluation metric and compute hits@10 by ranking each test triple against 50 negative triples and check whether the positive triple is ranked in top 10. We run each experiment five times and report the mean results.

heading: 3.2 Implementation Details
text: We implement our model using Pytorch. We optimize our model using Adam (Kingma and Ba, 2015). We choose the optimal hyperparameters via grid search. We select the batch size from {4,8,16,32}, the learning rate from {0.0005,0.001,0.002}, weight decay from {1e-2,1e3,1e-4}, dropout rate from {0.1,0.2,0.3}, the number of GAT layers from {1,2,3}, the subgraph size k from {2,3,4,5}, the maximum depth J of structure-
aware features from {1,2,3}, the dimension size of output entity embedding from {30,50,80}. The best hyper-parameters are selected according to the hits@10 on validation set. Finally, the learning rate is set to 0.001, the batch size is set to 16, the weight decay is set to 1e-3, the dropout rate is set to 0.1, the number of GAT layers is set to 2, and the subgraph size k is set to 4, the maximum depth of structure features is set to J = 2, the dimension size of output entity embedding is set to 80.

heading: 3.3 Baselines
text: We compare our QAAR with the following rulebased models: Neural-LP (Yang et al., 2017), DRUM (Sadeghian et al., 2019), RuleN (Meilicke et al., 2018), and GCN-based models: Meta-iKG (Zheng et al., 2021), MorsE (Chen et al., 2022), GraIL (Teru et al., 2020), CoMPILE (Mai et al., 2021), TACT (Chen et al., 2021), ConGLR (Lin et al., 2022) and NodePiece (Galkin et al., 2022).

heading: 3.4 Experimental Results
text: Table 2 and Table 3 show the experimental results on the four versions from WN18RR, FB15k237 and NELL-995 under the evaluation metric of hits@10. From Table 2 and Table 3, we can have two interesting findings: (1) The GCN-based methods, which leverage GCNs to aggregate and propagate neighborhood information, performs better than rule-based methods. This indicates that rule-based methods are insufficient to obtain good performance for inductive relation prediction due to the brittleness of logical rules. Instead, GCNbased methods are more robust to capture complex structure patterns in KGs. (2) Our QAAR achieves better performance than most of SOTA GCN-based
methods in term of hits@10 (e.g., our model achieves the best performance on 10 out of 12 versions of the inductive datasets). On the average hits@10 metrics of these three datasets, our QAAR outperforms other SOTA models. Compared to previous SOTA method ConGLR, our QAAR can achieve 2.7%, 11.63% and 4.89% improvements on the average hits@10 metrics of WN18RR, FB15k237 and NELL-995 datasets respectively. And compared to the SOTA method NodePiece, our QAAR outperforms NodePiece by 5.55%, 2.24% and 2.38% margins on these three datasets. Compared with previous SOTA GCN-based methods, such as GraIL, TACT, CoMPILE, ConGLR and NodePiece, the proposed QAAR is able to capture richer transferable entity-independent features (e.g., relation-aware, structure-aware and distance features) to learn the inductive embedddings for the nodes in a subgraph. The experimental results demonstrate that our QAAR is superior to current SOTA models.

heading: 3.5 Ablation Study
text: In this subsection, we perform an ablation study to investigate how each module in QAAR affects the performance. Specifically, for the node label-
ing, we use relation-aware, structure-aware and distance features to label each node. To study the contributions of these labeling strategies, we construct three variant models: "QAAR w/o R" denotes the model removing relation-aware features from QAAR model, "QAAR w/o S" denotes the model without using structure-aware features, and "QAAR w/o D" denotes the model without using distance features. Moreover, we also investigate the effect of query-aware graph attention layers by replacing it with GCN layers and obtain a variant model "QAAR w/o A".
Table 4 and Table 5 show the experimental results of ablation study on the inductive datasets regarding the evaluation metric of hits@10. From the results, we can see that the performances of all variant models have decreased to some extent. This demonstrates all modules in QAAR play important roles for the performance improvement. Concretely, different components in QAAR have different effect on these inductive datasets. For WN18RR dataset, the structure-aware and distance features have more positive impact on the performance, while the relation-aware features have less positive impact and even slightly hurt the performance on v4 of WN18RR. This is because the relations in WN18RR express little entity type information. Take the most frequent relation "hypernym" in WN18RR as an example, we cannot infer the entity type for it. Instead, for FB15k-237 and NELL-995 datasets, the relations express more useful entity type information, which can help to improve the performance. Take the relation "PlaceOfBirth" in FB15k-237 as an example, we can infer that the head entity of this relation is a person and the tail entity is a place. On average, all three types of features can contribute to the improvements of the performance. Moreover, the query-aware graph
attention reasoning is also beneficial to relation prediction (QAAR vs. "QAAR w/o A"), which indicates that the query-aware graph attention layers can aggregate more relevant neighborhood information during reasoning.

heading: 3.6 Efficiency Analysis
text: In this section, we investigate the reasoning efficiency regarding the enclosing subgraph extraction vs. the opening subgraph extraction. We note that most previous inductive relation prediction models, such as CoMPILE (Mai et al., 2021), Meta-iKG (Zheng et al., 2021), TACT (Chen et al., 2021), ConGLR (Lin et al., 2022), are all based on the enclosing subgraph extraction on the basis of GraIL (Teru et al., 2020), and take more time to perform reasoning since they consider more additional mechanisms. Therefore, we choose GraIL as the representative baseline. Table 6 shows the reasoning time for different number of candidate triples on the version v1 of inductive test datasets extracted from WN18RR, FB15k-237 and NELL995. It can be seen that our QAAR can spend less reasoning time than the GraIL model on these three datasets. Specifically, when the number of negative candidates is 50, the reasoning time of QAAR is about 9x faster than GraIL on WN18RR, 20x faster on FB15k-237, 5x faster on NELL-995. We can also find that the reasoning time of GraIL will dramatically increases when the number of negative candidates increases. Different from GraIL, the reasoning time of our QAAR remains at a stable level for different number of candidates. The reason is that our QAAR is able to rank all candidates in one opening subgraph with one-time reasoning, while GraIL will extract a specific enclosing subgraph for every candidate and perform multiple reasoning on all these subgraphs. These empirical results verify the claim that our QAAR can perform more efficient reasoning than previous methods.

heading: 4 Conclusion
text: In this paper, we propose a novel query adaptive anchor representation (QAAR) model for inductive relation prediction. First, we develop a model to rank all candidate triples in one opening subgraph. Second, we design three types of query adaptive anchors for subgraphs and extract three kinds of entity-independent features including relation-aware, structure-aware and distance features, which are useful for inductive relation prediction. Finally, we use a query-aware graph attention model to perform inductive reasoning on subgraphs. The experimental results on 12 versions of inductive relation prediction datasets demonstrate the effectiveness and efficiency of our proposed QAAR model.
Limitations
Although our QAAR achieves better performance on inductive relation prediction, it still suffers from some limitations. First, for a give query we extract a k-hop subgraph without using any sampling method, which will require large GPU memory when the extracted subgraph is large. Second, our QAAR does not leverage logical rules to enhance the performance which has shown useful in previous methods (Lin et al., 2022). We believe that our method can be further improved by incorporating logical rules. We will leave these opening issues in the future work.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China under Grants 61972290 and 61972173, the Fundamental Research Funds for the Central Universities (No. CCNU22QN015), and Wuhan Knowledge Innovation Project (No. 2022010801010278). This work was also supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada, an NSERC CREATE award in ADERSIM6 , the York Research Chairs (YRC) program and an ORFRE (Ontario Research Fund-Research Excellence) award in BRAIN Alliance7. We thank anonymous reviewers for their thorough review comments on this paper.
6http://www.yorku.ca/adersim 7http://brainalliance.ca

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6 Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.1
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A. Statistic of Datasets
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.1
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We will leave these opening issues in the future work