Query Structure Modeling for Inductive Logical Reasoning Over Knowledge Graphs
Year: 2023

Abstract:
Logical reasoning over incomplete knowledge graphs to answer complex logical queries is a challenging task. With the emergence of new entities and relations in constantly evolving KGs, inductive logical reasoning over KGs has become a crucial problem. However, previous PLMs-based methods struggle to model the logical structures of complex queries, which limits their ability to generalize within the same structure. In this paper, we propose a structuremodeled textual encoding framework for inductive logical reasoning over KGs. It encodes linearized query structures and entities using pre-trained language models to find answers. For structure modeling of complex queries, we design stepwise instructions that implicitly prompt PLMs on the execution order of geometric operations in each query. We further separately model different geometric operations (i.e., projection, intersection, and union) on the representation space using a pre-trained encoder with additional attention and maxout layers to enhance structured modeling. We conduct experiments on two inductive logical reasoning datasets and three transductive datasets. The results demonstrate the effectiveness of our method on logical reasoning over KGs in both inductive and transductive settings. 1

heading: 1 Introduction
text: Logical reasoning over knowledge graphs (KGs) aims to answer complex logical queries given largescale KGs (Guu et al., 2015; Hamilton et al., 2018). Recent years have witnessed increasing attention on logical reasoning over widely used KGs such as Freebase (Bollacker et al., 2008), Yago (Suchanek
∗Corresponding author 1Codes are publicly available at https://github.com/
WangsyGit/InductiveLR.
et al., 2007), NELL (Carlson et al., 2010) and Wikidata (Vrandečić and Krötzsch, 2014). With missing relations in the KG, it is challenging to deduce correct answers to complex queries by traversing the graph. Previous work primarily focuses on transductive logical reasoning where the training and testing are done on the same KG with the same group of entities. They typically rely on geometric embedding-based methods to map both entities
4706
and queries into a joint low-dimensional vector space (Hamilton et al., 2018; Ren et al., 2020; Ren and Leskovec, 2020). The goal is to push the embeddings of answer entities and queries to be close to each other, allowing answers to be predicted through embedding similarity even when the involved relation is absent. In contrast, the inductive setting of logical reasoning has been rarely studied which requires generalizing to unseen entities and relations or even new KGs. As real-world KGs are usually dynamic with emerging unseen entities and relations, it’s significant to explore the inductive setting for complex query answering.
Existing research on inductive logical reasoning mainly follows two directions. The first inherits embedding-based methods and incorporates type as additional information to improve inductive capability (Hu et al., 2022), which can not generalize to unseen types of entities and relations. The second direction leverages pre-trained language models (PLMs) to encode textual information of entities/relations for generalization to unseen elements (Wang et al., 2021b; Daza et al., 2021; Wang et al., 2021a). PLMs-based approaches provide more flexible solutions and generate better results. However, they only explore link prediction tasks of one-step reasoning, and simply linearize the triplet or subgraph into text sequence without modeling explicit reasoning structure (Yao et al., 2019; Zha et al., 2022). An example is shown in Figure 1. Two findings stand out. (1) The query q1 and q2 appear to be similar in format (both as a conjunction of three terms) but actually have different logical structures. PLMs-based methods that encode flattened queries can not model this structure information for correct logical reasoning. (2) Although queries q1 and q3 (also q2 and q4) contain different elements, they share the same logical structure. Motivated by these, we argue that structure modeling of different complex queries can further boost the generalization ability of logical reasoners.
In this paper, we propose to model query structure for inductive logical reasoning over KGs. Specifically, we transform the query structure into a sequence using textual names of involved entities, relations, and logical operators. For complex query structures composed of multiple geometric operations over entities and relations, we introduce two measures to enable logical structure modeling during text encoding. First, we design stepwise instructions for different query types to indicate
which operation in the query structure should be conducted at each step and feed them as the structural prompt to PLMs. Besides, we extend the pre-trained encoder with an additional attention layer and a maxout layer to respectively model different geometric operations including projection, intersection, and union on the representation space, to implicitly inject structured modeling into PLMs. Our proposed method is a generic inductive framework, which can be plugged into different PLMs for better performance.
We conduct experiments on two datasets for inductive logical reasoning over KGs, FB15k-237V2 and NELL-V3 (Teru et al., 2020) as well as three transductive datasets, FB15k (Bordes et al., 2013), FB15k-237 (Toutanova and Chen, 2015), and NELL995 (Xiong et al., 2017). The results demonstrate that our method achieves strong inductive performance on unseen entities and relations, even across different KGs, without sacrificing logical reasoning capability and generalizability to new query structures.

heading: 2 Methodology
text: In this work, we study the task of complex logical reasoning over KGs. The input is a first-order logic query q which can include any set of existential quantification (∃), conjunction (∧), and disjunction (∨) operators (such as the query in Figure 1). Our goal is to predict a set of entities A = {a1, a2, a3...} that answer the query q based on an incomplete KG G = (E ,R) which consists of a set of triplets (h, r, t) but lacks several involved relations. Here h, t ∈ E are the head and tail entities and r ∈ R is the relation between them. We mainly focus on the inductive setting of KG logical reasoning, where the evaluated queries contain entities/relations that are completely unseen during the training period.
Figure 2 shows the overall architecture of our model. We propose to encode the text sequences of query structures and predict the answer entities based on representation similarity for inductive logical reasoning over KGs. In this section, we first list different types of query structures studied in logical reasoning over KGs(§ 2.1). Then according to various query types, we introduce our structure linearization and structural prompts for textual encoding(§ 2.2). The geometric operation modeling and query answer prediction modules are described in (§ 2.3) and (§ 2.4). Finally, we provide the de-
tails about training and inference (§ 2.5).

heading: 2.1 Query Structure Types
text: Following (Ren et al., 2020), we consider 9 types of complex query structures which are composed of different sets of geometric operations (including projection, intersection and union) over entities and relations. These include six single-operation query structures and three mixed-operation ones. Specifically, three query types only focus on projection, including one-relation projection (1p), two-relation projection (2p), and three-relation projection (3p). Two query types focus on the intersection of two triplets (2i) and three triplets (3i), and another one focuses on the union of two triplets (2u). The three mixed-operation query structures are respectively the combinations of intersection&projection (ip), projection&intersection (pi), and union&projection (up). The different query structures are illustrated as the following formula:
[1p] q = v? : r1(e1, v?) [2p] q = v?.∃v : r1(e1, v) ∧ r2(v, v?) [3p] q = v?.∃v1, v2 : r1(e1, v1) ∧ r2(v1, v2) ∧ r3(v2, v?) [2i] q = v? : r1(e1, v?) ∧ r2(e2, v?) [3i] q = v? : r1(e1, v?) ∧ r2(e2, v?) ∧ r3(e3, v?) [pi] q = v?.∃v : r1(e1, v) ∧ r2(v, v?) ∧ r3(e2, v?) [ip] q = v?.∃v : r1(e1, v) ∧ r2(e2, v) ∧ r3(v, v?) [2u] q = v? : r1(e1, v?) ∨ r2(e2, v?) [up] q = v?.∃v : (r1(e1, v) ∨ r2(e2, v)) ∧ r3(v, v?)
(1)
where ei and vi are the anchor entities and existentially quantified bound variables entities, and v? are the target answer entities to the query. As these complex queries contain rich logical structural information, we need to model the structure knowledge during textual encoding for better inductive generalization within the same logical structure.

heading: 2.2 Query Structure Encoding
text: In order to use PLMs for better generalization to unseen entities/relations, we first need to linearize the query structures into text sequences. We also design instructions for each query type as a structural prompt to implicitly indicate the order of geometric operations execution to PLMs. We concatenate the linearized query and structural prompt as the input, and encode them to obtain the query representation for matching with the answer entities.
Query Structure Linearization Given a query structure q, we customize its linearization method according to the query type. For each triplet ri(ei, v) in the query, we formulate it as “[anchor] t(ei) [relation] t(ri)” and ignores the intermediate variable entities, where t(ei) and t(ri) are the textual names of anchor entity ei and relation ri. Then we add the names of logical operations before the involved subqueries. For example, the query structure of type [2p] can be linearized into sequence “[projection] [anchor] t(e1) [relation] t(r1) [projection] [relation] t(r2)” and the query structure of type [2i] can be mapped into “[intersection] [projection] [anchor]
t(e1) [relation] t(r1) [projection] [anchor] t(e2) [relation] t(r2)”.
For query types [ip] and [up] that are composed of intersection/union and projection, the last relation projection is conducted over the intersection/union of previous triplets. Directly flattening the query is unable to keep such structural information. Therefore, we propose to split the intersection/union and repeatedly connect each triplet with the last relation projection which moves the intersection/union operation to the final step. This transformation is equivalent to the original query structure. For example, the following two query structures are equivalent and are both of type [up].
(r1(e1, v) ∨ r2(e2, v)) ∧ r3(v, v?) (r1(e1, v) ∧ r3(v, v?)) ∨ (r2(e2, v) ∧ r3(v, v?))
(2)
Based on this transformation, we linearize the query structure of type [up] into the text sequence as “[union] [projection] [anchor] t(e1) [relation] t(r1) [projection] [relation] t(r3) [projection] [anchor] t(e2) [relation] t(r2) [projection] [relation] t(r3)”. The details of structure linearization templates for each query type are listed in Appendix A.
Structural Prompt Besides feeding the linearized query structure into PLMs, we also introduce stepwise instructions that indicate the order of geometric operations execution to prompt the pretrained encoder with implicit structural information of the query. Specifically, each prompt consists of two parts: the number of total execution steps and the operation order in the query, which is formulated as “total steps: operation order”. For query types [ip] and [up], we define the total steps and operation order according to the transformed query structure in Eq 2. The detailed structural prompts of each query type are presented in Table 1.
Then the input t can be obtained by concatenating the structural prompt s and the linearized query structure t(q), which is formulated as “[CLS] [qtype] s [SEP] t(q)”. We feed it to the pretrained encoder and obtain the output hidden states H = (h1, h2, ..., h|t|).

heading: 2.3 Geometric Operation Modeling
text: To further enhance the structure modeling during textual encoding, we propose to separately
model different geometric operations in logical queries to explore their spatial characteristics. As Transformer-based encoders are widely used to implicitly learn the translation function for simple link prediction and question answering (Yao et al., 2019; Wang et al., 2021a), we directly utilize it for modeling multi-hop relation projection. For each path of relation projection r1(e1, v1) ∧ ... ∧ rj(vj , v?), we extract the hidden states corresponding to sequence “[anchor] t(e1) [relation] t(r1) ... [projection] [relation] t(rj)” from H = (h1, h2, ..., h|t|). We then take the average as the representation of target entity v?, which also can be viewed as the representation hq of the query that only involves relation projection operation.
For the intersection and union of multiple subqueries, we adopt an attention layer (Bahdanau et al., 2014) and a maxout layer (Goodfellow et al., 2013) on top of the pre-trained encoder to respectively model these two operations in the geometric representation space. Specifically, we feed the representations of target entities in all subqueries to these two additional layers to achieve the intersection and union operations. The output can be taken as the query representation hq that contains intersection or union.
As presented in (Ren and Leskovec, 2020), the complete set of first-order logic operations encompasses existential quantification (∃), conjunction (∧), disjunction (∨) and negation(¬). Our approach covers the first three operations by modeling relation projection, intersection, and union respectively. The negation operation is not individually modeled as pre-trained encoders are capable of capturing se-
mantic exclusion for negation. We can add negative terms such as “not” before the corresponding relations within the input and feed it into pre-trained encoders to naturally address this task.

heading: 2.4 Query Answer Prediction
text: To answer the complex query q, we adopt the Siamese dual encoder (Gillick et al., 2018) to respectively encode the query q and the candidate entity ci to match the answer. We formulate the entity input as the sequence “[CLS] [target] t(ci)” and feed it into the pre-trained encoder to obtain the candidate entity representation hci by taking the average of the hidden states. Then we compute the similarity di between the query representation hq and entity representation hci , and encourage the query representation to be similar to the positive answer entities while dissimilar to negative entities. The entities whose representations are similar enough to the query will be predicted as the answers. We can pre-compute the representations of all candidate entities and utilize them to predict answers for different queries more efficiently.
The above matching scheme can handle the inductive setting when the candidate entity set is not closed and new entities may arise. To improve the answer prediction accuracy in the transductive setting where the candidate entity set is closed, we also employ a classification layer on top of the query representation hq. Given the fixed candidate entity set (c1, c2, ..., cN ), the classification layer with a softmax function will output an N -dimensional plausibility distribution (sc1 , sc2 , ..., scN ) for each candidate entity ci to be the answer.

heading: 2.5 Training & Inference
text: We simultaneously optimize a matching objective and a classification objective to train our inductive model for answering complex logical queries. For the former, we adopt contrastive learning (Chen et al., 2020) which needs to separate the positive and negative answers for the query. We take the given ground-truth answer c+ as positive and implement in-batch negative sampling to collect the negatives. We measure the similarity between the query and entities using dot product, and follow (He et al., 2020) to utilize InfoNCE as the contrastive loss. The loss function is formulated as Eq. 3 where τ is
LM = − log exp(hq · hc+/τ)∑N i=1(exp(hq · hci/τ)
(3)
the temperature hyper-parameter and N is the total number of candidate entities including positives and negatives. For the classification objective, we take all entities in each KG as candidate entities and calculate the cross-entropy loss as Eq. 4.
LC = − log sc+∑N
i=1 exp(sci) (4)
These two losses are combined in a weighted manner as L = LM + λLC and λ is the weighted hyper-parameter.
During inference, we perform differently for inductive and transductive logical query answering. For the inductive reasoning, we utilize the matching scheme and rank the representation similarities between the query and all candidate entities for query answer prediction. For the transductive inference, we only adopt the classification scheme and find the most plausible answer according to classification scores.

heading: 3 Experiments

heading: 3.1 Experiment Setup
text: We conduct experiments on complex logical reasoning over KGs in both inductive and transductive setting. For the inductive setting, we adopt two datasets, FB15k-237-V2 and NELL-V3, that have disjoint sets of entities for training and evaluation, as introduced by (Teru et al., 2020). To further challenge our model, we also illustrate the cross-KG inductive generalization performance by respectively taking FB15k and NELL995 for training/inference and inference/training that contain completely different entities and relations. In the transductive setting, we evaluate our model on the generated queries from three datasets: FB15k (Bordes et al., 2013), FB15k-237 (Toutanova and Chen, 2015), and NELL995 (Xiong et al., 2017), as proposed by (Ren et al., 2020). All these datasets cover nine types of query structures. We follow the setting of (Ren et al., 2020) to illustrate the generalization within the same structure to unseen entities and relations, and also the generalization to more complicated unseen structures composed of different structures. Specifically, we train our model on the first five types of query structures (1p, 2p, 3p, 2i, 3i) and evaluate it on all nine query types (1p, 2p, 3p, 2i, 3i, pi, ip, 2u, up), including both seen and unseen query structures during training. The data split statistics of logical queries in these datasets are provided in Appendix B.

heading: 3.2 Implementation Details
text: We take bert-large-cased and bert-base-cased (Devlin et al., 2018) as the pre-trained encoder for encoding the query structure in (FB15k-237V2, NELL-V3) and (FB15k, FB15k-237, and NELL995), respectively. All models are implemented using Huggingface (Wolf et al., 2019), and trained for 30 epochs on 4 NVIDIA Tesla V100 GPUs with 16 GB of memory. The Adam is taken as the optimizer and the learning rate is 1.5e-4. We use a linear learning rate scheduler with 10% warmup proportion. The weight hyper-parameter to balance losses is set to λ = 0.3 or λ = 0.4. For automatic evaluation, we use Hits@K (K = 3, 10) as the metrics, which calculate the proportion of correct answer entities ranked among the top-K.

heading: 3.3 Inductive Setting
text: Unseen Entities Generalization To illustrate the inductive performance of complex logical reasoning over KGs, we first make a compar-
ison on FB15k-237-V2 and NELL-V3 datasets for generalizing to unseen entities. We compare our structure-modeled inductive logical reasoning method (SILR), with the baseline embedding-based method Q2B (Ren et al., 2020) and the best version of the inductive model TEMP(GQE) (Hu et al., 2022). TEMP(GQE) enriches embedding method GQE (Hamilton et al., 2018) with type information which has achieved the state-of-the-art performance. BiQE (Kotnis et al., 2021) is also a textual encoding method with positional embedding for logical query but only in the transductive setting. We reimplement it by replacing the original classification-based prediction with the matching scheme for an inductive comparison.
The experimental results are shown in Table 2. We can see that our SILR outperforms all other models on both FB15k-237-V2 and NELL-V3 datasets by a considerable margin. This highlights the effectiveness of our method for inductive logical reasoning over unseen entities. Additionally, the im-
provement over the other positional textual encoding model, BiQE, demonstrates that our structure knowledge modeling during textual encoding is capable of enhancing the inductive complex query answering capability.
Cross-KG Generalization We further explore a more challenging cross-KG inductive setting, where the model is trained and tested on different datasets and requires generalizing to completely different KGs. Specifically, we take FB15k and NELL995 as the source/target and target/source datasets, respectively. In this scenario, we adopt the few-shot setting, where 500 random samples of the target domain are provided for continual learning to achieve better transferring. As embedding-based methods, even when aware of type information, are unable to embed most entities and relations in new KGs with unseen types, we only compare our SILR with the reimplemented BiQE. The results in Table 3 show that our SILR performs better than BiQE, and it can not only generalize to unseen entities but also perform logical reasoning over new KGs with only a few portions observed. This manifests the effectiveness of our method on inductive logical reasoning over KGs even in the real-world challenging cross-KG setting.

heading: 3.4 Transductive Setting
text: Although textual encoding methods have the inductive potential, their performance often lags behind embedding-based models due to learning inefficiency and the inability to structure knowledge modeling (Wang et al., 2022). We also compare our SILR with transductive logical reasoning methods to illustrate the logical reasoning performance over KGs with structure modeling. The compared models including GQE, Q2B, TEMP(Q2B) and BiQE where the first three are embedding-based models while the last one is a positional textual encoding model. Since BiQE does not evaluate query types 2u and up, and does not provide results for FB15k, we reimplement it for a fair comparison.
The results are shown in Table 4. Our SILR outperforms BiQE, particularly in query types involving intersection and union, indicating that our structure knowledge modeling can effectively improve the logical reasoning performance of textual encoding and help generalize to unseen query structures. Although textual encoding methods have the potential for inductive KG reasoning, they still lag behind embedding-based methods for the transductive setting, due to explicit structure modeling and better learning efficiency of embedding-
based methods (Wang et al., 2022). In this work, we mainly focus on improving textual encoding methods for inductive complex reasoning, but our method still achieves comparable transductive performance. This demonstrates the effectiveness of our inductive method with query structure modeling on transductive logical reasoning over KGs.

heading: 3.5 Further Analysis
text: Ablation Study To dive into the impact of different components in our model on both inductive and transductive logical reasoning over KGs, we conduct an ablation study on the FB15k-237-V2 and FB15k-237 datasets. We respectively take bertlarge-cased and bert-base-cased as baseline models for FB15k-237-V2 and FB15k-237. They remove Structural Prompt (SP) and Geometric Operation Modeling (GOM) from the final model SILR, which directly encodes linearized query structures for answer prediction.
As shown in Table 5, incorporating structural prompting and geometric operation modeling can both improve the baseline model, but still perform worse than our final SILR. This indicates that these two measures for modeling structure knowledge during query text encoding can enhance the inductive and transductive performance of logical reasoning over KGs.
Query Structure Generalization Embeddingbased methods are known to generalize well to unseen query structures due to their explicit spatial structure learning. To analyze the generalizability of our implicit structure-modeled textual encoding method to different logical query structures, we further construct four types of complicated query structures with more relations and geometric operations, including 4p, 5p, 3ip and i2p, based on the query structures in test sets. The detailed illustrations and explanations of these query struc-
tures are given in Appendix C. We directly evaluate our method on these more complicated queries in both inductive and transductive datasets FB15k237-V2 and FB15k-237. The results are listed in Table 6. We can see that compared to seen and unseen query structures in the original datasets in Table 2 and 4, our method can also generalize to these complicated structures with more logical operations and achieves impressive performance. This demonstrates that the design of structural prompt and implicit geometric operation modeling can effectively learn structure knowledge and improve query structure generalizability.

heading: 4 Related Work
text: Answering first-order logic queries over incomplete KGs is a challenging task (Guu et al., 2015). Previous research (Lin et al., 2015; Hamilton et al., 2018) mainly studies transductive logical reasoning, where the training and testing are performed on the same KG. Embedding-based methods are widely used to embed both logical queries and entities into a joint low-dimensional vector space and push answer entities and queries to be close enough, enabling answer prediction through embedding similarity, even when the involved relation is absent. Following this paradigm, some further propose extending the embedding of the query/entity from a single point to a region (Ren et al., 2020; Zhang et al., 2021; Choudhary et al., 2021b) or probabilistic distribution (Ren and Leskovec, 2020; Choudhary et al., 2021a) over vector space to map arbitrary first-order logic queries into sets of answer entities. However, these methods are unable to tackle the inductive problem, which requires generalizing to unseen entities and relations. Although (Hu et al., 2022) proposes enriching the entity and relation embedding with type information for inductive logical reasoning, it can only generalize to elements of observed types.
Another line of research focuses on inductive
logical reasoning over KGs using textual encoding methods. With the advance of large-scale pre-trained language models (Devlin et al., 2018; Liu et al., 2019), these methods propose transforming the graph structures into linearized text and utilize PLMs for encoding (Yao et al., 2019; Zha et al., 2022). With the strong generalizability of PLMs, they can easily generalize to unseen entities/relations, but struggle to model structure knowledge during text encoding. Some works (Wang et al., 2021b; Daza et al., 2021; Wang et al., 2021a) propose to follow TransE (Bordes et al., 2013) to apply the translation function between entity and relation representations for geometric structure learning. Nevertheless, these methods usually require descriptions of entities and relations for encoding and assume these descriptions are readily available. Besides, they only focus on simple link prediction tasks without exploring complex structure modeling in logical reasoning, which is essential for generalizing within the same query structure/type. We thus propose to simultaneously encode the linearized query and preserve the logical structure knowledge by structural prompt and separate geometric operation modeling for inductive logical reasoning over KGs.

heading: 5 Conclusion
text: In this paper, we present the first flexible inductive method for answering complex logical queries over incomplete KGs which can generalize to any unseen entities and relations. To accomplish this goal, we propose a structure-model textual encoding model that utilizes PLMs to encode linearized query structures to find the answer entities. For structure modeling of complex queries, we design structural prompts to implicitly indicate PLMs the order of geometric operations execution in each query, and separately model three geometric operations on representation space using a pre-trained encoder, an attention layer, and a maxout layer. Experimental results demonstrate the effectiveness of our model on logical reasoning over KGs in both inductive and transductive settings.
Limitations
This study has potential limitations. First, it only focuses on answering existential positive first-order logic queries but does not support the negation operation. We will later address this limitation by modeling the negation operation. Second, we uti-
lize BERT as the backbone model for inductive generalization due to computing resource limits. We plan to investigate the use of more powerful pre-trained language models with stronger generalizability in future research to improve inductive logical reasoning over KGs.

heading: Acknowledgments
text: This work is supported by National Natural Science Foundation of China (No. 6217020551) and Science and Technology Commission of Shanghai Municipality Grant (No.21QA1400600).

heading: A Structure Linearization Templates
text: In this part, we list the detailed linearization templates of different structural query types in Table 7.

heading: B Data Statistics
text: In Table 8, we summarize the statistics of logical queries in our experimented datasets for both inductive and transductive settings.

heading: C Constructed Query Types
text: We here introduce our generated more complicated query types involving more relations and logical operations, which are used to illustrate the complicated query structure generalizability.
[4p] q = v?.∃v1, v2, v3 : r1(e1, v1) ∧ r2(v1, v2) ∧ r3(v2, v3) ∧ r4(v3, v?) [5p] q = v?.∃v1, v2, v3, v4 : r1(e1, v1) ∧ r2(v1, v2) ∧ r3(v2, v3) ∧ r4(v3, v4) ∧ r5(v4, v?)
[3ip] q = v?.∃v : r1(e1, v) ∧ r2(e2, v) ∧ r3(e3, v) ∧ r4(v, v?) [i2p] q = v?.∃v1, v2 : r1(e1, v1) ∧ r2(e2, v1) ∧ r3(v1, v2) ∧ r4(v2, v?)
(5)
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
3 A4. Have you used AI writing assistants when working on this paper? Use Grammarly and ChatGPT for spelling checking and polishing.
B 3 Did you use or create scientific artifacts? Section 3.1 & 3.2
3 B1. Did you cite the creators of artifacts you used? Section 3.1 & 3.2
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3.1 & 3.2
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section B
C 3 Did you run computational experiments? Section 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We plan to investigate the use of more powerful pre-trained language models with stronger generalizability in future research to improve inductive logical reasoning over KGs