KGQA Without Retraining
Year: 2023

Abstract:
Popular models for Knowledge Graph Question Answering (KGQA), including semantic parsing and End-to-End (E2E) models, decode into a constrained space of KG relations. Although E2E models accommodate novel entities at test-time, this constraint means they cannot access novel relations, requiring expensive and time-consuming retraining whenever a new relation is added to the KG. We propose KG-Flex, a new architecture for E2E KGQA that instead decodes into a continuous embedding space of relations, which enables use of novel relations at test-time. KG-Flex is the first to support KG updates with entirely novel triples, free of retraining, while still supporting end-to-end training with simple, weak supervision of (Q, A) pairs. Our architecture saves on time, energy, and data resources for retraining, yet we retain performance on standard benchmarks. We further demonstrate zero-shot use of novel relations, achieving up to 82% of baseline hit@1 on three QA datasets. KG-Flex can also fine-tune, requiring significantly shorter time than full retraining; fine-tuning on target data for 10% of full training increases hit@1 to 89-100% of baseline.

text: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 212–218 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Knowledge Graph Question Answering (KGQA) is the task of answering questions using facts in a Knowledge Graph (KG). Common approaches to KGQA include semantic parsing (Rongali et al., 2020) and End-to-End (E2E) Question Answering techniques (Cohen et al., 2020). E2E approaches are promising due to being composed of entirely differentiable operations, including program prediction and execution using a Differentiable KG (DKG), and the ease of training with simple (question, answer) pairs. However, these methods decode into a constrained space of KG relations which are then used to traverse the KG. While this works well for benchmark datasets where the KGs
are static, it fails to scale to real use cases where KGs are frequently updated.
For example, Wikidata (Vrandečić and Krötzsch, 2014) is a commonly used Knowledge Graph that is actively updated. Between March 2022 and March 2023, the number of reported properties (relations in Wikidata) increased from 9.7K to 10.9K. At the time of writing, there are almost 200 new properties proposed for addition, including relations about new platforms or services (e.g., Patreon user ID, Peacock ID), and relations improving existing ontologies (e.g., Pokemon category, alternate universe counterpart).
In order to handle new relations, most KGQA methods require full retraining to learn a new output space of possible relations. These methods also require additional training data with examples using the new relations. We argue that incrementally updating the KG should not require full model retraining, a mostly redundant process which is energy- and time-intensive.
We present KG-Flex, an E2E model architecture that overcomes this problem by instead decoding
212
into an open embedding space in which relations are expressed in natural language. Given an input question, the model predicts the answer relation embedding, and available triples in the KG are scored against the prediction via their relations (example in Figure 1). KG-Flex is the first End-to-End KGQA model that allows updates to both KG entities and relations at test-time, without retraining.
We show that KG-Flex retains performance on standard benchmarks compared to a similar model, while demonstrating additional capabilities. In a zero-shot setting in three QA datasets, KG-Flex scores between 40-82% of baseline hit@1 on questions using relations that were held out during training time, a task which is impossible for previous models using a fixed decoder. Further, by finetuning for 10% of full training, scores are increased to 89-100% of baseline.

heading: 2 Related Work
text: Traditional approaches to KGQA involve semantic parsing of natural language into logical forms. Semantic parsing models use a constrained decoder over the output space of symbols. Since models such as Rongali et al. (2020) treat relations as whole symbols, adding new relations requires increasing the decoder output size and retraining the model. Further, collecting new training data of natural language to logical forms (e.g., SPARQL) is expensive (Finegan-Dollak et al., 2018).
Other techniques transform queries and KG triples to an embedding space (Saxena et al., 2020; Sun et al., 2020). These methods do not require annotated KG queries, however, adding new relations requires retraining to update the model.
Recent End-to-End methods for KGQA (Cohen et al., 2020; Sen et al., 2021) are weakly supervised with (question, answer) pairs which, conditioned on a question, predict a probability distribution over KG relations. Execution on the KG involves following relations and returning probabilistically weighted answer entities. While E2E methods also do not require supervision of KG pathways, they still constrain relation decoding, which means that adding new relations requires retraining.
Previous work in expanding the flexibility of KGQA models at test-time include Ravishankar et al. (2021), using a two-step process of first predicting an intermediate query template, then adding KG-specific relations. However, this method still relies on expensive SPARQL query annotation for
training data. Oguz et al. (2022) propose a method to unify structured and unstructured data sources by converting them all into text, however this sacrifices the useful structure of Knowledge Graphs.

heading: 3 KG-Flex
text: We introduce KG-Flex, a novel KGQA architecture that is designed for unconstrained relation decoding, and can be trained end-to-end using only weak supervision of questions and answers. KG-Flex is an encoder-decoder model using a Differentiable Knowledge Graph, in the family of End-to-End KGQA models such as ReifKB (Cohen et al., 2020) and Rigel-based models (Sen et al., 2021; Saffari et al., 2021). We make key changes in the decoder to enable the use of new relations at test-time. KGFlex has 4 key stages.

heading: 3.1 Precompute KG Relation Embeddings
text: Ahead of train- or test-time, we pre-compute vector embeddings for all relations available in KG triples (in training and test, these are frozen). To do so, every KG relation ri ∈ R is lightly preprocessed into natural language and encoded as a vector ri ∈ Rh, h = 768, using the RoBERTa-base v2 Sentence Transformer (Reimers and Gurevych, 2019).
• Freebase (Bollacker et al., 2008) property IDs are preprocessed into the template “(type) property”, e.g., film.film_festival.location ⇒ “(film festival) location”
• Wikidata property label text is already in natural language, e.g., “place of birth”
• MetaQA relations have underscores replaced with spaces, e.g., directed_by ⇒ “directed by”

heading: 3.2 Encode Question Text
text: At train- and test-time, a natural language question is encoded with RoBERTa-base v2 Sentence Transformer, into a vector representation q ∈ Rh. This is similar to earlier E2E models like Rigel (Sen et al., 2021), which uses a RoBERTa encoder.

heading: 3.3 Decode Relation Embedding
text: The decoder is the key improvement in KG-Flex, which predicts from amongst the relations available in the KG, unconstrained from a fixed schema. Conditioned on the question encoding q, the KGFlex decoder predicts an embedding for a relation which answers the question. All available relations are scored based on their Euclidean distance to
the prediction, and these scores are converted to a probability distribution via a softmax.
As in Sen et al. (2021) and Saffari et al. (2021), the decoding step is performed for T “hops” in the KG, where the hyperparameter T is fixed before training1. An attention mechanism is jointly learned as part of the model which predicts how many hops (up to T ) is required for a given question, conditioned on q. This is used to weight final entity predictions. For example, answering “What’s the mascot of Obama’s alma mater?” requires two hops from the entity Obama: first the ALMA-MATER relation, then MASCOT, so entities fetched in hop 2 will be weighted most heavily.
For each hop t ∈ [1, T ], we apply a decoder transformation Dt ∈ R(th)×h (with bias bt ∈ Rh). Dt predicts a relation embedding zt ∈ Rh, given the encoded question vector q ∈ Rh and any predictions of earlier hops z<t.
z1 = tanh(qD1 + b1) z2 = tanh([q; z1]D2 + b2) z3 = tanh([q; z1; z2]D3 + b3)
The decoder is trained to predict a relation embedding which minimizes the Euclidean distance (L2 norm) to the relation path that leads to the answer entity. Each decoder hop produces a probability distribution dt over relations ri ∈ R by a softmax over negated distances from zt to precomputed ri:
dt,i = e−||zt−ri||2
∑|R| j=0 e −||zt−rj ||2

heading: 3.4 Execute on DKG
text: As an E2E model, KG-Flex executes a probabilistic query over its Differentiable KG (DKG) to produce
1We assume that T = 3 hops is sufficient to cover all realistic human questions.
weighted answer entities; these are scored to feed back the training signal through the model.
A DKG is just a re-representation of a KG as three matrices. In hop t, given a distribution over subjects et and relations dt, the “follow” operation (Cohen et al., 2020) computes a probability distribution over KG triple objects et+1 using simple matrix multiplication:
et+1 = follow(et,dt)
In the first hop, e1 is a one-hot vector of KG entities (question entity set to 1) 2. For each hop 1 ≤ t ≤ T , a probability distribution is predicted over KG entities, which are fed into the subsequent hop. The final model prediction is a distribution over KG entities discovered in all hops, weighted by the hop attention mechanism. During training, entity predictions are compared to the gold label entities via binary cross-entropy loss, and updates are backpropagated through the decoder and encoder.

heading: 4 Experiments
text: In our experiments, we use three datasets: SimpleQuestions (Bordes et al., 2015), a large-scale dataset of simple, one-hop questions based on FreeBase; WebQuestionsSP (Yih et al., 2016), a dataset of natural language questions containing up to 2 hops linked to FreeBase; and MetaQA (Zhang et al., 2018), a movies QA dataset divided into one, two, and three-hop subsets. MetaQA uses a KG that is internal to the dataset.
KG-Flex models are trained until dev set convergence or max 40,000 steps on a single NVIDIA Tesla V100 GPU (see Appendix A for details).
2Like Cohen et al. (2020) and Sen et al. (2021), we begin with question entities pre-identified in the datasets.

heading: 4.1 Standard Benchmarks
text: First, we benchmark KG-Flex against existing methods on standard datasets. We report hit@1 scores, a metric that measures the percentage of questions where the highest probability entity predicted is correct3. Results are shown in Table 1. Compared to similar E2E models like Rigel, KGFlex attains competitive performance within 3 percentage points.

heading: 4.2 Zero-Shot Transfer to Held-out Relations
text: We simulate a real-world scenario where new KG domains are added after training. We demonstrate how KG-Flex can predict using these held-out relations, an impossible task for prior E2E models like Rigel and ReifKB.
In training, we remove a subdomain of relations from the KG and all questions involving those relations from train and dev sets. Then at test-time we reintroduce the relations to the KG and include the held-out questions. We report on the full test set as well as the subset consisting of just the heldout questions. For SimpleQuestions, we remove all relations in the domains Film (61 relations) or Medicine (66 relations). For WebQuestions, we remove Film (24 relations) or Sports (45 relations). These domains represent a reasonably-sized KG update (< 10% of total relations). Since MetaQA contains only movie questions, we remove the “directed_by” or “written_by” relations (further info in Appendix B).
3Since SimpleQuestions may have multiple correct answers, we count predictions as correct if any entity within a tie for most probable is correct.
Results are shown under Zero-Shot columns in Table 2. We compare to the Baseline KG-Flex results, which refer to training on the full dataset. On the held-out datasets the zero-shot models score up to 82% of the baseline score on simpler datasets (81.4 achieved / 98.9 baseline on written_by in MetaQA 1-hop; 60.1 / 73.3 on film in SimpleQuestions). However, they perform worse on more complex questions, reaching as low as 40% of the baseline on multi-hop question datasets (36.9 / 86.2 on directed_by in MetaQA 3-hop; 31.3 / 76.8 on film in WebQuestions). We attribute this to the compounding likelihood of error when predicting multiple relations at once. Since comparable models would score 0% on this task, we still consider this to be a valuable step forward.

heading: 4.3 Fine-tuned Transfer to Held-out Relations
text: To improve transfer to new relations, we further fine-tune the zero-shot models. Each zero-shot model is fine-tuned for 4,000 steps (10% of full training) on the entire dataset, including held-out relations. Results are under Fine-tuned in Table 2.
Our fine-tuned models are able to recapture between 89-100% of baseline KG-Flex performance on both the full test set and heldout test set. We
are able to fine-tune our model because we have an unconstrained decoder space, whereas comparable models such as ReifKB (Cohen et al., 2020) would require retraining from scratch. By fine-tuning for only a fraction of the full training time (see Table 3), we demonstrate that KG-Flex can efficiently adapt to new relations.

heading: 5 Conclusions
text: We present KG-Flex, a new model architecture for KGQA which is the first to use an unconstrained decoder over KG relations and to train end-to-end using simple (question, answer) pairs. While maintaining performance on benchmarks, KG-Flex is demonstrated to make use of incrementally changing live KGs without requiring expensive retraining. We show that KG-Flex uses novel relations added at test-time, handling simple questions with new relations in zero-shot, and handling more complex multihop questions by fine-tuning for only 10% of the training steps required for full retraining.

heading: 6 Limitations
text: We present KG-Flex, an end-to-end model that can access new relations at test-time without retraining. Our current KG-Flex model does not perform entity resolution, and so we rely on resolved entities provided by the datasets. However, resolved entities may not always be available, so tools such as automatic entity recognition may be necessary. While it is possible for end-to-end models to jointly learn to resolve entities in questions before relation following (Saffari et al., 2021), we consider this outside the scope of this focused work.
Additionally, KG-Flex is limited in the kinds of reasoning it can do over a Knowledge Graph. Currently, KG-Flex only performs relation following, so it cannot handle questions which require complex reasoning like counts, comparatives, min/max, etc, such as “Who is the tallest NBA player?” We hope to address this in future work.
Further, we test KG-Flex on popular datasets representing possible real human questions. However, we do not deeply investigate the semantic properties of these questions. Notably, McKenna and Steedman (2022) show that searching for similar relations in embedding space (as done in KGFlex) may work better for paraphrastic inference, and only in certain cases for directional inference where semantic precision matters, e.g. DEFEAT entails PLAY, but PLAY does not entail DEFEAT. We
leave deeper investigation of KG-Flex semantics and edge cases to future work.

heading: A Training Specifications
text: A.1 Hardware We performed our experiments on one AWS EC2 instance of p3.2xlarge, which is equipped with one NVIDIA Tesla V100 GPU (16 GiB memory).
A.2 Hyperparameters We train and test a KG-Flex model constructed to the specifications shown in Table 4.

heading: B Added Domains
text: Our experiments in §4.2 use expert KGs containing specific domains of Freebase. We show summary information for the domains in Table 5.

heading: Future Work
text: Currently, KG-Flex only performs relation following, so it cannot handle questions which require complex reasoning like counts, comparatives, min/max, etc, such as “Who is the tallest NBA player?” We hope to address this in future work. Further, we test KG-Flex on popular datasets representing possible real human questions. However, we do not deeply investigate the semantic properties of these questions. Notably, McKenna and Steedman (2022) show that searching for similar relations in embedding space (as done in KGFlex) may work better for paraphrastic inference, and only in certain cases for directional inference where semantic precision matters, e. g. DEFEAT entails PLAY, but PLAY does not entail DEFEAT. We
leave deeper investigation of KG-Flex semantics and edge cases to future work