Knowledge Graph Simple Question Answering for Unseen Domains
Year: 2020

Abstract:
Knowledge graph simple question answering (KGSQA), in its standard form, does not take into account that human-curated question answering training data only cover a small subset of the relations that exist in a Knowledge Graph (KG), or even worse, that new domains covering unseen and rather different to existing domains relations are added to the KG. In this work, we study KGSQA in a previously unstudied setting where new, unseen domains are added during test time. In this setting, question-answer pairs of the new domain do not appear during training, thus making the task more challenging. We propose a data-centric domain adaptation framework that consists of a KGSQA system that is applicable to new domains, and a sequence to sequence question generation method that automatically generates question-answer pairs for the new domain. Since the effectiveness of question generation for KGSQA can be restricted by the limited lexical variety of the generated questions, we use distant supervision to extract a set of keywords that express each relation of the unseen domain and incorporate those in the question generation method. Experimental results demonstrate that our framework significantly improves over zero-shot baselines and is robust across domains.

heading: 1. Introduction
text: Large-scale structured Knowledge Graphs (KGs) such as Freebase [Bollacker et al., 2008] and Wikidata [Pellissier Tanon et al., 2016] store real-world facts in the form of subject–relation– object triples. KGs are being increasingly used in a variety of tasks that aim to improve user experience [Bota et al., 2016]. One of the most prominent tasks is Knowledge Graph Simple Question Answering (KGSQA), which aims to answer natural language questions by retrieving KG facts [Yih et al., 2015]. In practice, many questions can be interpreted by a single fact in the KG. This has motivated the KGSQA task [Bordes et al., 2015, Mohammed et al., 2018, Petrochuk and Zettlemoyer, 2018], which is the focus of this paper. In KGSQA, given a simple question, e.g. “who directed the godfather?”, the system should interpret the question and arrive at a single KG fact that answers it: (The Godfather (film), film.film.directed by, Francis Ford Coppola).
KGSQA systems are trained on manually annotated datasets that consist of questionfact pairs. In practice, the applicability of such systems in the real-world is limited by two factors: (i) modern KGs store millions of facts that cover thousands of different relations, but KGSQA training datasets can only cover a small subset of the existing relations in the KG [ElSahar et al., 2018], and (ii) KGs are dynamic, i.e. they are updated with new domains that cover new relations [Pellissier Tanon et al., 2016]. Solving (i) and (ii)
by exhaustively gathering question-fact pair annotations would be prohibitively laborious, thereby we need to rely on automatic methods.
Motivated by the above, in this work, we study the KGSQA task in a setting where we are interested in answering questions about a new, unseen domain that covers relations, for which we have instances in the KG, but we have not seen any question-fact pair during training. We model this as a domain adaptation task [Mansour et al., 2009, Pan and Yang, 2010] and propose a data-centric domain adaptation framework to address it. Datacentric domain adaptation approaches focus on transforming or augmenting the training data, instead of designing specialized architectures and training objectives as model-centric domain adaptation approaches do [Chu and Wang, 2018]. Our framework consists of: (a) a KGSQA system which can handle the unseen domain, and (b) a novel method that generates training data for the unseen test domain.
The KGSQA system we introduce performs mention detection, entity candidate generation and relation prediction on the question, and finally selects the fact that answers the question from the KG. To improve relation prediction on questions that cover relations of the unseen domain, we automatically generate synthetic questions from KG facts of the unseen domain (i.e. knowledge graph question generation – QG). The resulting synthetic question-fact pairs are used to train the KGSQA system for the unseen domain. We find that the effectiveness of QG for KGSQA can be restricted not only by the quality of the generated questions, but also by the lexical variety of the questions. This is because users ask questions underlying the same relation using different lexicalizations (e.g. “who is the author of X”, “who wrote X”). To address this, we use distant supervision to extract a set of keywords for each relation of the unseen domain and incorporate those in the question generation method.
Our main contributions are the following: (i) we introduce a new setting for the KGSQA task, over new, previously unseen domains, (ii) we propose a data-centric domain adaptation framework for KGSQA that is applicable to unseen domains, and (iii) we use distant supervision to extract a set of keywords that express each relation of the unseen domain and incorporate them in QG to generate questions with a larger variety of relation lexicalizations. We experimentally evaluate our proposed method on a large-scale KGSQA dataset that we adjust for this task and show that our proposed method consistently improves performance over zero-shot baselines and is robust across domains.1

heading: 2. Problem Statement
text: Let E denote the set of entities and R the set of relations. A KG K is a set of facts (es, r, eo), where es, eo ∈ E are the subject and object entities respectively, and r ∈ R is the relation between them. Each relation r has a unique textual label rl and falls under a single domain D. For instance, music.album.release type and music.artist.genre fall under the Music domain. Simple questions mention a single entity and express a single relation. For instance, the question “who directed the godfather?” mentions the entity “The Godfather” and expresses the relation film.film.directed by.
Given a simple question q that consists of a sequence of tokens t1, t2, . . . , tT , the KGSQA task is to retrieve a fact (ês, r̂, êo), where (ês, r̂) accurately interprets q (i.e., ês is mentioned
1. Our code is available at https://github.com/GSidiropoulos/kgsqa_for_unseen_domains.
in q and r̂ is expressed in q) while êo provides the answer to q. In our setting, we aim to build a KGSQA system that can perform well on a previously unseen domain. A domain is “unseen” when facts that cover relations of that domain do exist in K, but gold-standard question-fact pairs of that domain do not appear in the training data. This setting is an instance of domain adaptation, where a model is trained on data S, which is drawn according a source distribution, and tested on data T coming from a different target distribution. Domain adaptation over KG domains is more challenging compared to domain adaptation over single KG relations [Yu et al., 2017, Wu et al., 2019], because it is less likely for relations with similar lexicalizations to appear in the training set.

heading: 3. KGSQA system
text: In this section, we detail our KGSQA system. In Section 4, we will describe how we generate synthetic training data to make this system applicable to unseen domains. Following current state-of-the-art on KGSQA [Petrochuk and Zettlemoyer, 2018], we split the task into four sub-tasks, namely, entity mention detection (MD), entity candidate generation (CG), relation prediction (RP), and answer selection (AS). The skeleton of our KGSQA system generally follows previous work, and we modify the MD and RP architectures.
Mention Detection (MD) Given the question q, MD outputs a single entity mention m in q, where m is a sub-sequence of tokens in q. We model this problem as sequence tagging, where given a sequence of tokens, the task is to assign an output class for each token [Huang et al., 2015, Lample et al., 2016]. In our case, the output classes are entity (E) and context (C). For instance, the correct output for the question “who directed the godfather?” is “[C C E E]”. We use a BiLSTM with residual connections (R-BiLSTM) [He et al., 2016], since it outperformed vanilla RNN, BiRNN, and a CRF on top of a BiRNN [Petrochuk and Zettlemoyer, 2018] in preliminary experiments.
Candidate Generation (CG) Given the mention m extracted from the previous step, CG maps m to a set of candidate entities CS ⊂ E. For instance, CG maps the mention “the godfather” to the entities { The Godfather(film), The Godfather(book). . . }. The CG method we use was proposed in Türe and Jojic [2017]. Briefly, the method pre-builds an inverted index I from n-grams of mentions to entities, and it looks-up the n-grams of m in I to obtain CS .
Relation Prediction (RP) Given the question q and the set of entities Cs extracted in the previous step, RP outputs a single relation r̂ ∈ R that is expressed in q. Previous work models RP as a large-scale multi-label classification task where the set of output classes is fixed [Petrochuk and Zettlemoyer, 2018]. In our domain adaptation scenario, however, we want to be able to predict relations that we have not seen during training. Therefore, we model RP as a relation ranking task, as in [Yu et al., 2017], and use the textual label rl to represent the relation r (instead of using a categorical variable). This way we can in principle represent any relation r ∈ R during inference time. Below we describe the architecture we use for RP and how we perform training and inference. Our architecture is a simpler version of [Yu et al., 2017], where they model a relation both as a sequence and a categorical variable, and they use more complex sequence encoders.
First we describe how we encode the question q and the relation r. In order to generalize beyond specific entity names, we first replace the previously detected entity mention m in q with a placeholder token; e.g “who directed SBJ”. We then map each term to its embedding and feed the word embeddings to an LSTM; embeddings are initialized with pretrained word2vec embeddings [Mikolov et al., 2013]. The final hidden state of the LSTM γ(q) is used as the encoding of the question. In order to represent r, we use its label rl (e.g. film.film.directed by). Similarly with the question encoding, we encode rl with an LSTM to obtain γ(r). However, since questions and relations significantly differ both grammatically and syntactically, the two LSTM encoders do not share any parameters. The ranking function f is calculated as f(q, r) = cos (γ(q),γ(r)), where cos(·) is the cosine similarity.
We train f using standard pairwise learning to rank. The loss is defined as follows:
L(θ) = ∑ r ∑ r′∈R′ max(0, µ− f(q, r) + f(q, r′)), (1)
where θ are the parameters of the model, µ is a hyperparameter, and R′ is the set of sampled negative relations for a question q. We design a specialized negative sampling method to select R′. With probability P−R we uniformly draw a sample from R
− = {r′|r′ ∈ R∧r′ 6= r}; the set of all available relations except the positive relation r. With probability 1− P−R we draw a random sample from R̂− = {r′|r′ ∈ D+R ∧ r′ 6= r}; the set of relations that are in the same domain as the positive relation r. This way, we expose the model to conditions it will encounter during inference. At inference time, given a question q and a set of relations we score all question-relation pairs (q, r) with f and select the relation r̂ with the highest score. Unfortunately, computing a score with respect to all possible relations in R leads to poor performance when there is no linguistic signal to disambiguate the choice. In order to address this issue, we constrain the set the potential output relations Rc to be the union of the relations expressed in the facts where the entities in CS participate in [Petrochuk and Zettlemoyer, 2018]. Formally, we define the target relation classes to be Rc = {r ∈ R|(es, r, eo) ∈ K ∧es ∈ CS}. For example, given the question “who directed the godfather”, the potential relations are { film.film.directed by, book.written work.author, . . . }. Using the aforementioned constraint we can safely ignore relations like tv.tv series episode.director by taking into account that The Godfather does not appear in any tv-related facts.
Answer Selection (AS) Given the set of entities CS obtained from CG, and the top ranked relation r̂ obtained from RP, AS selects a single fact (ês, r̂, êo), where êo answers the question q. The set of candidate answers may contain more than one facts (e′s, r̂, e ′ o), where ∀e′s ∈ CS . Since there is no explicit signal on which we can rely to disambiguate the choice of subject, all the potential answers are equally probable. We therefore use a heuristic based on popularity, introduced by Mohammed et al. [2018]: we choose ês to be the entity that appears the most in the facts in K either as a subject or as an object. Having ês and r̂ we can retrieve the fact (ês, r̂, êo). For our running example (“who directed the godfather”), given film.film.directed by (from RP) and entities { The Godfather(film), The Godfather(book). . . } (from CG) we can select the fact (The Godfather (film), film.film.directed by, Francis Ford Coppola).

heading: 4. KGSQA to unseen domains using question generation
text: Even though all the components of the aforementioned KGSQA system were designed to work with unseen domains, preliminary experiments demonstrated that RP does not generalize well to questions originating from unseen domains. This is expected since RP is a large-scale problem (thousands of relations), and it is very challenging to model less frequent or even unknown relations that are expressed with new lexicalizations.
We therefore focus on improving RP for questions originating from unseen domains. Inspired from the recent success of data-centric domain adaptation in neural machine translation [Chu and Wang, 2018], we perform synthetic question generation from KG facts of the unseen domain to generate question-fact pairs for training the RP component (see Section 3).2 In the remainder of this section we briefly describe the base question generation (QG) model we build upon and how we augment the model to more effectively use textual evidence and thus better generalize to relations of the unseen domain.

heading: 4.1 Base model for QG
text: Given a fact (es, r, eo) from the target domain, QG aims to generate a synthetic question q̂. During training, only question-fact pairs from the known domains are used. Our base model is the state-of-the-art encoder-decoder architecture for QG [ElSahar et al., 2018]. It takes as input the fact (es, r, eo) alongside with a set of textual contexts C = {cs, cr, co} on the fact. Those textual contexts are obtained as follows: cs and co are the types of entities es and eo respectively, whereas cr is a lexicalization of the relation r obtained by simple pattern mining on Wikipedia sentences that contain instances of r. For instance, given the fact (The Queen Is Dead, music.album.genre, Alternative Rock), the textual contexts are: cs = {“album”}, cr = {“album by”} and co = {“genre”}.
The encoder maps es, r and eo to randomly initialized embeddings and concatenates those to encode the whole fact. Also, it encodes the text in cs, cr and co separately using RNN encoders. The decoder is a separate RNN that takes the representation of the fact and the RNN hidden states of the textual contexts to generate the output question q̂. It relies on two attention modules: one over the encoded fact and one over the encoded textual contexts. The decoder generates tokens not only from the output vocabulary but also from the input (using a copy mechanism) to deal with unseen input tokens.

heading: 4.2 Using Richer Textual Contexts for QG
text: The role of the textual contexts C in the aforementioned base model is critical, since it enables the model to provide new words/phrases that would have been unknown to the model otherwise [ElSahar et al., 2018]. Even though the base model generally generates high quality questions, in our task (KGSQA), we aim to generate a larger range of lexicalizations for a single relation during training in order to generalize better at test time. This is because users with the same intent may phrase their questions using different lexicalizations (e.g. “who is the author of X”, “who wrote X”). Thus, in this section we focus on how to provide
2. Note that Dong et al. [2017] also performed QG for improving the overall KGSQA performance. However, their model is not applicable to our domain adaptation scenario since their model relies on modifying existing questions and all domains were predefined.
the model with a diverse set of lexicalizations for a relation r instead of a single one as in the base model, in order to be able to generate a more diverse set of questions in terms of relation lexicalizations. More precisely, given a relation r, we extract k keywords that will constitute the relation’s textual context cr. To this end, we first extract a set of candidate sentences Sr that express a specific relation r between different pairs of entities. Second, we extract keywords from the set Sr, rank them and select the top-k keywords that constitute the set cr. We detail each of these steps below.
Extracting sentences Given a set of facts Fr of relation r between different pairs of entities, we aim to extract a set of sentences Sr, where each sentence s ∈ Sr expresses a single fact (es, r, eo) in Fr [Voskarides et al., 2015]. For this, for each fact (es, r, eo) in Fr, we need to (a) extract a set of candidate sentences S that might express (es, r, eo) and (b) select the sentence that best expresses the relation. For (a), we collect the set of sentences S using distant supervision, similarly to [Mintz et al., 2009]: S consists of sentences that mention eo in the Wikipedia article of es and sentences that mention es in the Wikipedia article of eo. For (b), we score each sentence s ∈ S w.r.t. the label rl of the relation r using the cosine similarity cos(e(s), e(rl)), where cos(·) is the cosine similarity and e(x) is calculated as e(x) = (1/|x|) ∑ t∈xwt, where wt is the embedding of word t. Finally, we take the sentence s′ with the highest score and add it to the set Sr.
Extracting keywords After extracting the set of sentences Sr, we aim to extract the set of keywords cr. For this, we treat Sr as a single document and score each word t that appears in Sr using tf-idf, score(t) = tf(t, Sr) · idf(t, SR), where SR is the union of all Sr′ , r
′ ∈ R. The top-k scoring words constitute the set of keywords cr. Table 1 depicts example keywords generated by the procedure described above.
The keyword extraction approach described above is conceptually simple yet we later show that it significantly improves upon the base model when applied to KGSQA.

heading: 5. Experimental Setup
text: In this section, we discuss how we design the experiments to answer the following research questions: RQ1) How does our method for generating synthetic training data for the unseen domain perform on RP compared to a set of baseline methods? RQ2) How does our full method perform on KGSQA for unseen domains compared to state-of-the-art zero-shot datacentric methods? RQ3) How does our data-centric domain adaptation method compare to a state-of-the-art model-centric method on RP?
Dataset In our experiments we use the SimpleQuestions dataset, which is an established benchmark for studying KGSQA [Bordes et al., 2015]. The dataset consists of 108,442
questions written in natural language by human annotators, paired with the ground truth fact that answers the question. The ground truth facts originate from Freebase [Bollacker et al., 2008]. The dataset covers 89,066 unique entities, 1,837 unique relations and 82 unique domains. In our setup, we leave one domain out to simulate a new, previously unseen domain, and train on the rest. We choose six challenging domains as target domains: Film, Book, Location, Astronomy, Education and Fictional Universe; the first three are among the largest domains and the last three are medium-sized. The aforementioned domains are challenging because they have very low overlap in terms of relation lexicalization w.r.t. the rest of the domains used as source domains. The training data consists of the questionfact pairs that appear in the source domains, augmented with synthetically generated data of the target/unseen domain. In practice, we replace all questions from the target domain that initially appear in the full training set with their corresponding synthetically generated questions.3 As a source of text documents for the textual context collection (see Section 4.2), we use Wikipedia articles augmented with dense entity links provided by DAWT [Spasojevic et al., 2017].
Baselines To answer RQ1, we keep the KGSQA system unchanged and alternate the way of generating synthetic questions. We compare the RP performance on the unseen domain given the following ways of generating synthetic data of the unseen domain: (i) No synthetic data, (ii) Wiki-raw-sentences: uses the raw Wikipedia sentence that expresses the ground truth fact that answers the question (automatically extracted using the procedure in Section 4.2), and (iii) the state-of-the-art QG method proposed in ElSahar et al. [2018]. To answer RQ2, we replace our RP component with two state-of-the-art RP models: (i) Petrochuk and Zettlemoyer [2018], which uses a BiLSTM to classify relations, and (ii) Yu et al. [2017], a zero-shot RP model that uses a HR-BiLSTM and is specifically designed to deal with unseen or less frequently seen relations. To answer RQ3, we compare the performance of our data-centric model on RP against a state-of-the-art model-centric zero-shot approach [Wu et al., 2019]: it the HR-BiLSTM proposed by Yu et al. [2017] with an adversarial adapter combined and a reconstruction loss. The adapter uses embeddings trained on Freebase and Wikipedia by JointNRE [Han et al., 2018] and learns representations for both seen and unseen relations.
Evaluation metrics We run the experiments three times and report the median (only marginal and not significant differences were found among different runs) [Mohammed et al., 2018]. In contrast to the classic KGSQA where the task is to retrieve a single entity, it is standard practice when using the SimpleQuestions dataset to treat the problem as question interpretation [Petrochuk and Zettlemoyer, 2018]. More specifically, the objective is to rewrite the natural language question in the form of subject-relation pair. We evaluate our overall approach in terms of top-1 accuracy, i.e. whether the retrieved subject-relation pair matches the ground truth. We measure accuracy both at a macro- (domains) and at a micro-level (samples). Statistical significance is determined using a paired two-sided t-test.
3. One may hypothesize that since entities can appear in multiple domains (e.g. actors who are also singers), question generation becomes an unrealistically simple task. However, this is not the case because in our dataset, the entity overlap between seen and unseen domains is only 4.6%.
Parameter configurations We initialize word embeddings with pretrained Google News 300-dimensional embeddings [Mikolov et al., 2013]. We use the Adam optimizer [Kingma and Ba, 2014]. Our MD model consists of 2 hidden layers, 600 hidden units, 0.4 dropout rate, frozen embeddings, and learning rate of 10−3; 50 training epochs. For the RP model, we use 1 layer encoder for both questions and relations that consists of 400 hidden units, with a frozen embedding layer, and a learning rate of 10−3; trained for 10 epochs. During training, we sample 10 negative questions per question using the procedure described in Section 3. We use a batch size of 300 and 200 for MD and RP respectively. For the QG model [ElSahar et al., 2018] and the model-centric RP [Wu et al., 2019] model we compare against, we use the hyperparameters as presented in their work. Note that for both our method and the baselines, the hyperparameters were tuned on the initial split of the SimpleQuestions dataset. We keep the parameters fixed for both our method and the baselines for all source-target domain setups. We set the number of keywords for each relation k = 10 (Section 4.2).

heading: 6. Results and Discussion
text: In this section we present and discuss our experimental results. All models under comparison have all their components fixed, except RP. Therefore, any improvement observed, is due to RP.
Effect of synthetic data on RP (RQ1) Here we compare the RP performance of our method for generating synthetic training data with a set of baselines. For this experiment, the RP component of the KGSQA system remains unchanged and we only alter the data it is trained with. Table 2 shows the results. We observe that our QG method is the best performing one. It significantly outperforms the baseline QG method, which confirms that our method for generating rich textual contexts for relations (Section 4.2) is beneficial for KGSQA. As expected, Wiki-raw-sentences performs better than when not using training data from the target domain at all but performs much worse than the QG methods. This is expected since Wikipedia sentences are very different both syntactically and grammatically from the real questions that the KGSQA system encounters during test time. Next, we investigate how RP performance varies depending on the number of the target domain questions used to augment the training set. Figure 1 shows the results. First, we observe that, in the low data regime (less than 100 questions), the gap in performance between training with gold or synthetic questions is small. This is encouraging for applying our
framework on domains in the long tail. As the number of questions increases, the performance for both gold and synthetic increase, however the gap between them increases, which is expected.
Overall KGSQA performance for data-centric methods (RQ2) Next, we compare our full framework to variations that use state-of-the-art RP models. Table 3 shows the results. We observe that our full method (second to last row) improves over all the baselines and significantly outperforms the best performing baseline. As expected, we see that even though our full method holds strong generalization ability for unseen domains, there is a gap in the performance when using the automatically generated synthetic questions (second to last row) or the human generated questions (last row). This gap suggests that there is room for improvement for QG. Next, we test the systems under comparison in terms of generalization ability across domains. Figure 2 shows the results. First, we observe that our method achieves an accuracy of at least 60% for all domains which shows that it is robust across domains. Also, it outperforms the baselines in all but one domain. In order to gain further insights, we sampled success and failure cases from the test set. We found that the errors in the failure cases generally originate from the fact that the model relies
on lexicalizations that are frequent in the seen domains. We show such cases in Table 4. Furthermore, our analysis showed that one way of improving QG is to improve keyword extraction by collecting a larger set of relevant sentences that express a single relation, possibly by looking into other sources of text (e.g. news articles). In Table 5, the automatic evaluation results for the synthetic questions generated by our QG model against those generated by [ElSahar et al., 2018] further strengthen our claim. As can be seen from the table, our model outperforms the baseline, indicating that using a larger set of relevant sentences for a single relation, can be beneficial to the generated questions.
Comparison to a model-centric method (RQ3) Here, we compare our data-centric method for domain adaptation to a state-of-the-art model-centric method on RP [Wu et al., 2019]. In order to perform a fair comparison when testing for RP, we follow their setup (see Section 5.1. in [Wu et al., 2019]) and for this particular experiment we assume that MD and CG produce the correct output. Our method outperforms their method both on macro-accuracy (75.54% vs 75.02%), and micro-accuracy (77.08% vs 72.17%). Note that we use randomly initialized embeddings whereas in their work they use JointNRE relation embeddings trained on Wikipedia and Freebase, which provides an advantage to
their method. Also note that their method (model-centric) is orthogonal to ours (datacentric) and therefore, an interesting future work direction would be to explore how to combine the two methods to further improve performance.
Qualitative error analysis In order to gain insights on how each part of the pipeline affects the final prediction, we perform an empirical error analysis. We sample 60 examples for which our system provided a wrong answer (10 for each target domain), and investigate what led to the wrong prediction. Out of these examples, 43 mistakes were due to RP, 6 due to ED, 5 due to CG and 6 due to AS. For RP, 22 were assigned to conceptually similar relations within the target domain and 15 to similar relations outside the target domain, while the rest were assigned to a common relation outside the target domain. For ED, 4 were due to predicting an extra token as part of the entity mention and 2 were due to missing a token from the entity mention. For CG, 2 were because the gold entity was not part of the mention candidates and 1 was due to an error in the human annotation; the early termination proposed in [Türe and Jojic, 2017], is responsible for the rest. From this analysis, we confirm that RP remains the most challenging part of KGSQA. Within RP, what seems to be the challenge is that there are relations for which there is a high lexical similarity between the corresponding questions, but also between the relations per se.
KGSQA performance on seen domains Finally, even though the focus of this paper is to perform KGSQA on unseen domains and thus we do not aim to improve state-ofthe-art on seen domains, we also test our KGSQA system on the standard split of the SimpleQuestions dataset. Our model achieves a top-1 accuracy of 77.0%, which is ranked third among the state-of-the-art methods while having a simpler method than the two topperforming ones ([Petrochuk and Zettlemoyer, 2018]), [Gupta et al., 2018]). We provide a thorough comparison w.r.t. the state-of-the-art on seen domains in Table 6.4

heading: 7. Related Work
text: Methods on the standard KGSQA task are split to those following a pipeline approach–MD, CG, RP & AS– ([Türe and Jojic, 2017, Mohammed et al., 2018, Petrochuk and Zettlemoyer, 2018]) or an end-to-end approach [Lukovnikov et al., 2017, Gupta et al., 2018]. In our work we follow the former approach for solving KGSQA on unseen domains, since we found that all the components except RP are relatively robust for unseen domains. We leave the exploration of end-to-end approaches for our task for future work. More related our setting, Yu et al. [2017] and Wu et al. [2019] tackle RP for KGSQA on unseen relations
4. Note that Zhao et al. [2019] reported an accuracy of 85.44%. However, they calculate accuracy w.r.t. the correctness of the object entity, which is not standard when testing on the SimpleQuestions dataset (see Section 5). When we calculate accuracy that way, [Petrochuk and Zettlemoyer, 2018] achieves an accuracy of 91.50% and our method achieves 87.31%.
(instead of whole domains). Both are model-centric domain adaptation approaches, while ours is data-centric. We experimentally showed that we outperform both in the setting of KGSQA on unseen domains. An interesting direction for future work would be to combine model-centric and data-centric approaches for our task.
More broadly, our work is also related to cross-domain semantic parsing [Su and Yan, 2017, Yu et al., 2018, Herzig and Berant, 2018, Zhang et al., 2019]. In contrast to the aforementioned line of work that maps questions to executable logical forms, we focus on questions that can be answered with a single KG fact.

heading: 8. Conclusion
text: In this paper, we proposed a data-centric domain adaptation framework for KGSQA that is applicable to unseen domains. Our framework performs QG to automatically generate synthetic training data for the unseen domains. We propose a keyword extraction method that when integrated in our QG model, it allows it to generate questions of various lexicalizations for the same underlying relation, thus better resembling the variety of real user questions. Our experimental results on the SimpleQuestions dataset show that our proposed framework significantly outperforms state-of-the-art zero-shot baselines, and is robust across different domains. We found that there is room for further improving QG particularly for KGSQA, which is a promising direction for future work.

heading: Acknowledgements
text: This research was supported by the NWO Innovational Research Incentives Scheme Vidi (016.Vidi.189.039), the NWO Smart Culture - Big Data / Digital Humanities (314-99-301), NWO under project nr CI-14-25, the H2020-EU.3.4. - Societal Challenges - Smart, Green And Integrated Transport (814961), the Google Faculty Research Awards program. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

Review summaries:
This paper presents a simple approach for domain adaptation in Knowledge Graph Question Answering. The paper consider the setting where the knowledge graph used to back the QA system contains the necessary facts for a test-time domain, but the training domain didn't cover an examples that required inference over that subdomain. To bridge the gap, the paper proposed a simple procedure for constructing synthetic questions over the relations in the test domain.

This paper studies the problem of answering "first-order" questions (more on the terminology later) that correspond to a single fact in a knowledge graph (KG) and focuses on the cross-domain setting where no curated training examples are provided for the unseen test domain. The proposed base KGQA model is modified from the state-of-the-art model on SimpleQuestions from (Petrochuk and Zettlemoyer, 2018) but with the relation prediction component changed from a classification model to a ranking model to better handle unseen relations (more on this later). The key contribution is a way of generating synthetic questions for the relations in the unseen domain for data augmentation. The generation model is from (ElSahar et al., 2018) but is augmented with relation-specific keywords mined from Wikipedia via distant supervision. Evaluation on reshuffled SimpleQuestions shows that the proposed method can achieve a reasonable performance on 6 selected test domains of large to moderate scale, and the question generation strategy is better than several baselines.