Knowledge Graph Simple Question Answering for Unseen Domains

1 . Introduction . Large-scale structured Knowledge Graphs ( KGs ) such as Freebase [ Bollacker et al. , 2008 ] and Wikidata [ Pellissier Tanon et al. , 2016 ] store real-world facts in the form of subject–relation– object triples . KGs are being increasingly used in a variety of tasks that aim to improve user experience [ Bota et al. , 2016 ] . One of the most prominent tasks is Knowledge Graph Simple Question Answering ( KGSQA ) , which aims to answer natural language questions by retrieving KG facts [ Yih et al. , 2015 ] . In practice , many questions can be interpreted by a single fact in the KG . This has motivated the KGSQA task [ Bordes et al. , 2015 , Mohammed et al. , 2018 , Petrochuk and Zettlemoyer , 2018 ] , which is the focus of this paper . In KGSQA , given a simple question , e.g . “ who directed the godfather ? ” , the system should interpret the question and arrive at a single KG fact that answers it : ( The Godfather ( film ) , film.film.directed by , Francis Ford Coppola ) . KGSQA systems are trained on manually annotated datasets that consist of questionfact pairs . In practice , the applicability of such systems in the real-world is limited by two factors : ( i ) modern KGs store millions of facts that cover thousands of different relations , but KGSQA training datasets can only cover a small subset of the existing relations in the KG [ ElSahar et al. , 2018 ] , and ( ii ) KGs are dynamic , i.e . they are updated with new domains that cover new relations [ Pellissier Tanon et al. , 2016 ] . Solving ( i ) and ( ii ) by exhaustively gathering question-fact pair annotations would be prohibitively laborious , thereby we need to rely on automatic methods . Motivated by the above , in this work , we study the KGSQA task in a setting where we are interested in answering questions about a new , unseen domain that covers relations , for which we have instances in the KG , but we have not seen any question-fact pair during training . We model this as a domain adaptation task [ Mansour et al. , 2009 , Pan and Yang , 2010 ] and propose a data-centric domain adaptation framework to address it . Datacentric domain adaptation approaches focus on transforming or augmenting the training data , instead of designing specialized architectures and training objectives as model-centric domain adaptation approaches do [ Chu and Wang , 2018 ] . Our framework consists of : ( a ) a KGSQA system which can handle the unseen domain , and ( b ) a novel method that generates training data for the unseen test domain . The KGSQA system we introduce performs mention detection , entity candidate generation and relation prediction on the question , and finally selects the fact that answers the question from the KG . To improve relation prediction on questions that cover relations of the unseen domain , we automatically generate synthetic questions from KG facts of the unseen domain ( i.e . knowledge graph question generation – QG ) . The resulting synthetic question-fact pairs are used to train the KGSQA system for the unseen domain . We find that the effectiveness of QG for KGSQA can be restricted not only by the quality of the generated questions , but also by the lexical variety of the questions . This is because users ask questions underlying the same relation using different lexicalizations ( e.g . “ who is the author of X ” , “ who wrote X ” ) . To address this , we use distant supervision to extract a set of keywords for each relation of the unseen domain and incorporate those in the question generation method . Our main contributions are the following : ( i ) we introduce a new setting for the KGSQA task , over new , previously unseen domains , ( ii ) we propose a data-centric domain adaptation framework for KGSQA that is applicable to unseen domains , and ( iii ) we use distant supervision to extract a set of keywords that express each relation of the unseen domain and incorporate them in QG to generate questions with a larger variety of relation lexicalizations . We experimentally evaluate our proposed method on a large-scale KGSQA dataset that we adjust for this task and show that our proposed method consistently improves performance over zero-shot baselines and is robust across domains.1 2 . Problem Statement . Let E denote the set of entities and R the set of relations . A KG K is a set of facts ( es , r , eo ) , where es , eo ∈ E are the subject and object entities respectively , and r ∈ R is the relation between them . Each relation r has a unique textual label rl and falls under a single domain D. For instance , music.album.release type and music.artist.genre fall under the Music domain . Simple questions mention a single entity and express a single relation . For instance , the question “ who directed the godfather ? ” mentions the entity “ The Godfather ” and expresses the relation film.film.directed by . Given a simple question q that consists of a sequence of tokens t1 , t2 , . . . , tT , the KGSQA task is to retrieve a fact ( ês , r̂ , êo ) , where ( ês , r̂ ) accurately interprets q ( i.e. , ês is mentioned 1 . Our code is available at https : //github.com/GSidiropoulos/kgsqa_for_unseen_domains . in q and r̂ is expressed in q ) while êo provides the answer to q . In our setting , we aim to build a KGSQA system that can perform well on a previously unseen domain . A domain is “ unseen ” when facts that cover relations of that domain do exist in K , but gold-standard question-fact pairs of that domain do not appear in the training data . This setting is an instance of domain adaptation , where a model is trained on data S , which is drawn according a source distribution , and tested on data T coming from a different target distribution . Domain adaptation over KG domains is more challenging compared to domain adaptation over single KG relations [ Yu et al. , 2017 , Wu et al. , 2019 ] , because it is less likely for relations with similar lexicalizations to appear in the training set . 3 . KGSQA system . In this section , we detail our KGSQA system . In Section 4 , we will describe how we generate synthetic training data to make this system applicable to unseen domains . Following current state-of-the-art on KGSQA [ Petrochuk and Zettlemoyer , 2018 ] , we split the task into four sub-tasks , namely , entity mention detection ( MD ) , entity candidate generation ( CG ) , relation prediction ( RP ) , and answer selection ( AS ) . The skeleton of our KGSQA system generally follows previous work , and we modify the MD and RP architectures . Mention Detection ( MD ) Given the question q , MD outputs a single entity mention m in q , where m is a sub-sequence of tokens in q . We model this problem as sequence tagging , where given a sequence of tokens , the task is to assign an output class for each token [ Huang et al. , 2015 , Lample et al. , 2016 ] . In our case , the output classes are entity ( E ) and context ( C ) . For instance , the correct output for the question “ who directed the godfather ? ” is “ [ C C E E ] ” . We use a BiLSTM with residual connections ( R-BiLSTM ) [ He et al. , 2016 ] , since it outperformed vanilla RNN , BiRNN , and a CRF on top of a BiRNN [ Petrochuk and Zettlemoyer , 2018 ] in preliminary experiments . Candidate Generation ( CG ) Given the mention m extracted from the previous step , CG maps m to a set of candidate entities CS ⊂ E. For instance , CG maps the mention “ the godfather ” to the entities { The Godfather ( film ) , The Godfather ( book ) . . . } . The CG method we use was proposed in Türe and Jojic [ 2017 ] . Briefly , the method pre-builds an inverted index I from n-grams of mentions to entities , and it looks-up the n-grams of m in I to obtain CS . Relation Prediction ( RP ) Given the question q and the set of entities Cs extracted in the previous step , RP outputs a single relation r̂ ∈ R that is expressed in q . Previous work models RP as a large-scale multi-label classification task where the set of output classes is fixed [ Petrochuk and Zettlemoyer , 2018 ] . In our domain adaptation scenario , however , we want to be able to predict relations that we have not seen during training . Therefore , we model RP as a relation ranking task , as in [ Yu et al. , 2017 ] , and use the textual label rl to represent the relation r ( instead of using a categorical variable ) . This way we can in principle represent any relation r ∈ R during inference time . Below we describe the architecture we use for RP and how we perform training and inference . Our architecture is a simpler version of [ Yu et al. , 2017 ] , where they model a relation both as a sequence and a categorical variable , and they use more complex sequence encoders . First we describe how we encode the question q and the relation r. In order to generalize beyond specific entity names , we first replace the previously detected entity mention m in q with a placeholder token ; e.g “ who directed SBJ ” . We then map each term to its embedding and feed the word embeddings to an LSTM ; embeddings are initialized with pretrained word2vec embeddings [ Mikolov et al. , 2013 ] . The final hidden state of the LSTM γ ( q ) is used as the encoding of the question . In order to represent r , we use its label rl ( e.g . film.film.directed by ) . Similarly with the question encoding , we encode rl with an LSTM to obtain γ ( r ) . However , since questions and relations significantly differ both grammatically and syntactically , the two LSTM encoders do not share any parameters . The ranking function f is calculated as f ( q , r ) = cos ( γ ( q ) , γ ( r ) ) , where cos ( · ) is the cosine similarity . We train f using standard pairwise learning to rank . The loss is defined as follows : L ( θ ) = ∑ r ∑ r′∈R′ max ( 0 , µ− f ( q , r ) + f ( q , r′ ) ) , ( 1 ) where θ are the parameters of the model , µ is a hyperparameter , and R′ is the set of sampled negative relations for a question q . We design a specialized negative sampling method to select R′ . With probability P−R we uniformly draw a sample from R − = { r′|r′ ∈ R∧r′ 6= r } ; the set of all available relations except the positive relation r. With probability 1− P−R we draw a random sample from R̂− = { r′|r′ ∈ D+R ∧ r′ 6= r } ; the set of relations that are in the same domain as the positive relation r. This way , we expose the model to conditions it will encounter during inference . At inference time , given a question q and a set of relations we score all question-relation pairs ( q , r ) with f and select the relation r̂ with the highest score . Unfortunately , computing a score with respect to all possible relations in R leads to poor performance when there is no linguistic signal to disambiguate the choice . In order to address this issue , we constrain the set the potential output relations Rc to be the union of the relations expressed in the facts where the entities in CS participate in [ Petrochuk and Zettlemoyer , 2018 ] . Formally , we define the target relation classes to be Rc = { r ∈ R| ( es , r , eo ) ∈ K ∧es ∈ CS } . For example , given the question “ who directed the godfather ” , the potential relations are { film.film.directed by , book.written work.author , . . . } . Using the aforementioned constraint we can safely ignore relations like tv.tv series episode.director by taking into account that The Godfather does not appear in any tv-related facts . Answer Selection ( AS ) Given the set of entities CS obtained from CG , and the top ranked relation r̂ obtained from RP , AS selects a single fact ( ês , r̂ , êo ) , where êo answers the question q . The set of candidate answers may contain more than one facts ( e′s , r̂ , e ′ o ) , where ∀e′s ∈ CS . Since there is no explicit signal on which we can rely to disambiguate the choice of subject , all the potential answers are equally probable . We therefore use a heuristic based on popularity , introduced by Mohammed et al . [ 2018 ] : we choose ês to be the entity that appears the most in the facts in K either as a subject or as an object . Having ês and r̂ we can retrieve the fact ( ês , r̂ , êo ) . For our running example ( “ who directed the godfather ” ) , given film.film.directed by ( from RP ) and entities { The Godfather ( film ) , The Godfather ( book ) . . . } ( from CG ) we can select the fact ( The Godfather ( film ) , film.film.directed by , Francis Ford Coppola ) .

Summary: This paper studies the problem of answering "first-order" questions (more on the terminology later) that correspond to a single fact in a knowledge graph (KG) and focuses on the cross-domain setting where no curated training examples are provided for the unseen test domain. The proposed base KGQA model is modified from the state-of-the-art model on SimpleQuestions from (Petrochuk and Zettlemoyer, 2018) but with the relation prediction component changed from a classification model to a ranking model to better handle unseen relations (more on this later). The key contribution is a way of generating synthetic questions for the relations in the unseen domain for data augmentation. The generation model is from (ElSahar et al., 2018) but is augmented with relation-specific keywords mined from Wikipedia via distant supervision. Evaluation on reshuffled SimpleQuestions shows that the proposed method can achieve a reasonable performance on 6 selected test domains of large to moderate scale, and the question generation strategy is better than several baselines.