Learning to Decompose Compound Questions with Reinforcement Learning

1 INTRODUCTION . Knowledge-Based Question Answering ( KBQA ) is one of the most interesting approaches of answering a question , which bridges a curated knowledge base of tremendous facts to answerable questions . With question answering as a user-friendly interface , users can easily query a knowledge base through natural language , i.e. , in their own words . In the past few years , many systems ( Berant et al. , 2013 ; Bao et al. , 2014 ; Yih et al. , 2015 ; Dong et al. , 2015 ; Zhang et al. , 2017 ; Hao et al. , 2017 ) have achieved remarkable improvements in various datasets , such as WebQuestions ( Berant et al. , 2013 ) , SimpleQuestions ( Bordes et al. , 2015 ) and MetaQA ( Zhang et al. , 2017 ) . However , most of them ( Yih et al. , 2014 ; Bordes et al. , 2015 ; Dai et al. , 2016 ; Yin et al. , 2016 ; Yu et al. , 2017 ) assume that only simple questions are answerable . Simple questions are questions that have only one relation from the topic entity to unknown tail entities ( answers , usually substituted by an interrogative word ) while compound questions are questions that have multiple1 relations . For example , ” Who are the daughters of Barack Obama ? ” is a simple question and ” Who is the mother of the daughters of Barack Obama ? ” is a compound question which can be decomposed into two simple questions . In this paper , we aim to relax the assumption of answerable questions from simple questions to compound questions . Figure 1 illustrates the process of answering compound questions . Intuitively , to answer a compound question , traditional approaches firstly detect topic entity mentioned in the question , as the starting point for traversing the knowledge graph , then find a chain of multiple ( ≤ 3 ) relations as a multi-hop2 path to golden answers . We propose a learning-to-decompose agent which assists simple-question answerers to solve compound questions directly . Our agent learns a policy for decomposing compound question into simple ones in a meaningful way , guided by the feedback from the downstream simple-question answerers . The goal of the agent is to produce partitions and compute the compositional structure of questions 1We assume that the number of corresponding relations is at most three . 2We are aware of the term multi-hop question in the literature . We argue that compound question is a better fit for the context of KBQA since multi-hop characterizes a path , not a question . As for document-based QA , multi-hop also refers to routing over multiple evidence to answers . with maximum information utilization . The intuition is that encouraging the model to learn structural compositions of compound questions will bias the model toward better generalizations about how the meaning of a question is encoded in terms of compositional structures on sequences of words , leading to better performance on downstream question answering tasks . We demonstrate that our agent captures the semantics of compound questions and generate interpretable decomposition . Experimental results show that our novel approach achieves state-of-the-art performance in two challenging datasets ( WebQuestions and MetaQA ) , without re-designing complex neural networks to answer compound questions . 2 RELATED WORK . 2.1 KNOWLEDGE-BASED QUESTION ANSWERING . For combinational generalization ( Battaglia et al. , 2018 ) on the search space of knowledge graph , many approaches ( Yih et al. , 2014 ; Yin et al. , 2016 ; Zhang et al. , 2017 ) tackle KBQA in a tandem manner , i.e. , topic entity linking followed by relation detection . An important line of research focused on directly parsing the semantics of natural language questions to structured queries ( Cai & Yates , 2013 ; Kwiatkowski et al. , 2013 ; Yao & Van Durme , 2014 ; Yao et al. , 2014 ; Bao et al. , 2014 ; Yih et al. , 2014 ; 2015 ) . An intermediate meaning representation or logical form is generated for query construction . It often requires pre-defined rules or grammars ( Berant et al. , 2013 ) based on hand-crafted features . By contrast , another line of research puts more emphasis on representing natural language questions instead of constructing knowledge graph queries . Employing CNNs ( Dong et al. , 2015 ; Yin et al. , 2016 ) or RNNs ( Dai et al. , 2016 ; Yu et al. , 2017 ) , variable-length questions are compressed into their corresponding fix-length vector . Most approaches in this line focus on solving simple questions because of the limited expression power of fix-length vector , consistent with observations ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) in Seq2Seq task such as Neural Machine Translation . Closely related to the second line of research , our proposed model learns to decompose compound question into simple questions , which eases the burden of learning vector representations for compound question . Once the decomposition process is completed , a simple-question answerer directly decodes the vector representation of simple questions to an inference chain of relations with the desired order , which resolves the bottleneck of KBQA . 2.2 REINFORCEMENT LEARNING FOR NATURAL LANGUAGE UNDERSTANDING . Many reinforcement learning approaches learn sentence representations in a bottom-up manner . Yogatama et al . ( 2017 ) learn tree structures for the order of composing words into sentences using reinforcement learning with Tree-LSTM ( Tai et al. , 2015 ; Zhu et al. , 2015 ) , while Zhang et al . ( 2018 ) employ REINFORCE ( Williams , 1992 ) to select useful words sequentially . Either in tree structure or sequence , the vector representation is built up from the words , which benefits the downstream natural language processing task such as text classification ( Socher et al. , 2013 ) and natural language inference ( Bowman et al. , 2015 ) . By contrast , from the top down , our proposed model learns to decompose compound questions into simple questions , which helps to tackle the bottleneck of KBQA piece by piece . See section 3 for more details . Natural question understanding has attracted the attention of different communities . Iyyer et al . ( 2017 ) introduce SequentialQA task that requires to parse the text to SQL which locates table cells as answers . The questions in SequentialQA are decomposed from selected questions of WikiTableQuestions dataset ( Pasupat & Liang , 2015 ) by crowdsourced workers while we train an agent to decompose questions using reinforcement learning . Talmor & Berant ( 2018 ) propose a ComplexWebQuestion dataset that contains questions with compositional semantics while Bao et al . ( 2016 ) collects a dataset called ComplexQuestions focusing on multi-constrained knowledge-based question answering . The closest idea to our work is Talmor & Berant ( 2018 ) which adopts a pointer network to decompose questions and a reading comprehension model to retrieve answers from the Web . The main difference is that they leverage explicit supervisions to guide the pointer network to correctly decompose complex web questions based on human logic ( e.g. , conjunction or composition ) while we allow the learning-to-decompose agent to discover good partition strategies that benefit downstream task . Note that it is not necessarily consistent with human intuition or linguistic knowledge . 2.3 DEEP SEMANTIC ROLE LABELING . Without heavy feature engineering , semantic role labeling based on deep neural networks ( Collobert et al. , 2011 ) focus on capturing dependencies between predicates and arguments by learning to label semantic roles for each word . Zhou & Xu ( 2015 ) build an end-to-end system which takes only original text information as input features , showing that deep neural networks can outperform traditional approaches by a large margin without using any syntactic knowledge . Marcheggiani et al . ( 2017 ) improve the role classifier by incorporating vector representations of both input sentences and predicates . Tan et al . ( 2018 ) handle structural information and long-range dependencies with self-attention mechanism . This line of work concentrates on improving role classifier . It still requires rich supervisions for training the role classifier at the token level . Our approach also requires to label an action for each word , which is similar to role labeling . However , we train our approach at the sentence level which omits word-by-word annotations . Our learning-to-decompose agent generates such annotations on the fly by exploring the search space of strategies and increases the probability of good annotations according to the feedback . 3 MODEL . Figure 2 illustrates an overview of our model and the data flow . Our model consists of two parts : a learning-to-decompose agent that decomposes each input question into at most three partitions and three identical simple-question answerers that map each partition to its corresponding relation independently . We refer to the learning-to-decompose agent as the agent and three simple-question answerers as the answerers in the rest of our paper for simplicity . 3.1 LEARNING-TO-DECOMPOSE AGENT . Our main idea is to best divide an input question into at most three partitions which each partition contains the necessary information for the downstream simple-question answerer . Given an input question of N words3 x = { x1 , x2 , . . . , xN } , we assume that a sequence of words is essentially a partially observable environment and we can only observe the corresponding vector representation ot = xt ∈ RD at time step t. Figure 3 summarizes the process for generating decision of compound question decomposition . Memory Unit The agent has a Long Short-Term Memory ( LSTM ; Hochreiter & Schmidhuber ( 1997 ) ) cell unrolling for each time step to memorize input history . it = σ ( Wi [ xt , ht−1 ] + bi ) ft = σ ( Wf [ xt , ht−1 ] + bf ) gt = tanh ( Wg [ xt , ht−1 ] + bg ) ot = σ ( Wo [ xt , ht−1 ] + bo ) ct = ft ct−1 + it gt ht = ot tanh ( ct ) ( 1 ) where Wi , Wf , Wg , Wo ∈ RH× ( D+H ) , bi , bf , bg , bo ∈ RH , and [ · , · ] denotes the concatenation of two vectors . σ ( · ) is the element-wise sigmoid activation function . 3Note that the length of sequences may vary from questions to questions . We handle such dynamic directly without padding zeros . The similar situation exists in a game environment that allows early stopping . The state st ∈ R2H of the agent is defined as st = [ ct , ht ] ( 2 ) which maintained by the above memory cell ( Eq . 1 ) unrolling for each time step . [ · , · ] denotes the concatenation of two vectors . Action Unit The agent also has a stochastic policy network π ( α|s ; Wπ ) where Wπ is the parameter of the network . Specifically , we use a two-layer feedforward network that takes the agent ’ s state s as its input : π ( α|s ; Wπ ) ∝ exp ( W ( 2 ) π ( ReLU ( W ( 1 ) π s + b ( 1 ) π ) ) + b ( 2 ) π ) ( 3 ) where W ( 1 ) π ∈ RH×2H , b ( 1 ) π ∈ RH , W ( 2 ) π ∈ R3×H and b ( 2 ) π ∈ R3 . Following the learned policy , the agent decomposes a question of lengthN by generating a sequence of actions αt ∈ { 1st , 2nd , 3rd } , t = 1 , 2 , . . . , N . Words under the same decision ( e.g . 1st ) will be appended into the same sub-sequence ( e.g . the first partition ) . Formally , x ( k ) = { x ( k ) 1 , x ( k ) 2 , . . . , x ( k ) tk } , k = 1 , 2 , 3 denotes the partitions of a question . Note that in a partition , words are not necessarily consecutive4 . The relative position of two words in original question is preserved . t1 + t2 + t3 = N holds for every question . Reward The episodic reward R will be +1 if the agent helps all the answerers to get the golden answers after each episode , or −1 otherwise . There is another reward function R = Σ logP ( Y ∗ | X ) that is widely used in the literature of using reinforcement learning for natural language processing task ( Bahdanau et al. , 2017 ; Zhang et al. , 2018 ) . We choose the former as reward function for lower variance . Each unique rollout ( sequence of actions ) corresponds to unique compound question decomposition . We do not assume that any question should be divided into exactly three parts . We allow our agent to explore the search space of partition strategies and to increase the probability of good ones . The goal of our agent is to learn partition strategies that benefits the answerers the most .

Summary: the paper is interested in parsing compound questions for querying on knowledge graph, e.g. MetaQA by Zhang et al. (2017). The paper proposes to have two modules, one that segments the question into partitions (up to three) and the other that looks at each segment to get the relation. The relations are merged to obtain a single KG path, which is queried to obtain the answer. Since the segmentation is a non-differentiable process, the paper uses reinforcement learning to propagate gradient to the segmentation model. The segmentation is a process of classifying each word for which partition it should be tied to. Answering is a process of classifying the partition into one of the possible relation edges. The model shows expected results in a synthetic arithmetic dataset, and obtains the state of the art in MetaQA, improving nearly 5% over the baseline. The model especially does much better on 3-hop questions, with nearly 20% improvement.