Question Decomposition with Dependency Graphs

1 . Introduction . Training neural networks to reason over multiple parts of their inputs across modalities such as text , tables , and images , has been a focal point of interest in recent years [ Antol et al. , 2015 , Pasupat and Liang , 2015 , Johnson et al. , 2017 , Suhr et al. , 2019 , Welbl et al. , 2018 , Talmor and Berant , 2018 , Yang et al. , 2018 , Hudson and Manning , 2019 , Dua et al. , 2019 , Chen et al. , 2020 , Hannan et al. , 2020 , Talmor et al. , 2021 ] . The most common way to probe whether a model is capable of complex reasoning , is to pose in natural language a complex question , which requires performing multiple steps of computation over the input . One natural way of answering such complex questions is to break them down into a sequence of simpler sub-steps [ Christiano et al. , 2018 , Min et al. , 2019 , Perez et al. , 2020 ] . Wolfson et al . [ 2020 ] recently proposed QDMR , a meaning representation where complex questions are represented through a sequence of simpler atomic executable steps ( see Fig . 1 ) , and the final answer is the answer to the final step . QDMR has been shown to be useful for multi-hop question answering ( QA ) [ Wolfson et al. , 2020 ] and also for improving interpretability in visual QA [ Subramanian et al. , 2020 ] . State-of-the-art QDMR parsers use the typical seq2seq approach . However , it is natural to think of QDMR as a dependency graph over the input question tokens . Consider the example in Fig . 1 . The first QDMR step selects the span “ Indiana Jones ” . Then , the next step uses a PROJECT operation to find the “ movies ” of Indiana Jones , and the next step uses another PROJECT operation to find the date when the movies were “ released ” . Such relations can be represented as labeled edges over the relevant question tokens . In this work , we propose to use the dependency graph view of QDMR to improve QDMR parsing . We describe a conversion procedure that automatically maps QDMR structures into dependency graphs , using a structured intermediate logical form representation ( Fig 1 , bottom-left ) . Once we have graph supervision for every example , we train a dependency graph parser , in the spirit of Dozat and Manning [ 2018 ] , where we predict a labeled relation for every pair of question tokens , representing the logical relation between the tokens . Unlike seq2seq models , this is a non-autoregressive parser , which decodes the entire output structure in a single step . In addition , we study the effect of using dependency graphs as auxiliary supervision for a seq2seq QDMR parser , where the graph is decoded from the encoder representations . Towards that end , we propose a Latent-RAT encoder , which uses relation-aware transformer [ Shaw et al. , 2018 ] to explicitly represent the relation between every pair of input tokens . Relation-aware transformer has been shown to be useful for encoding graph structures in the context of semantic parsing [ Wang et al. , 2020 ] . Last , to fairly compare QDMR parsers that use different representations , we propose an evaluation metric , LF-EM , based on the aforementioned intermediate logical form . We show that LF-EM correlates better with human judgements compared to existing metrics . We find that our graph parser leads to a small reduction in LF-EM compared to seq2seq models ( 0.47→0.44 ) , but is 16x faster due to its non-autoregressive nature , and is by design more interpretable . Moreover , it has better sample complexity and outperforms the seq2seq model when trained on 10 % of the data or less . When training a seq2seq model with the auxiliary graph supervision , the parser obtains similar performance as when trained on the entire dataset ( 0.471 LF-EM ) , but substantially improves performance when generalizing to new domains . Moreover , it performs better on examples with a large number of computation steps . Our code is available at https : //github.com/matanhasson/ qdecomp_with_dependency_graphs . 2 . Overview . The core of this work is to examine the utility of a dependency graph ( DG ) representation for QDMR . We propose conversion procedures that enable training and evaluating with DGs ( see Fig . 2 ) . First , we convert gold QDMR structures into logical forms ( LF ) , where each computation step in QDMR is represented with a formal operator , properties and arguments ( §3 ) . Then , we obtain gold DGs by projecting the logical forms onto the question tokens ( §4 ) . Once we have question- DG pairs , we can train a graph parser . At test time , QDMRs and DGs are converted into LFs for evaluation . We propose a new evaluation metric over LFs ( §3 ) , and show it is more robust to semantic-preserving changes compared to prior metrics . Our proposed parsers are in §5 . On top of standard seq2seq models , we describe ( a ) a graph parser , and ( b ) a multi-task model , where the encoder of the seq2seq model is also trained to predict the DG . 3 . QDMR Logical Forms . QDMR [ Wolfson et al. , 2020 ] is a text-based meaning representation focused on representing the meaning of complex questions . It is based on a decomposition of questions into a sequence of simpler executable steps ( Fig . 1 ) , where each step corresponds to a SQL-inspired operator ( Table 5 , §A.1 ) . We briefly review QDMR and then define a logical form ( LF ) representation based on these operations . We use the LFs both for mapping QDMRs to DGs , and also to fairly evaluate the output of parsers that output either QDMRs directly or DGs . QDMR Definition Given a question with n tokens , q = q1 . . . qn , its QDMR is a sequence of m steps s1 , . . . , sm , where step si conceptually maps to a single logical operator oi ∈ O . A step , si , is a sequence of ni tokens s i = si1 . . . s i ni , where token s i j is either a question token ∈ Vq ( or some inflection of it ) , a word from a constant predefined lexicon ∈ Vconst , or a reference token ∈ V iref = { # 1 , . . . , # ( i− 1 ) } , referring to a previous step . Fig . 3 shows an example for a question and its QDMR structure . QDMR Logical Form ( LF ) Given a QDMR S = 〈q ; s1 , . . . , sm〉 , its logical form is a sequence of logical form steps Z = 〈q ; z1 , . . . , zm〉 . The LF step zi , corresponding to si , is a triplet zi = 〈oi , ρi , Ai〉 where oi ∈ O is the logical operator ; ρi ∈ PROPoi are operator-specific properties ; and Ai is a dictionary of arguments , mapping an operator-specific argument η ∈ ARGoi , ρi to a span τ from the QDMR step si . For convenience , we denote zi with the string oi [ ρi ] ( ηi1 = τ i 1 , . . . ) . Table 1 provides a few examples for the mapping from QDMR to LF steps , and Table 5 ( §A.1 ) provides the full list . QDMR→LF We convert QDMRs to LFs with a rule-based method , extending the procedure for detecting operators from Wolfson et al . [ 2020 ] to also find properties and arguments . To detect properties we use a lexicon ( see Table 6 in §A.1 ) . LF-based Evaluation ( LF-EM ) The official evaluation metric for QDMR1 is normalized EM ( NormEM ) , where the predicted and gold QDMR structures are normalized using a rule-based procedure , and then exact string match is computed between the two normalized QDMRs . Since in this work we convert both QDMRs ( §3 ) and DGs ( §4 ) to LFs , we propose a LF-based evaluation metric . LF-EM essentially involves computing exact match between the predicted and gold LFs . To further capture semantic equivalences , we perform more normalization steps , which for brevity are described in §A.2 . We manually evaluate the metrics NormEM and LF-EM on 50 random development set examples using predictions from the CopyNet+BERT model ( see §6 ) . We find that both metrics have perfect precision ( no false-positives ) ; but the LF-EM covers more examples ( 52.0 % vs 40.0 % ) . Thus , it provides a tighter lower bound on the performance of a QDMR parser and correlates better with notions of semantic equivalence . 1. https : //leaderboard.allenai.org/break/submissions/public 4 . From LFs to Dependency Graphs . Given a QDMR decomposition S = 〈q ; s1 , . . . , sm〉 , we construct a dependency graph G = 〈N , E〉 , where the nodes N correspond to question tokens , and the edges E describe the logical operations , resulting in a graph with the same meaning as S. The LF→DG procedure is shown in Fig . 4 and consists of the following steps : • Token alignment : align each token in the question to a token in a QDMR step ( §A.3 ) . • Spans Dependency Graph ( SDG ) extraction : construct a graph where each node corresponds to a list of tokens in a QDMR step , and edges describe the dependencies between the steps ( §A.4 ) . • Dependency Graph ( DG ) extraction : convert the SDG to a DG over the question tokens . Here , we add span edges for tokens that are in the same step , and deal with some representation issues ( §A.5 ) . Because we convert predicted DGs to LFs for evaluation , the LF→DG conversion must be invertible . Our conversion succeeds in 97.12 % of the BREAK dataset [ Wolfson et al. , 2020 ] . 5 . Models . Once we have methods to convert QDMRs to DGs and LFs , and DGs to LFs , we can evaluate the advantages and disadvantages of standard autoregressive decoders compared to graph-based parsers . We describe three models : ( a ) An autoregressive parser , ( b ) a graph parser , ( c ) an autoregressive parser that is trained jointly with a graph parser in a multi-task setup . For a fair comparison , all models have the same BERT-based encoder [ Devlin et al. , 2019 ] . CopyNet+BERT ( baseline ) This autoregressive QDMR parser is based on the CopyNet baseline from Wolfson et al . [ 2020 ] , except we replace the BiLSTM encoder with a transformer initialized with BERT . The model encodes the question q and then decodes the QDMR S step-by-step and token-by-token . The decoder is an LSTM [ Hochreiter and Schmidhuber , 1997 ] augmented with a copy mechanism [ Gu et al. , 2016 ] , where at each time step the model either decodes a token from the vocabulary or a token from the input . Since the input is tokenized with word pieces , we average word pieces that belong to a single word to get word representations , which enables word copying . Training is done with standard maximum likelihood . Biaffine Graph Parser ( BiaffineGP ) The biaffine graph parser takes as input the question q augmented with the special tokens described in §A.5 and predicts the DG by classifying for every pair of tokens whether there is an edge between them and the label of the edge . The model is based on the biaffine dependency parser of Dozat and Manning [ 2018 ] , except here we predict a DAG and not a tree , so each node can have more than one outgoing edge . Let H = 〈h1 , . . . , h|H|〉 be the sequence of representations output by the BERT encoder concatenated with the POS embeddings . The biaffine parser uses four 1-hidden layer feed-forward networks over each contextualized token representation ht : hedge-headt = FF edge-head ( ht ) , h edge-dep t = FF edge-dep ( ht ) , hlabel-headt = FF label-head ( ht ) , h label-dep t = FF label-dep ( ht ) . The probability of an edge from token i to token j is given by σ ( hedge-dep > i Wedgeh edge-head j ) , where Wedge is a parameter matrix . Similarly , the score of an edge labeled by the label l from token i to token j is given by slij = h label-dep > i Wlh label-head j , where Wl is the parameter matrix for this label . We then compute a distribution over the set of labels L with softmax ( s1ij , . . . , s |L| ij ) . Training is done with maximum likelihood both on the edge probabilities and label probabilities . Inference is done by taking all edges with edge probability > 0.5 and then labeling those edges according to the most probable label . There is no guarantee that the biaffine parser will output a valid DG . For example , if an SDG node has an outgoing edge labeled with filter-sub and another labeled with project-sub , we can not tell if the operator is FILTER or PROJECT . This makes parsing fail , which occurs in 1.83 % of the cases . To create a SDG , we first use the span edges to construct SDG nodes with lists of tokens , and then add edges between SDG nodes by projecting the edges between tokens to edges between the SDG nodes . To prevent cases where parsing fails , we can optionally apply an ILP that takes the predicted probabilities as input , and outputs a valid DG . The exact details are given in our open source implementation . Multi-task Latent-RAT Encoder ( Latent-RAT ) In this model , our goal is to improve the seq2seq parser by providing more information to the encoder using the DG supervision . Our model will take the question q ( with special tokens as before ) as input , and predict both the graph G directly and the QDMR structure S with a decoder . We would like the information on relations between tokens to be part of the transformer encoder , so that the decoder can take advantage of it . To accomplish that , we use RAT transformer layers [ Shaw et al. , 2018 , Wang et al. , 2020 ] , which explicitly represent relations between tokens , and have been shown to be useful for encoding graphs over input tokens in the context of semantic parsing . RAT layers inject information on the relation between tokens inside the transformer self-attention mechanism [ Vaswani et al. , 2017 ] . Specifically , the similarity score eij computed using queries and keys is given by eij ∝ hiWQ ( hjWK + rKij ) T , where WQ , WK are the query and key parameter matrices and the only change is the term rKij , which represents the relation between the tokens i and j . Similarly , the relation between tokens is also considered when summing over self-attention values ∑H j=1 αij ( xjWV + r V ij ) , where WV is the value parameter matrix , αij is the attention distribution and the only change is the term r V ij . Unlike prior work where the terms rKij , r V ij were learned parameters , here we want these vectors to ( a ) be a function of the contextualized representation and ( b ) be informative for classifying the dependency label in the gold graph . By learning latent representations from which the gold graph can be decoded , we will provide useful information for the seq2seq decoder . Specifically , given the encoder output representations hi , hj for tokens i and j , we represent relations and compute a loss in each RAT layer as follows ( Fig . 5 ) : rKij = FF K ( hi − hj ) , SK = RKW out + bK ∈ Rn×n×|L| , LossK = CE ( SK ) . FFK is a 1-hidden layer feed-forward network , RK ∈ R ( n×n ) ×dtransformer is a concatenation of all rKij for all pairs of tokens , W out ∈ Rdtransformer×|L| is a projection matrix that provides a score for all possible labels ( including the NONE label ) . We compute an analogous loss LossV for rVij and the final graph loss is Loss K + LossV over all RAT layers . To summarize , by performing multi-task training with this graph loss we push the transformer to learn representations rij that are informative of the gold graph , and can then be used by the decoder to output better QDMR structures .

Summary: The paper investigates the utility of dependency graph (DG) representation in converting questions to Question Decomposition Meaning Representation (QDMR) annotations. As an intermediate step, the paper proposes to convert QDMR annotations to logical form (LF) representation (this was also done in the original QDMR paper). This LF representation is used to then extract a DG representation from it in a deterministic manner. The DG can also be converted to LF in a similar manner. Additionally, the paper also proposes a LF based evaluation metric which apparently is a better measure for semantic equivalence between QDMRs as compared to the previously proposed metric.