Question Decomposition with Dependency Graphs
Year: 2021

Abstract:
QDMR is a meaning representation for complex questions, which decomposes questions into a sequence of atomic steps, and has been recently shown to be useful for question answering. While state-of-the-art QDMR parsers use the common sequence-to-sequence (seq2seq) approach, a QDMR structure fundamentally describes labeled relations between spans in the input question, and thus dependency-based approaches seem appropriate for this task. In this work, we present a QDMR parser that is based on dependency graphs (DGs), where nodes in the graph are words and edges describe logical relations that correspond to the different computation steps. We propose (a) a non-autoregressive graph parser, where all graph edges are computed simultaneously, and (b) a seq2seq parser that uses the gold graph as auxiliary supervision. We find that a graph parser leads to a moderate reduction in performance (0.47→0.44), but to a 16x speed-up in inference time due to its non-autoregressive nature, and to improved sample complexity compared to a seq2seq model. Second, training a seq2seq model with auxiliary DG supervision leads to better generalization on out-of-domain data and on QDMR structures with long sequences of computation steps.

heading: 1. Introduction
text: Training neural networks to reason over multiple parts of their inputs across modalities such as text, tables, and images, has been a focal point of interest in recent years [Antol et al., 2015, Pasupat and Liang, 2015, Johnson et al., 2017, Suhr et al., 2019, Welbl et al., 2018, Talmor and Berant, 2018, Yang et al., 2018, Hudson and Manning, 2019, Dua et al., 2019, Chen et al., 2020, Hannan et al., 2020, Talmor et al., 2021]. The most common way to
probe whether a model is capable of complex reasoning, is to pose in natural language a complex question, which requires performing multiple steps of computation over the input.
One natural way of answering such complex questions is to break them down into a sequence of simpler sub-steps [Christiano et al., 2018, Min et al., 2019, Perez et al., 2020]. Wolfson et al. [2020] recently proposed QDMR, a meaning representation where complex questions are represented through a sequence of simpler atomic executable steps (see Fig. 1), and the final answer is the answer to the final step. QDMR has been shown to be useful for multi-hop question answering (QA) [Wolfson et al., 2020] and also for improving interpretability in visual QA [Subramanian et al., 2020].
State-of-the-art QDMR parsers use the typical seq2seq approach. However, it is natural to think of QDMR as a dependency graph over the input question tokens. Consider the example in Fig. 1. The first QDMR step selects the span “Indiana Jones”. Then, the next step uses a PROJECT operation to find the “movies” of Indiana Jones, and the next step uses another PROJECT operation to find the date when the movies were “released”. Such relations can be represented as labeled edges over the relevant question tokens.
In this work, we propose to use the dependency graph view of QDMR to improve QDMR parsing. We describe a conversion procedure that automatically maps QDMR structures into dependency graphs, using a structured intermediate logical form representation (Fig 1, bottom-left). Once we have graph supervision for every example, we train a dependency graph parser, in the spirit of Dozat and Manning [2018], where we predict a labeled relation for every pair of question tokens, representing the logical relation between the tokens. Unlike seq2seq models, this is a non-autoregressive parser, which decodes the entire output structure in a single step.
In addition, we study the effect of using dependency graphs as auxiliary supervision for a seq2seq QDMR parser, where the graph is decoded from the encoder representations. Towards that end, we propose a Latent-RAT encoder, which uses relation-aware transformer [Shaw et al., 2018] to explicitly represent the relation between every pair of input tokens. Relation-aware transformer has been shown to be useful for encoding graph structures in the context of semantic parsing [Wang et al., 2020].
Last, to fairly compare QDMR parsers that use different representations, we propose an evaluation metric, LF-EM, based on the aforementioned intermediate logical form. We show that LF-EM correlates better with human judgements compared to existing metrics.
We find that our graph parser leads to a small reduction in LF-EM compared to seq2seq models (0.47→0.44), but is 16x faster due to its non-autoregressive nature, and is by design more interpretable. Moreover, it has better sample complexity and outperforms the seq2seq model when trained on 10% of the data or less. When training a seq2seq model with the auxiliary graph supervision, the parser obtains similar performance as when trained on the entire dataset (0.471 LF-EM), but substantially improves performance when generalizing to new domains. Moreover, it performs better on examples with a large number of computation steps. Our code is available at https://github.com/matanhasson/ qdecomp_with_dependency_graphs.

heading: 2. Overview
text: The core of this work is to examine the utility of a dependency graph (DG) representation for QDMR. We propose conversion procedures that enable training and evaluating with DGs (see Fig. 2). First, we convert gold QDMR structures into logical forms (LF), where each computation step in QDMR is represented with a formal operator, properties and arguments (§3). Then, we obtain gold DGs by projecting the logical forms onto the question tokens (§4). Once we have question- DG pairs, we can train a graph parser. At test time, QDMRs and DGs are converted into LFs for evaluation. We propose a new evaluation metric over LFs (§3), and show it is more robust to semantic-preserving changes compared to prior metrics.
Our proposed parsers are in §5. On top of standard seq2seq models, we describe (a) a graph parser, and (b) a multi-task model, where the encoder of the seq2seq model is also trained to predict the DG.

heading: 3. QDMR Logical Forms
text: QDMR [Wolfson et al., 2020] is a text-based meaning representation focused on representing the meaning of complex questions. It is based on a decomposition of questions into a sequence of simpler executable steps (Fig. 1), where each step corresponds to a SQL-inspired operator (Table 5, §A.1). We briefly review QDMR and then define a logical form (LF) representation based on these operations. We use the LFs both for mapping QDMRs to DGs, and also to fairly evaluate the output of parsers that output either QDMRs directly or DGs.
QDMR Definition Given a question with n tokens, q = q1 . . . qn, its QDMR is a sequence of m steps s1, . . . , sm, where step si conceptually maps to a single logical operator oi ∈ O. A step, si, is a sequence of ni tokens s i = si1 . . . s i ni , where token s i j is either a question token ∈ Vq (or some inflection of it), a word from a constant predefined lexicon ∈ Vconst, or a reference token ∈ V iref = {#1, . . . ,#(i− 1)}, referring to a previous step. Fig. 3 shows an example for a question and its QDMR structure.
QDMR Logical Form (LF) Given a QDMR S = 〈q; s1, . . . , sm〉, its logical form is a sequence of logical form steps Z = 〈q; z1, . . . , zm〉. The LF step zi, corresponding to si , is a triplet zi = 〈oi, ρi, Ai〉 where oi ∈ O is the logical operator; ρi ∈ PROPoi are operator-specific properties; and Ai is a dictionary of arguments, mapping an operator-specific argument
η ∈ ARGoi,ρi to a span τ from the QDMR step si. For convenience, we denote zi with the string oi[ρi](ηi1 = τ i 1, . . . ). Table 1 provides a few examples for the mapping from QDMR to LF steps, and Table 5 (§A.1) provides the full list.
QDMR→LF We convert QDMRs to LFs with a rule-based method, extending the procedure for detecting operators from Wolfson et al. [2020] to also find properties and arguments. To detect properties we use a lexicon (see Table 6 in §A.1).
LF-based Evaluation (LF-EM) The official evaluation metric for QDMR1 is normalized EM (NormEM), where the predicted and gold QDMR structures are normalized using a rule-based procedure, and then exact string match is computed between the two normalized QDMRs. Since in this work we convert both QDMRs (§3) and DGs (§4) to LFs, we propose a LF-based evaluation metric.
LF-EM essentially involves computing exact match between the predicted and gold LFs. To further capture semantic equivalences, we perform more normalization steps, which for brevity are described in §A.2. We manually evaluate the metrics NormEM and LF-EM on 50 random development set examples using predictions from the CopyNet+BERT model (see §6). We find that both metrics have perfect precision (no false-positives); but the LF-EM covers more examples (52.0% vs 40.0%). Thus, it provides a tighter lower bound on the performance of a QDMR parser and correlates better with notions of semantic equivalence.
1. https://leaderboard.allenai.org/break/submissions/public

heading: 4. From LFs to Dependency Graphs
text: Given a QDMR decomposition S = 〈q; s1, . . . , sm〉, we construct a dependency graph G = 〈N , E〉, where the nodes N correspond to question tokens, and the edges E describe the logical operations, resulting in a graph with the same meaning as S.
The LF→DG procedure is shown in Fig. 4 and consists of the following steps: • Token alignment: align each token in the question to a token in a QDMR step (§A.3). • Spans Dependency Graph (SDG) extraction: construct a graph where each node
corresponds to a list of tokens in a QDMR step, and edges describe the dependencies between the steps (§A.4). • Dependency Graph (DG) extraction: convert the SDG to a DG over the question tokens. Here, we add span edges for tokens that are in the same step, and deal with some representation issues (§A.5).
Because we convert predicted DGs to LFs for evaluation, the LF→DG conversion must be invertible. Our conversion succeeds in 97.12% of the BREAK dataset [Wolfson et al., 2020].

heading: 5. Models
text: Once we have methods to convert QDMRs to DGs and LFs, and DGs to LFs, we can evaluate the advantages and disadvantages of standard autoregressive decoders compared to graph-based parsers. We describe three models: (a) An autoregressive parser, (b) a graph parser, (c) an autoregressive parser that is trained jointly with a graph parser in a multi-task setup. For a fair comparison, all models have the same BERT-based encoder [Devlin et al., 2019].
CopyNet+BERT (baseline) This autoregressive QDMR parser is based on the CopyNet baseline from Wolfson et al. [2020], except we replace the BiLSTM encoder with a
transformer initialized with BERT. The model encodes the question q and then decodes the QDMR S step-by-step and token-by-token. The decoder is an LSTM [Hochreiter and Schmidhuber, 1997] augmented with a copy mechanism [Gu et al., 2016], where at each time step the model either decodes a token from the vocabulary or a token from the input. Since the input is tokenized with word pieces, we average word pieces that belong to a single word to get word representations, which enables word copying. Training is done with standard maximum likelihood.
Biaffine Graph Parser (BiaffineGP) The biaffine graph parser takes as input the question q augmented with the special tokens described in §A.5 and predicts the DG by classifying for every pair of tokens whether there is an edge between them and the label of the edge. The model is based on the biaffine dependency parser of Dozat and Manning [2018], except here we predict a DAG and not a tree, so each node can have more than one outgoing edge.
Let H = 〈h1, . . . ,h|H|〉 be the sequence of representations output by the BERT encoder concatenated with the POS embeddings. The biaffine parser uses four 1-hidden layer feed-forward networks over each contextualized token representation ht:
hedge-headt = FF edge-head(ht),h edge-dep t = FF edge-dep(ht),
hlabel-headt = FF label-head(ht),h label-dep t = FF label-dep(ht).
The probability of an edge from token i to token j is given by σ(hedge-dep >
i Wedgeh edge-head j ),
where Wedge is a parameter matrix. Similarly, the score of an edge labeled by the label l from token i to token j is given by slij = h label-dep> i Wlh label-head j , where Wl is the parameter matrix for this label. We then compute a distribution over the set of labels L with softmax(s1ij , . . . , s |L| ij ).
Training is done with maximum likelihood both on the edge probabilities and label probabilities. Inference is done by taking all edges with edge probability > 0.5 and then labeling those edges according to the most probable label.
There is no guarantee that the biaffine parser will output a valid DG. For example, if an SDG node has an outgoing edge labeled with filter-sub and another labeled with project-sub, we cannot tell if the operator is FILTER or PROJECT. This makes parsing fail, which occurs in 1.83% of the cases. To create a SDG, we first use the span edges to construct SDG nodes with lists of tokens, and then add edges between SDG nodes by projecting the edges between tokens to edges between the SDG nodes. To prevent cases where parsing fails, we can optionally apply an ILP that takes the predicted probabilities as input, and outputs a valid DG. The exact details are given in our open source implementation.
Multi-task Latent-RAT Encoder (Latent-RAT) In this model, our goal is to improve the seq2seq parser by providing more information to the encoder using the DG supervision. Our model will take the question q (with special tokens as before) as input, and predict both the graph G directly and the QDMR structure S with a decoder.
We would like the information on relations between tokens to be part of the transformer encoder, so that the decoder can take advantage of it. To accomplish that, we use RAT transformer layers [Shaw et al., 2018, Wang et al., 2020], which explicitly represent relations
between tokens, and have been shown to be useful for encoding graphs over input tokens in the context of semantic parsing.
RAT layers inject information on the relation between tokens inside the transformer self-attention mechanism [Vaswani et al., 2017]. Specifically, the similarity score eij computed using queries and keys is given by eij ∝ hiWQ(hjWK + rKij )T , where WQ,WK are the query and key parameter matrices and the only change is the term rKij , which represents the relation between the tokens i and j. Similarly, the relation between tokens is also considered when summing over self-attention values ∑H j=1 αij(xjWV + r V ij ), where WV is the value parameter matrix, αij is the attention distribution and the only change is the term r V ij .
Unlike prior work where the terms rKij , r V ij were learned parameters, here we want these vectors to (a) be a function of the contextualized representation and (b) be informative for classifying the dependency label in the gold graph. By learning latent representations from which the gold graph can be decoded, we will provide useful information for the seq2seq decoder. Specifically, given the encoder output representations hi,hj for tokens i and j, we represent relations and compute a loss in each RAT layer as follows (Fig. 5):
rKij = FF K(hi − hj),
SK = RKW out + bK ∈ Rn×n×|L|, LossK = CE(SK).
FFK is a 1-hidden layer feed-forward network, RK ∈ R(n×n)×dtransformer is a concatenation of all rKij for all pairs of tokens, W
out ∈ Rdtransformer×|L| is a projection matrix that provides a score for all possible labels (including the NONE label).
We compute an analogous loss LossV for rVij and the final graph loss is Loss K + LossV
over all RAT layers. To summarize, by performing multi-task training with this graph loss we push the transformer to learn representations rij that are informative of the gold graph, and can then be used by the decoder to output better QDMR structures.

heading: 6. Experiments

heading: 6.1 Experimental Setup
text: We build our models in AllenNLP [Gardner et al., 2018], and use BERT-base [Devlin et al., 2019] to produce contextualized token representations in all models. We train with the Adam optimizer [Kingma and Ba, 2015]. Our Latent-RAT model includes 4 RAT layers, each with 8 heads. Full details on hyperparameters and training procedure in Appendix §A.7.
We examine the performance of our models in three setups:
• Standard : We use the official BREAK dataset.
• Sample Complexity (SC): We examine the performance of models with decreasing amounts of training data. The goal is to test which model has better sample complexity.
• Domain Generalization (DomGen): We train on 7 out of 8 sub-domains in BREAK and test on the remaining domain, for each target domain. The goal is to test which model generalizes better to new domains.
As an evaluation metric, we use LF-EM and also the official BREAK metric, NormEM, when reporting test results on BREAK.

heading: 6.2 Results
text: Standard Setup Table 2 compares the performance of the different models (§5) to each other and to the top entries on the BREAK leaderboard.
To assess the potential success of the Latent-RAT architecture, we add an oracle setup (termed Latent-RAToracle) where learned representations of the gold dependencies are fed into the RAT layers. Its outstanding performance (0.759 on the development set), indicates the benefit the sequence-to-sequence model produces from encoding the LF-based dependencies into the tokens representation.
As expected, initializing CopyNet with BERT dramatically improves test performance (0.388→0.47). The Latent-RAT seq2seq model achieves similar performance (0.471), and the biaffine graph parser, BiaffineGP, is slightly behind with an LF-EM of 0.44. Adding an ILP layer on top of BiaffineGP to eliminate constraint violations in the output graph improves performance to 0.454. The graph-head of the Latent-RAT, termed LatentRATgraph, gets close performance (0.435) to the biaffine graph parser, indicates that the hybrid architecture learns dependency representations.
While our proposed models do not significantly improve performance in the LF-EM setup, we will see next that they improve domain generalization and sample complexity. Moreover, since BiaffineGP is a non-autoregressive model that predicts all output edges simultaneously, it dramatically reduces inference time.
Last, the top entry on the BREAK leaderboard uses BART [Lewis et al., 2020], a pretrained seq2seq model (we use a pre-trained encoder only), which leads to a state-of-the-art LF-EM of 0.496.
Domain Generalization Table 3 shows LF-EM on each of BREAK’s sub-domains when training on the entire dataset (top), when training on all domains but the target domain (middle), and the relative drop compared to the standard setup (bottom). The performance of BiaffineGP and Latent-RAT is higher compared to CopyNet+BERT in the DomGen setup. In particular, the performance of Latent-RAT is the best in 7 out of 8 sub-domains, and the performance of BiaffineGP is the best in the last domain. Moreover, Latent-RAT outperforms CopyNet+BERT in all sub-domains. We also observe that the performance drop is lower for BiaffineGP and Latent-RAT compared to CopyNet+BERT. Overall, this shows that using graphs as a source of supervision leads to better domain generalization.
Sample Complexity Table 4 shows model performance as a function of the size of the training data. While the LF-EM of BiaffineGP is lower given the full training set (Table 2), when the size of the training data is small it substantially outperforms other models, improving performance by 3-4 LF-EM points given 1%-10% of the data. With 20%-50% of the data Latent-RAT and CopyNet+BERT have comparable performance.
Inference Time The graph parser, BiaffineGP, is a non-autoregressive model that predicts all output edges simultaneously, as opposed to a seq2seq model that decodes a single token at each step. We measure the average runtime of the forward pass for both
BiaffineGP and CopyNet+BERT and find that BiaffineGP has an average runtime of 0.08 seconds, compared to 1.306 seconds of CopyNet+BERT – a 16x speed-up.

heading: 6.3 Analysis
text: Model Agreement Figure 6a shows model agreement between the models from §5. Roughly 60% of the examples are predicted correctly by one of the models, indicating that ensemble of the three models could result in further performance improvement. The agreement of Latent-RAT with CopyNet+BERT (5.5%) and BiaffineGP (4.27%) is greater than the agreement of CopyNet+BERT with BiaffineGP, perhaps since it is a hybrid of a seq2seq and graph parser.
Length Analysis We compared the average LF-EM of models for each possible number of steps in the QDMR structure (Fig. 6b). We observe that CopyNet+BERT outperforms Latent-RAT when the number of steps is small, but once the number of steps is ≥ 5, Latent-RAT outperforms CopyNet+BERT, showing it handles complex decompositions better, and in agreement with the tendency of seq2seq models to struggle with long output sequences.
Error Analysis We manually analyzed randomly sampled errors from each model (§A.8). For all models, the largest error category (34-72%) is actually cases where the prediction is correct but not captured by the LF-EM metric, showing that the performance of current QDMR parsers is actually quite high.

heading: 7. Conclusion
text: In this work, we propose to represent QDMR structures with a dependency graph over the input tokens, and propose a graph parser and a seq2seq model that uses graph supervision as an auxiliary loss. We show that a graph parser is 16x faster than a seq2seq model, and that it exhibits better sample complexity. Moreover, using graphs as auxiliary supervision improves out-of-domain generalization and leads to better performance on questions that represent a long sequence of computational steps. Last, we propose a new evaluation metric for QDMR parsing and show it better corresponds to human intuitions.

heading: Acknowledgments
text: We thank Vivek Kumar Singh for his helpful ILP guidelines, and Tomer Wolfson for having kindly assisted in running our evaluation metric on our predictions for BREAK test set. This research was partially supported by The Yandex Initiative for Machine Learning, and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800).

heading: Dataset_for_Real-World_Visual_Reasoning_and_Compositional_CVPR_2019_paper.
text: html.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1988–1997. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.215. URL https://doi.org/ 10.1109/CVPR.2017.215.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-tosequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online, 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.703. URL https://www.aclweb.org/anthology/2020.acl-main.703.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097–6109, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1613. URL https://www.aclweb.org/anthology/P19-1613.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China, 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15-1142. URL https: //www.aclweb.org/anthology/P15-1142.
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864–8880, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.713. URL https://www.aclweb.org/anthology/2020.emnlp-main.713.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2074. URL https: //www.aclweb.org/anthology/N18-2074.
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. Achieving interpretability in compositional neural networks. In Association for Computational Linguistics (ACL), 2020.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418– 6428, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1644. URL https://www.aclweb.org/anthology/P19-1644.
Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1059. URL https://www.aclweb.org/anthology/N18-1059.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images. In International Conference on Learning Representations (ICLR), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017. URL https://proceedings.neurips. cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online, 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.677. URL https://www.aclweb.org/anthology/2020.acl-main.677.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302, 2018. doi: 10.1162/tacl a 00021. URL https://www.aclweb.org/anthology/Q18-1021.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183–198, 2020. doi: 10.1162/tacl a 00309. URL https://www.aclweb.org/anthology/2020.tacl-1.13.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://www.aclweb.org/ anthology/D18-1259.

heading: Appendix A. Appendices

heading: A.1 QDMR LF
text: Table 5 shows the different operators, their properties and examples of LFs. Table 6 shows terms that are used to identify the QDMR step operator’s properties. We use the same lexicon from BREAK [Wolfson et al., 2020] for detecting operators, extended with some specifications for numeric properties such as equals 0.

heading: A.2 LF-Based Evaluation (LF-EM)
text: In §3 we describe a LF-based evaluation metric that is based on normalization steps. We now elaborate the normalization process.
Given a logical form Z, we transform each step to a normalized form, and the final textual representation is given by representing each step as described in §3: OPERATOR[property ](arg=. . . ; . . . ). We apply the following steps (Fig. 7):
Remove and normalize tokens Each LF step includes a list of tokens in its arguments. In this normalization step, we remove lexical items, such as “max”, which are used to detect the operator and property (Table 6 in §A.1), as those are already represented outside the arguments. In addition, we remove words from a stop word list (Vaux, see Fig. 3). Finally, we use a synonym list to represent words in such a list with a single representative (countries→country, see Table 7).
Merge Steps QDMR annotations sometime vary in their granularity. For example one example might contain “return metal objects”, while another might have “return objects; return #1 that are metal”. This is especially common in FILTER and PROJECT steps. We merge chains of FILTER steps, as well as FILTER or PROJECT steps that follow a SELECT step. .
Reorder steps QDMR describes a directed acyclic graph of computation steps, and there are multiply ways to order the steps (Fig. 7). We recursively compute the layer of each step as layer(s) = maxs→s′ {layer(s′)}+ 1, where the maximization is over all the steps s refers to. We then re-order steps by layer and then lexicographically.
We manually evaluate the metrics NormEM and LF-EM on 50 random development set examples using predictions from the CopyNet-BERT model (see §6). We find that both (binary) metrics have perfect precision: they only assign credit when indeed the QDMR reflects the correct question decomposition, as judged by the authors. However, LF-EM covers more examples, where the LF-EM on this sample is 52.0, while NormEM is 40.0. Thus, LF-EM provides a tighter lower bound on the performance of a QDMR parser and is a better metric for QDMR parsing.

heading: A.3 Token Alignment
text: We denote the question tokens by q = q1 . . . qn and the ith QDMR step tokens by ∀i ∈ [1..m], si = si1 . . . s i ni . An alignment is defined by M = {(qi, s k j ) | qi ≈ skj ; i ∈ [1..n], k ∈ [1..m], j ∈ [1..nk]}, where by t ≈ t′ we mean t, t′ are either identical or equivalent. Roughly speaking, these equivalences are based on the BREAK annotation lexicon (Fig. 3) – in particular, the inflections of the question tokens Vq (e.g , “object” and “objects”), and equivalence classes on top of the constant lexicon Vconst (e.g , “biggest” and “longest”). See Table 7 (§A.2) for more details.
To find the best alignment M , we formulate an optimization problem in the form of an Integer Linear Program (ILP) and use a standard ILP solver.2 The full details are given as part of our open source implementation. The objective function uses several heuristics to assign a high score to an alignment that has the following properties (Fig. 8):
• Minimalism: Aligning each question token to at most one QDMR step token and vice versa is preferable. • Exact Match: Aligning a question token to a QDMR token that is identical is preferable. • Sequential Preference: Aligning long sequences from the question to a single step is
preferable. When a step has a reference token (#1 ), we take into account the tokens in the referenced step (see Fig. 8, top right).
• Steps Coverage: Covering more steps is preferable.

heading: A.4 Spans Dependencies Extraction
text: Given the QDMR, LF, and alignment M , we construct the Span Dependency Graph (SDG). Each QDMR step is a node labeled by a list of tokens (spans). The list of tokens is computed with the alignment M , where given a QDMR step sk, the list contains all question tokens
2. https://developers.google.com/optimization
qi, such that (qi, s k j ) ∈ M , where skj is a word in sk. The list is ordered according to the position in the question.
Edges in the SDG are computed using reference tokens. If step si has a reference token to step sj , we add a directed edge (si, sj) (we abuse notation and refer to SDG nodes and QDMR steps with the same notation). Each edge has a label, which is a triple consisting of the operator oi of the source node si, the property ρi of the source node, and the named argument ηiref that contains the reference token. For readability we denote the label triplet label(i, j) = 〈oi, ρi, ηiref〉 by oi-ηiref[ρi]. Figure 4 shows an extracted SDG.

heading: A.5 SDG→DG
text: We construct a DG by projecting the SDG on the question tokens. This is done by: (a) For each SDG node and its list of tokens, add edges between the tokens from left-to-right with a new span label (black edges in Fig 4); (b) use the rightmost word in every span as its representative for the edges between different spans.
However, this transformation is non-trivial for two reasons. First, some SDG nodes do not align to any question token. Second, some question tokens align to multiple SDG nodes, which does not allow the DG to be converted back to an SDG unambiguously for evaluation. We resolve such representation issues by adding special tokens at the end of the sequence and using them as extra tokens for alignment. We give the details in §A.6.

heading: A.6 DG Representation Issues
text: In §A.5 we describe the conversion procedure from SDG to DG. This transformation is non-trivial for two reasons. First, some SDG nodes do not align to any question token. Second, some question tokens align to multiple SDG nodes, which does not allow the DG to be converted back to an SDG unambiguously for evaluation.
We now explain how we resolve such representation issues, mostly based on adding more tokens to the input question. Fig. 9 illustrates the different types of challenges and our proposed solution.
Domain-specific concepts QDMR annotators were allowed to use a small number of tokens that are pragmatically assumed to exist in the domain (Vstore in Fig. 3). For example, when annotating ATIS questions [Hemphill et al., 1990], the word “flight” is allowed to be used in the QDMR structure even if it does not appear in the question, since this is a flight-reservation domain. We concatenate all the words in Vstore to the end of each question after a special separator token, which allows token alignment (§A.3) to map such QDMR steps to a question word (Fig. 9, top).
Empty SDG nodes Some steps only contains tokens that are not in the question (e.g., “Number of #2” in Fig. 9 bottom), and thus their list of tokens in the SDG node is empty. In this case, we cannot ground the SDG node in the question. Therefore we add a constant number of dummy tokens, [DUM], which are used to ground such SDG nodes.
Single tokens to multiple SDG nodes A single question token can be aligned to multiple SDG nodes. Recall the tokens of each SDG nodes are connected with a chain of span edges. This leads to cases where two chains that pass through the same question token cannot be distinguished when converting the DG back to an SDG for evaluation. We solve this by concatenating a constant number of special [DUP] tokens that conceptually duplicate another token by referring to it with a new duplicate label. Now, each span chain uses a different copy of the shared token by referring to the [DUP] instead of the original one.

heading: A.7 Experiments Parameters
text: CopyNet+BERT The LSTM decoder has hidden size 768. We use a batch size of 32 and train for up to 25 epochs (∼35k steps) with beam search of size 5.
BiaffineGP The POS embeddings are of size 100. The four FFNs consist of 3-layers with hidden size 300 and use ELU activation function. We use dropout of rate 0.6 on the contextualized encodings, and of rate 0.3 on the FF representations. We use a batch size of 32 and train for up to 80 epochs (∼111k steps).
Latent-RAT We stack 4 relation-aware self-attention layers on top of the contextualized encodings, each with 8 heads and dropout with rate 0.1. The FFNs for relation representation uses 3-layers with hidden size of 96, ReLU activation function and dropout rate of 0.1. We tie the layers, and multiply the graph loss by 100. The rest is identical to the CopyNet-BERT configuration.
Optimization We used the Adam optimizer [Kingma and Ba, 2015] with the default hyperparameters. The learning rate changes during training according to the slanted triangular schema, in which it linearly increases from 0 to lr for the first warmup steps = 0.06 ·max steps, and afterwards linearly decreases back to 0. We use learning rate of 1 · 10−3, and a separate learning rate of 5 · 10−5 for the BERT-based encoder.

heading: A.8 Error Analysis
text: We randomly sampled 50 errors from each model and manually analyzed them. Table 8 describes the error classes for each model, and Appendix §A.9 provides examples for these classes. Each example can have more than one error category.
For all models, the largest error category is actually cases where the prediction is correct but not captured by the LF-EM metric: 56% for CopyNet+BERT, 34% for BiaffineGP and 72% for Latent-RAT. This shows that the performance of current QDMR parsers is actually quite high, but capturing this with an automatic evaluation is challenging.
When taking a more loose definition of correctness (termed “Correct (soft)”), the rates increase to: 62% for CopyNet+BERT, 50% for BiaffineGP and 80% for Latent-RAT. In these cases we focus on the final returned result correctness, and ignore unused steps, duplicate steps or references, implicit information (commonsense), etc. §A.10 provides some examples for such predictions.
Cases where the output is correct include: • Equivalent Solutions: the prediction is logically equivalent to the gold structure.
• Elaboration Level : the model prediction is more/less granular compared to the gold structure, but the prediction is correct.
• Redundancy : additional information is predicted/omitted that does not effect the computation. For example the second occurrence of “yards” in “2. return yards of #1; 3. #1 where #2 is lower than 10 yards”.
• Wrong Gold - cases where the predication is more accurate than the gold decomposition. The main classes of errors are: • Missing Information: missing steps, missing references or missing tokens that affect the
result of the computation.
• Additional Steps: duplicate steps or additional steps that change the result of the computation.
• Wrong Global Structure: The computation described by the predicted structure is wrong (for example, addition instead of subtraction).
• Wrong Step Structure: incoherent structure of a particular step that cannot be mapped to a proper structure.
• Out of Vocabulary : seq2seq models sometimes predict tokens that are not related to the question nor the decomposition. For example, ”rodents” in a question about flowers.
The seq2seq models preserve better global and local structure (Wrong Global Structure, Wrong Step Structure). The graph parser, by design, has no Out of Vocabulary tokens and less Redundancy, but suffers from incoherence (Additional Steps, Wrong Global Structure, Wrong Step Structure) due to non-autoregressiveness. The combined architecture, Latent-RAT seems to utilize the dependence information to improve Redundancy and Out of Vocabulary issues as well as the additional/missing information (Missing Information, Additional Steps) compared to the seq2seq model it is based on - CopynNet+BERT.

heading: A.9 Error Analysis Examples
text: Some examples for each error class from §A.8. The gold decompositions are given on left, and the predictions are on the right.

heading: A.10 Correct (Soft) Examples
text: Examples for semi-correct predictions (§A.8). For convenience, some of the LFs are given in a QDMR form.

Review summaries:
The paper investigates the utility of dependency graph (DG) representation in converting questions to Question Decomposition Meaning Representation (QDMR) annotations. As an intermediate step, the paper proposes to convert QDMR annotations to logical form (LF) representation (this was also done in the original QDMR paper). This LF representation is used to then extract a DG representation from it in a deterministic manner. The DG can also be converted to LF in a similar manner. Additionally, the paper also proposes a LF based evaluation metric which apparently is a better measure for semantic equivalence between QDMRs as compared to the previously proposed metric. 

The authors propose a novel view of decomposing questions into QDMR steps using dependency parsing. Specifically, they propose representing QDMR decompositions as a dependency graph in which the edges correspond to computation steps. Using this representation, they present two new methods for decomposing questions; a biaffine graph parser that directly predicts all of the edges, and a seq2seq parser using a relation-aware transformer to incorporate dependency information as additional supervision. The first model shows a significant speed-up at the cost of some accuracy, while the second model is shown to improve generalization performance on unseen domains. Additionally, the authors propose a new evaluation metric for QDMR parsing that better correlates with human judgements. 

This paper is in the area of Question Decomposition Meaning Representation (QDMR), originally introduced by Wolfson et al (2020). The core idea proposed in this paper is a new representation based on dependency graphs; based on such a graph, QDMR logical forms can be computed using a deterministic procedure. In addition, the paper describes an evaluation protocol based on the logical forms. Finally, the paper also introduces two different automatic parsing methods that produce the dependency graphs, one of which uses the well-known biaffine approach by Dozat et al., and one using an autoregressive approach based on the Relation-Aware Transformer; the biaffine parser is faster but somewhat less accurate.