Single Sequence Prediction over Reasoning Graphs for Multi-hop QA
Year: 2023

Abstract:
Recent generative approaches for multi-hop question answering (QA) utilize the fusion-indecoder method (Izacard and Grave, 2021) to generate a single sequence output which includes both a final answer and a reasoning path taken to arrive at that answer, such as passage titles and key facts from those passages. While such models can lead to better interpretability and high quantitative scores, they often have difficulty accurately identifying the passages corresponding to key entities in the context, resulting in incorrect passage hops and a lack of faithfulness in the reasoning path. To address this, we propose a single-sequence prediction method over a local reasoning graph (SEQGRAPH)1 that integrates a graph structure connecting key entities in each context passage to relevant subsequent passages for each question. We use a graph neural network to encode this graph structure and fuse the resulting representations into the entity representations of the model. Our experiments show significant improvements in answer exact-match/F1 scores and faithfulness of grounding in the reasoning path on the HotpotQA dataset and achieve stateof-the-art numbers on the Musique dataset with only up to a 4% increase in model parameters.

heading: 1 Introduction
text: Multi-hop Question Answering (QA) involves reasoning over multiple passages and understanding the relationships between those pieces of information to answer a question. Compared with singlehop QA, which often extracts answers from a single passage, multi-hop QA is more challenging as it requires a model to determine the relevant facts from multiple passages and connect those facts for reasoning to infer the final answer.
To tackle multi-hop QA, recent works have investigated large pretrained generative models (Lewis
∗Equal contribution 1Code/Models will be released at https://github.com/
gowtham1997/SeqGraph
et al., 2020b; Roberts et al., 2020; Brown et al., 2020) and demonstrated their effectiveness over traditional extractive models (Chen et al., 2017). Compared with extractive models, the ability of generative models to effectively aggregate and combine evidence from multiple passages proves advantageous for multi-hop QA. In particular, Izacard and Grave (2021) propose a method called FID (Fusion-in-Decoder), which leverages passage retrieval with a generative model, such as T5 (Raffel et al., 2020) or BART (Lewis et al., 2020a), to achieve state-of-the-art performance on various single-hop QA tasks. However, this approach does not extend well to multi-hop QA tasks (Yavuz et al., 2022), as it sorely relies on a black-box generative model to generate answers directly without explicitly modeling the multi-hop reasoning process. Additionally, FID encodes multiple context passages independently for multi-hop QA, ignoring the structural and semantic relationship between these passages (Yu et al., 2022). Building on FID, PATH-FID (Yavuz et al., 2022) addresses the interpretability issue by training a model to generate a reasoning path that contains supporting pas-
11466
sage titles, facts, and the final answer. However, our analysis of PATH-FID outputs shows disconnected reasoning with incorrect passage hops in the model’s reasoning path which affects final answer generation. Recently, there have been multiple techniques (Jiang and Bansal, 2019; Lee et al., 2021; Ye et al., 2021) to counter disconnected reasoning which operate at the dataset level, using adversarial training, adding extra annotations or using dataset rebalancing for training. While these approaches optimize models to mitigate disconnected reasoning (Trivedi et al., 2020), the performance on the original test set often suffers from a significant decrease.
In this paper, we propose a single-sequence prediction method over a local reasoning graph (SEQGRAPH) that integrates a graph structure connecting key entities in each context passage to relevant subsequent passages for each question. Different from the prior works, our method not only mitigates the disconnected reasoning issue but also maintains robust performance on the original dataset. Intuitively, for each multi-hop question, our method leverages the structural relationship between different passages to learn structured representations through a graph neural network (GNN) (Hamilton et al., 2017; Kipf and Welling, 2017). The structured representations are fused to bias the generative model toward predicting a faithful, connected reasoning path which improves answer predictions. Our experiments on the HOTPOT-QA dataset (Yang et al., 2018) show clear improvements in exact-match(EM)/F1 scores compared to generative baselines in the distractor setting while minimizing disconnected reasoning quantified by the DIRE score (Trivedi et al., 2020). We also achieve the state-of-the-art performance on the MUSIQUE-Answerable test dataset (Trivedi et al., 2022a) with a 17-point improvement in answer F1 over the current best-performing model in the end-to-end (E2E) category.
To summarize, our contributions are as follows:
• We propose an interpretable single-sequence prediction approach over local reasoning graphs, SEQGRAPH, to bias the model representations
• SEQGRAPH achieves notable performance improvements on two multi-hop QA benchmarks, HOTPOT-QA and MUSIQUE (SOTA), with only a minimal increase in the model size.
• SEQGRAPH reduces disconnected reasoning as measured by DIRE score while maintaining
strong performance gains on the original dataset.

heading: 2 Preliminaries
text: Problem Setup: In a multi-hop QA task, each QA pair in a labeled dataset D is given along with a set of N passages, Pq = {p1, p2, ..., pN}, i.e., (q, a,Pq) ∈ D, where a passage has its title and content pi = (ti, ci). The task is to learn a model parameterized θ to generate an answer string a for the given question q and Pq. In this paper, we focus on the distractor setting, where Pq is given for each question and contains m distractors that are not useful to the answer prediction. Thus, this task requires a model to reason over multiple hops of the remaining N −m relevant passages. In addition to predicting the final answer a, we also aim to train a model to predict a reasoning path R of important elements (e.g., relevant passage titles, supporting facts in a passage) that lead to the final answer.
Multi-hop QA as Single Sequence Generation: Recent generative question answering (QA) approaches (e.g., FID (Izacard and Grave, 2021), PATH-FID (Yavuz et al., 2022)) utilize an encoderdecoder model as the backbone to generate answers in a single text sequence. In particular, FID is one of the popular formulations. Specifically, for each passage pi = (ti, ci) ∈ Pq of a question q, FID encodes a combined sequence of the question, the passage title and contents into an embedding. These embeddings for all passages are concatenated as inputs to the decoder for generating the final answer.
PATH-FID builds upon this by explicitly modeling a reasoning path as part of the generation output in addition to the answer. Specifically, special index tokens [fi] are added to demarcate all sentences in each passage context. The sentences supporting the prediction of a final answer are considered facts. The decoder is then trained to generate the reasoning path R as a linearized sequence consisting of the passage titles and the index tokens of facts used within those passages to obtain the final answer. Figure 1 shows an example of a reasoning path.
Disconnected Reasoning in PATH-FID: Since the model predictions now include the reasoning path, we can analyze which facts in the passage are utilized by the model to determine the next passage to hop to and arrive at the final answer. For a perfectly faithful model, all predictions with correct answers should have correctly identified passages and facts. However, due to the presence
of shortcuts in the datasets as well as the model’s predicted reasoning path not being faithful, we observe model predictions containing correct final answers but incorrect identification of passage titles or facts. This unfaithful prediction issue is referred to as disconnected reasoning (Trivedi et al., 2020). Different from PATH-FID, we use the presence of a local graph structure between different passages in the context to bias the representations of the model and help alleviate this problem.

heading: 3 Method
text: In this section, we describe our proposed method for solving disconnected reasoning for multi-hop QA in the distractor setting.
Overview: Our method first constructs a local graph over passage contexts for each question (§3.1), and integrates the graph information with the key entities to improve the generation of reasoning paths (§3.2). Different from prior works that encode all the passages independently, we connect the passages through the key pivot entities into a local graph for a question, which allows us to encode structural representations across passages by a graph neural network. These graph structured representations are then fused with the contextualized text representations from a text encoder, guiding the model to leverage structural information to alleviate disconnected reasoning over passages.

heading: 3.1 Graph Construction
text: In contrast to the full-wiki setting where a model must retrieve relevant passages from Wikipedia or a large corpus, the distractor setting provides the model with a list of N passages Pq consisting of N − m relevant passages and m distractors for each question q. Conventionally, these passages are collected from Wikipedia, as Wikipedia remains one of the largest faithful knowledge sources available for public usage. Even for text passages out of Wikipedia, there are existing out-of-box entity linkers (e.g., SLING (Ringgaard et al., 2017), BLINK (Wu et al., 2020)) that can identify key entities from texts and link them to their Wikipedia pages. As a result, each provided passage may contain pivot entities with hyperlinks connecting to their corresponding Wikipedia pages. We exploit such entity hyperlinks to construct a local directed graph G = (N ,L) containing two types of nodes (i.e., entities and passage titles) and links between these nodes. Specifically, for each pivot entity e in
a passage pi, we create a link from e to the title tj of another passage pj (denoted as le→tj ) whenever the entity span e points to a Wikipedia article that contains the passage pj .
For example, an entity span “David Noughton” appears in the passage context: “An American Werewolf in London is a 1981 horror comedy film starring David Noughton, Jenny Agutter. ...” This entity would be connected to a passage with the title of “David Walsh Noughton”, forming the link (David Noughton[Entity] → David Walsh Noughton[Passage]). If there are multiple passages with the title “David Walsh Noughton” among the N passages, the entity span would be connected to all of them with distinct links. Figure 1 shows an example of an entity-passage graph.

heading: 3.2 Entity-to-Passage Fusion
text: Next, we describe how we encode such a local directed graph into vector representations for all nodes and fuse these node representations with the contextualized text representations of the corresponding entities from the language model.
We utilize the same model as PATH-FID with a pre-trained T5 model as our backbone architecture. The input for this method consists of the N sequences, where each sequence is a concatenation of the question q, the title and contents of a passage pi from the collection pi ∈ Pq together with their indicator tokens, denoted as Si below:
Si := [Question] q [Title] ti [Content] ci (1)
Given the T5’s encoder of M transformer layers, we first encode Si through the first L layers to obtain the intermediate hidden representations ZLi in Eq. (2), which capture the shallow contextualized information of the input sequence.
ZLi = TextEncoder(Si, L) (2)
We utilize these shallow representations to initialize the node embeddings for a graph neural network. Specifically, we extract the representations of the entity spans or passage title spans (i.e., nodes in the graph G) from ZLi according to their span positions [a, b] in Si. Next, for a text span Si,a:b representing either an entity or a title in Si, we average the extracted representations of the text span to obtain an initial node embedding, i.e., n = avg(ZLi,a:b). Finally, we stack the initial embeddings for all nodes denoted as N and apply
a graph neural network (GNN) to further encode the structural embeddings on the graph G:
ZG = GraphEncoder(N,G) (3)
As we record the text span position [a, b] for each node in G, we can leverage the node embeddings ZG to construct a new structured representation ZGi (with the same size as Z L i ) for each sequence Si where we fill in the node embeddings from ZG to their corresponding text span positions [a, b] in Si and fill in 0 to the other non-span positions.
Finally, we fuse the contextualized text representations ZLi from the text encoder and the structured node representations ZGi by an aggregation operator ⊕, and pass them to the remaining layers of the text encoder to obtained the fused representations Si for each input sequence Si:
Si = TextEncoder(ZGi ⊕ ZLi ,M − L) (4)
In this work, the aggregation operator used is a simple addition. Complex aggregation mechanisms
such as learning a weighted combination of the representations can be explored in future work.
We concatenate the fused representations Si from all of the N context sequences to form S = [S1;S2 · · · ;SN ]. Subsequently, S is passed as inputs to the T5 decoder that estimates the conditional probability Pθ(R|S) of predicting a reasoning path R. Depending on the annotations in different datasets, a reasoning path R can take various formats. For example, the reasoning path takes the form “R := [title] ti [facts] fi [answer] a” for HOTPOT-QA and “R := [title] ti [intermediate_answer] ansi [answer] a” for MUSIQUE. We also investigate variants of reasoning paths for MUSIQUE in our experiments. As we can construct ground-truth reasoning paths R∗ during training, the model is optimized using a cross-entropy loss between the conditional probability Pθ(R|S) and R∗.

heading: 4 Experimental Setting
text: In this section, we elaborate on the datasets, the baseline models and the variants of SEQGRAPH we consider for our experiment settings. We consider two multi-hop QA datasets, HOTPOT-QA and MUSIQUE. Since SEQGRAPH is primarily focused only on improving the efficacy of encoding, we consider only the distractor setting for both datasets. Table 4 shows the standard train/dev/test statistics.
HOTPOT-QA: The final answer to each question in the distractor setting is extracted from 10 passages. The dataset includes two main types of questions: bridge (80%) and comparison (20%). Bridge questions often require identifying a bridge entity in the first passage to correctly hop to the second passage that contains the answer, while comparison questions do not have this requirement. Each question is also provided with annotations of 2 supporting passages (2-hop) and up to 5 corresponding relevant sentences as their supporting facts.
MUSIQUE: MUSIQUE has questions that range in difficulty from 2 to 4-hops and six types of reasoning chains. MUSIQUE uses a stringent filtering process as well as a bottom-up technique to iteratively combine single-hop questions from several datasets into a k-hop benchmark that is more difficult than each individual dataset and significantly less susceptible to the disconnected-reasoning problem. Unlike HOTPOT-QA, MUSIQUE does not provide annotations of relevant sentences but provides supporting passage titles, question decomposition(decomposition of a multi-hop question into simpler 1-hop sub-questions) and also intermediate answers to the decomposed questions. Given this variety, we use the following reasoning path variants to train the model to generate:
• DA: Question decomposition and final answer • SA: Supporting titles and final answer • SIA: Supporting titles, intermediate answers
and final answer • DSIA: Question decomposition, supporting ti-
tles, intermediate answers and final answer
Table 6 shows an example of different reasoning paths. While the last variant (predicting every decomposition/intermediate answer or support title) is more interpretable, it encounters the challenge of producing a long sequence. SIA is our bestperforming reasoning path variant which is used for all of our results and analysis.

heading: 4.1 Models in Comparison
text: Our main baselines are generative approaches to multi-hop QA that include and build upon the FID approach. For all of the models, we use the pretrained T5 encoder-decoder as the backbone and consider two sizes—base and large variants.
• FID: Model generation includes only the final answer.
• PATH-FID: Model generation includes the reasoning path as well as the final answer.
• SEQGRAPH: Model that utilizes a fusion of representations from the language model and the Graph Neural Network. Similar to PATH-FID, we train the model to generate the reasoning path in addition to the final answer.

heading: 4.2 Evaluation Metrics
text: For both HOTPOT-QA and MUSIQUE, we use the standard quantitative metrics of exact-match and F1 scores to evaluate the quality of predicted answers. For models that predict the reasoning path in addition to the final answer, we can quantify how accurately they can identify the supporting facts (or supporting titles for MUSIQUE) using the SupportEM and Support-F1 scores Yang et al. (2018).
To quantify the level of disconnected reasoning, we compute dire F1 scores on the answer spans (Answer), supporting paragraphs (Suppp), supporting sentences (Supps), joint metrics (Ans+Suppp, Ans+Supps) of the Dire HOTPOT-QA subset.

heading: 4.3 Implementation details
text: We train all models using an effective batch size of 64. We use an initial learning rate of 1e-4, a linear rate scheduler, a warmup of 2,000 steps (1,000 steps for MUSIQUE), and finetune the models for 10 epochs. For SEQGRAPH, we use GAT (Veličković et al., 2017) for our GNN layers. A maximum sequence length of 256 tokens is used for constructing the input. All experiments have been conducted on a machine with either 4×40G A100 GPUs or 4×80G A100 GPUs. A detailed list of hyperparameters can be found in Appendix E.

heading: 5 Results and Analysis
text: In this section, we present the main results of the baselines and our proposed approach on HOTPOTQA and MUSIQUE (§5.1), and then perform finegrained analysis thereafter.

heading: 5.1 Multi-hop Performance
text: The quantitative performance of the models in terms of exact-match and F1 scores for both the final answer and the predicted supports are shown in Table 1. We find that across both model sizes (BASE and LARGE), explicitly predicting the reasoning path helps PATH-FID in improving the answer EM and F1 scores over the vanilla FID approach. By biasing the model with graph representations, SEQGRAPH outperforms the baselines on both the HOTPOT-QA and the MUSIQUE datasets.
SEQGRAPH achieves a 2-point improvement in both answer and support EM when considering the base variant and 1.5 point improvement for the large variant on the dev set of HOTPOT-QA.
On the more challenging MUSIQUE dataset, we observe stronger results from SEQGRAPH where it records up to a 4-point improvement in both answer and support scores across both model sizes on the dev set. On the test set (in Table 8 of the appendix), the current best performing approach is a two stage ROBERTA/ LONGFORMER-Large model, Select-Answer, where the passage selection/ranking and answer generation stage is optimized separately using different models. SEQGRAPH-Large achieves state-of-the-art numbers on Answer-F1 with a 5-point improvement over the Select-Answer model2 even though it is a single stage approach. When comparing with the top score in the end-to-end (E2E) category which all of our models belong to, SEQGRAPH gets a massive 17-point improvement in answer F1 and a 9-point improvement in support F1 establishing the efficacy of our approach. It should also be noted that all of the current models on the leaderboard are discriminative approaches with an encoder-only model
2https://leaderboard.allenai.org/musique_ans/
(LONGFORMER-Large) encoding a very long context length of 4,096, while all of our models are generative in nature with a much smaller context length of 256. MUSIQUE is also designed to be more challenging than HOTPOT-QA and explicitly tackles the issue of disconnected reasoning during dataset curation, making it harder for the model to take shortcuts and cheat. The larger performance improvements of SEQGRAPH on MUSIQUE compared to HOTPOT-QA showcases the advantage of our proposed approach, providing promising results for further research in this direction to mitigate disconnected reasoning.

heading: 5.2 Faithfulness of Reasoning Paths
text: We follow Yavuz et al. (2022) to perform analysis at the passage and individual fact level to determine how faithful the generated reasoning paths are across different models.
Predicted Answer in Predicted Titles/Support: how often are the predicted answers found in one of the predicted passages or in the predicted supporting facts3.
Gold Answer in Predicted Titles/Support: how often are the gold answers found in one of the predicted passages or in the predicted supporting facts.
Predicted Answer in Gold Titles/Support: how often are the predicted answers found in one of the gold passages or in the gold supporting facts.
Figure 3 shows the described faithfulness metric scores on HOTPOT-QA. We find that SEQGRAPH
3We do this analysis only on Bridge type questions where the final answer span can be found in context passages, unlike comparison questions where the final answer is usually yes/no
is more faithful with a 0.5-1.5% improvement over PATH-FID across all considered categories.

heading: 5.3 Performance vs Number of hops
text: We break down the final answer exact-match and F1 scores based on how many supporting facts(or titles for Musique) are required to answer the question. Figure 5 shows this performance breakdown for HOTPOT-QA and Figure 6 shows it for MUSIQUE. We observe that SEQGRAPH improves over PATHFID in the cases where the support includes two or three supporting facts (or titles), but the answer EM takes a hit when the number of supporting facts(titles) ≥ 4. We notice that SEQGRAPH has a higher support EM over PATH-FID in such cases where shortcuts may exist in the dataset and PATHFID relies on those shortcuts to get a higher answer EM but a lower support EM. Section §5.4 quantifies the extent to which PATH-FID suffers from disconnected reasoning as compared to SEQGRAPH.

heading: 5.4 Probing Disconnected Reasoning
text: HOTPOT-QA suffers from information leakage in the form of reasoning shortcuts leading to disconnected reasoning. This affects the generalization
capability of such models and inflates the performance on the evaluation sets. Table 4 shows some qualitative examples of disconnected reasoning in PATH-FID that are avoided by SEQGRAPH
Trivedi et al. (2020) construct a probe of HOTPOT-QA by splitting the two supporting paragraphs for the original question across two questions. If the model can answer modified questions correctly without the complete context, it suggests that the model uses disconnected reasoning for the original question. By measuring the performance of a model on such a dataset, we arrive at the DIRE score with a higher value implying more disconnected reasoning. Table 2 shows the DIRE scores for the various models. We see that SEQGRAPH resorts to lower disconnected reasoning compared to PATH-FID while maintaining strong performance gains on the original evaluation set.

heading: 5.5 Comparison with PathFiD+
text: Yavuz et al. (2022) extend PATH-FID and introduce PATH-FID + to improve the cross-passage interactions before feeding to the FiD decoder and show an improvement of 7 EM points and achieve state-of-the-art results on HOTPOT-QA distractor
dataset. However, we find the following limitations of the approach:
Hop-assumption: PATH-FID + adds pairs of contexts as input to the FID encoder, which assumes a fixed number of hops (in case of HOTPOTQA, two) and doubles the input sequence length, leading to increased training time.
Multi-step: To efficiently encode pairs of passages (instead of inefficient ( N 2 ) passages, where N is the total number of passages), PATH-FID + also needs to run the vanilla PATH-FID or train another model to choose the first relevant context P∗ to jump to and then construct pairs (P∗, Pn). This makes it inefficient and not scalable to questions with higher hops or complex datasets like MUSIQUE
In contrast, our approach does not make any assumptions about the number of hops and is scalable. It produces output in a single shot without requiring multiple steps or increased sequence length. While PATH-FID + may achieve stronger performance in 2-hop HOTPOT-QA, our proposed method is more general, efficient and scalable, making it a more
practical solution for real-world applications and also easily extendable to open-domain setting.

heading: 6 Related Works
text: Multihop question answering requires a model to perform reasoning over multiple pieces of information, utilizing multiple sources and inferring relationships between them to provide a correct answer to a given question. There have been various approaches and datasets proposed for training QA systems, such as HotpotQA (Yang et al., 2018), IIRC(Ferguson et al., 2020) and Musique (Trivedi et al., 2022b).
In the HOTPOT-QA full-wiki setting, the task is to find relevant facts from all Wikipedia articles and then use them to complete the multi-hop QA task. Retrieval models play an important role in this setting, such as DPR (Karpukhin et al., 2020), which focuses on retrieving relevant information in the semantic space. Other methods, such as Entitiescentric (Das et al., 2019), and Golden Retriever (Qi et al., 2019), use entities mentioned or reformulated in query keywords to retrieve the next hop
document. Additionally, PathRetriever (Asai et al., 2020) and HopRetriever (Li et al., 2020) use RNN to select documents to form a paragraph-level reasoning path iteratively. The above methods mainly focus on the open-domain setting (full-wiki) and improve the retriever’s performance and do not address the disconnected reasoning problem.
Multiple techniques (Jiang and Bansal, 2019; Lee et al., 2021; Ye et al., 2021) to counter disconnected reasoning operate at the dataset level, using adversarial training, adding extra annotations or using dataset augmentations to get a balanced train set and prevent the model from cheating.
We highlight differences between our approach and other related works on HOTPOT-QA-distractor and other works that combine language models with graphs below :
Generative approaches: Our generative-FiD approach differs from others using KG/GNN (Ju et al., 2022; Yu et al., 2022) as we use an entity-passage graph with Wikipedia hyperlinks. Also, our focus is primarily on the distractor setting of multi-hop QA, while other baselines (Ju et al., 2022; Yu et al., 2022) are either single-hop or improving retrieval in open-domain setting
Pipeline vs single-stage: Other baselines (Tu et al., 2019; Chen et al., 2019; Qiu et al., 2019; Wang et al., 2021; Li et al., 2023) use a pipeline approach with distinct encoder models in the reasoning process, while we use a single-stage, oneshot prediction process without assumptions on the number of hops.
Graph construction: Other methods (Tu et al., 2019; Qiu et al., 2019) select relevant passages heuristically from among distractors to construct graphs. However, we construct our entity-passage graph on all passages (including distractors) and fuse the representations in the encoder.
While a direct comparison with pipeline-based approaches is not possible or fair, we provide comparisons in Table 3 for completeness.

heading: 7 Conclusion
text: In this paper, we propose SEQGRAPH, an approach that utilizes the structured relationship between passages in the context of multi-hop questions to reduce disconnected reasoning. We construct a localized entity-passage graph using Wikipedia hyperlinks, encode it using a GNN, and fuse the structured representations with the text encoder for predicting a reasoning path. Our approach results in strong performance gains in terms of both answer and support EM/F1 on HOTPOT-QA and reduces disconnected reasoning measured using DIRE score. We also obtain state-of-the-art performance on the more challenging MUSIQUE benchmark with a 17-point improvement in answer F1 over the current best end-to-end(E2E) model. Experimenting with sophisticated methods of encoding the graph structure and fusing the text and graph representations can be explored in future work.
Limitations
We identify the following limitations of our work:
Longer Output Sequences While outputting the reasoning path as a single short sequence makes the model more interpretable, it increases the challenge of producing a long /coherent sequence when the question is complex (more than 3 hops). Producing a longer sequence also increases the inference time. Simplifying this output while not sacrificing interpretability is a good future direction
Entity Identification Our method needs wikipedia outlinks or a entity linker to construct a localized graph for every question. Generalizing this step by pretraining the model to do entity linking (Févry et al., 2020; Sun et al., 2021; Verga et al., 2020) might eliminate the need to use an external module.

heading: A Breakdown of Performance by Question Type - HOTPOT-QA

heading: B Reasoning Path variants in MUSIQUE
text: The different reasoning path variants that can be constructed based on ground truth can be found in Table 6. Results of training baselines on these different variants can be found in Table 7

heading: C Performance by Number of Hops - Graphs
text: We hypothesize that the answer F1 of SEQGRAPH on questions with ≥ 4 hops gets impacted due to presence of shortcuts since the support F1 score is higher than PATH-FID.

heading: D Comparison of Musique-Answerable test F1 scores
text: Table 8 shows the comparison of our models with the current best performing ones on the MUSIQUEAnswerable test set leaderboard. Our End-to-End single stage model SEQGRAPH-large trained to output title + intermediate answers (SIA) outperforms the Longformer-Large(Beltagy et al., 2020) End-to-End model by 17 points in answer F1 and by 9-points in support F1. Furthermore, we also outperform the current state-of-the-art SA model which is a two stage model (Roberta Large(Liu et al., 2019) + Longformer Large) by 5 points on Answer F1 and 3 points on Support F1.

heading: E Hyperparameter Settings
text: Tables 9, 10, 11 detail the hyperparameters we use for FID,PATH-FID and SEQGRAPH for HOTPOT-QA and MUSIQUE.
The 2-layer GNN module is 17M parameters for the large model and 9.5M for the base, accounting for only upto 4% increase in model parameters.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: General paraphrasing of content
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix E

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Experimenting with sophisticated methods of encoding the graph structure and fusing the text and graph representations can be explored in future work. Limitations
We identify the following limitations of our work:
Longer Output Sequences While outputting the reasoning path as a single short sequence makes the model more interpretable, it increases the challenge of producing a long /coherent sequence when the question is complex (more than 3 hops). Producing a longer sequence also increases the inference time. Simplifying this output while not sacrificing interpretability is a good future direction
Entity Identification Our method needs wikipedia outlinks or a entity linker to construct a localized graph for every question. Generalizing this step by pretraining the model to do entity linking (Févry et al. , 2020; Sun et al. , 2021; Verga et al. , 2020) might eliminate the need to use an external module