Uncertainty Guided Global Memory Improves Multi-Hop Question Answering
Authors: Alsu Sagirova
Mikhail Burtsev
Venue: EMNLP 2023 Main (2023)
Keywords: Transformer
Memory
Multi-hop Question Answering

Abstract: Transformers have become the gold standard for many natural language processing tasks and, in particular, for multi-hop question answering (MHQA). This task includes processing a long document and reasoning over the multiple parts of it. The landscape of MHQA approaches can be classified into two primary categories. The first group focuses on extracting supporting evidence, thereby constraining the QA model's context to predicted facts. Conversely, the second group relies on the attention mechanism of the long input encoding model to facilitate multi-hop reasoning. However, attention-based token representations lack explicit global contextual information to connect reasoning steps. To address these issues, we propose GEMFormer, a two-stage method that first collects relevant information over the entire document to the memory and then combines it with local context to solve the task. Our experimental results show that fine-tuning a pre-trained model with memory-augmented input, including the most certain global elements, improves the model's performance on three MHQA datasets compared to the baseline. We also found that the global explicit memory contains information from supporting facts required for the correct answer.

Uncertainty Guided Global Memory Improves Multi-Hop Question Answering
Alsu Sagirova Moscow Institute of
Mikhail Burtsev London Institute for
Physics and Technology,
Mathematical Sciences, London, UK mb@limss ac uk
Dolgoprudny, Russia alsu. www.ssaiirovveehystee.e.. edu
Abstract
from long document and then solve the question- answering (QA) task based on the detected evi- dence facts (Nishida et al., 2021, 2019; Tu et al., 2019; Bhargav et al., 2020; Zhao et al., 2023). The resulting performance of such models highly de- pends on the evidence extraction method quality, limiting the QA model context to small num- ber of pre-selected facts. Another group of meth- ods addresses the general long document encoding problem by sparsifying the attention patterns enlarge the maximal input sequence length (Belt- agy et al., 2020; Ainslic et al., 2020; Zaheer et al., 2020). Despite the merits of these models, their attention- based token representations combine lo- cal and global information in the same vector. The high-level contextual features are spread over long sequence which makes it harder access them. To address the described problems, we pro- pose GEMFormer (Global Explicit Memory Trans- former). GEMFormer a method for augmenting the pre-trained language model with memory to store global information relevant to the task. It processes long input concatenated with memory sequence consisting of tokens from input that are important to solve the task. Token importance is defined by the language model uncertainty.
Transformers have become the gold standard for many natural language processing tasks and, in particular, for multi-hop question answer- ing (MHQA). This task includes processing long document and reasoning over the multi- ple parts of it. The landscape of MHQA ap- proaches can be classified into two primary cat- egories. The first group focuses on extracting supporting evidence, thereby constraining the QA model's context to predicted facts. Con- versely, second group relies on the atten- tion mechanism of the long input encoding model to facilitate multi-hop reasoning. How- ever, attention-based token representations lack explicit global contextual information to con- nect reasoning steps. To address these issues, we propose GEMFormer, two-stage method that first collects relevant information over the entire document to the memory and then com- bines it with local context solve the task'. Our experimental results show that fine-tuning a pre-trained model with memory- augmented input, including the most certain global ele- ments, improves the model's performance on three MHQA datasets compared the baseline. We also found that the global explicit mem- ory contains information from supporting facts required for the correct answer.
1 Introduction
Related work
Transformer (Vaswani et al., 2017) and its variants (Lin et al., 2022) have become one of the most pop- ular solutions for various NLP tasks. Particularly, Transformers are applied to solve the multi-hop question answering (MHQA) tasks (Tu et al., 2019; Zemlyanskiy et al., 2021; Khattab al., 2021) that require reasoning over multiple parts of the long document to answer the question.
Augmentation of the neural network model with memory provides additional space store relevant information that can be used to improve model per- formance and reduce computational costs. Early examples of memory-augmented neural network architectures such as RNN and LSTM (Hochreiter and Schmidhuber, 1997) used hidden states in- ternal memory. Graves et al. (2014) and Graves et al. (2016) introduced the external type of mem- ory, where separate network manipulated memory. With the growing popularity of the attention mech- anism, attention was adopted for model-memory interaction (Weston et al., 2015; Sukhbaatar et al.,
The problem of reasoning in MHQA has at- tracted lot of research recently (Mavi et al., 2022). One group of methods focuses on the usage of sub- networks or dedicated modules to extract evidence 'https://githhan http:/ttww..com/looroo////////ea

---

stage: Memory population
2nd stage: Target task training
question
segmenty
question
segmenln
question
GM
segmenty
question
GM
segmenln
RoBERTa
RoBERTa
RoBERTa
RoBERTa
head
LM head
task heads
task heads
Y1
Y2
y3
Global Memory
Figure Global Explicit Memory Transformer. The input segments are processed by RoBERTa+LM head generate prediction distributions for uncertainty estimation. Global memory is populated based the given entropy condition. Then, the memory-augmented segments question-memory-cootee are used to obtain the MHQA predictions y1, 3/2 M/-
2015; Chandar et al., 2016).
memory population conditions for entropy:
A number of recent papers used memory to store global input representations. Gupta and Berant (2020); Zemlyanskiy al. (2021); Wu et al. (2022) employed memory to encode and retrieve the com- pressed view of the long input. Memory (Burtsev and Sapunov, 2020) and Recurrent Memory (Bu- latov et al., 2022) Transformers used memory to store non-local representations of the processed sequence. The interpretable working memory in the Transformer decoder (Sagirova and Burtsev, 2022a,b) presented store contextual infor- mation that was not explicitly stated in the input. Sorokin et al. (2022) summarized representations events from the distant past into memory to im- prove the model predictions. GEMFormer adapts the idea of interpretable memory by storing tokens from the input sequence. This augmentation of memory enriches long inputs with essential global contextual information important for the task.
Highest H arg top & (H)),, E x], Low H [t4 E x H(t)) < A).
(1)
where A is a model uncertainty threshold and & is selected memory size (see details in Section 4), The general motivation behind the entropy- based memory population the following. The model is fed with a question and context, and the en- tropy computed for each contextual token. This entropy is conditional with respect to the question and token-surrounding context. Such entropy value measures how much entropy token has remain- ing if we have already learned the question and the context. In other words, how casily the token can be predicted given the question and the context. Taking into account that document is collection of question-relevan and distractor paragraphs, the entropy of task-relovant token should be lower than the entropy of the irrelevant one.
The GEMFormer architecture is depicted in Figure 1. To fit RoBERTa maximum sequence length limit, the contextual document is split into segments and each segment is concatenated the question. Input processing consists of two stages: 1) document comprehension and memory population, and 2) task predictions generation us- ing memory-enhanced inputs. In the first stage, question-context segments are passed through the RoBERTa model with LM head to obtain cross- dictionary distributions for entropy estimation. Context tokens with little independent semantic content, such special separators, punctuation marks, and stop words, are not considered for the memory population (see details in Appendix B). Global memory then collected from the remain- ing tokens from the document based on the chosen
3 Global Explicit Memory
We implemented GEMFormer with RoBERTa- base (Liu et al., 2019) model as a backbone. Global explicit memory a sequence of document tokens that are important for correct reasoning and an- swer prediction. The model's uncertainty is used to measure the importance of inputs. Given an input sequence = [11, t,...., and vocabulary of size n, we first obtain vec- tor p Softmar (LM RoB RRoBERTalse (x)) of token candidates' probabilities using RoBERTa with lan- guage modeling (LM) head. Then we calculate the entropy H = - - Σ log for each in- put position. In our experiments, we tested two
fmm]
of m tokens
[MM_RoBBBBBB

---

Model
QA 0 = 0.3 soid
2WikMMHQA, 0 = 0.45
MuSiQue, 0 = 0.2
Ans
Supp=
Ans=ata Supp
Supp+
Task-tuned RoBERTa
73.8 1.0
82.861 0.09
63. 161 0.37
0,37
95. 09. 0.02
31 16.66.
62.6 0.25
YAKE! keywords memory
72.9 1.03
61.444 01 0:33
63.98 1.01
63.13 0:27
30.30 na
63.45: 0.64
GEMFormer Highest H
73.87 0:26
82.61 +
62.85 0.51
14.88:oa. 64
94.4 = 0.06
29.18 2.50|
59.77
GEMFormer Low (H < 0)
75.13+ 0.5
83.8
= 0.43
64.77 0.66
67.14 0.46
95.67 = 0.36
31,56+ 041
63.85 0.71
GEMFormer Low (H < %%)
74.19 027
83.13
63.59 0.14
66.08
95.15
32.22 :: 0.57
62.544
Table 1:
GEMFormer with Low conditions outperforms the baselines. Table shows answer, supporting evidence, and joint Fl scores with standard deviations on dev sets (average over 3 runs) and entropy thresholds.
entropy conditions (Eq. 1). In the second stage, question and global memory tokens are concate- nated with each segment for the MHQA task train- ing. The model's weights for the first stage are updated every epoch with the weights of the model trained to solve the target task in the second stage. three English MHQA datasets: HotporQA distractor setting (Yang et al., 2018), 2WikiMulliHopQAA (Ho et al., 2020) and MuSiQue-Ans (Trivedi al., 2022). Fur- ther in the paper, we will refer to them as HP, 2W, and MSQ respectively. Data preprocessing. training, and inference details are described in Ap- pendix B.
the pre-trained model associates newly appeared tokens (<t>, </t>, [/sent] tokens) with a no- tably high entropy. Moreover, comparison of rare tokens entropy distributions to the overall context entropy distributions (see Table 8 in Appendix D) confirmed that rare tokens tend to have higher en- tropy and are infrequent in global memory. During fine-tuning, the entropy of rare tokens related the question becomes lower compared irrelevant ones. This allows the model to preferentially store in the memory more relevant entities. Besides, the model with a global memory filled with top-200 highest entropy tokens (Highest H in Equation 1 and Table 1) produced no improvement over the baselines.
We evaluated GEMFormer
4 Results and Discussion
We hypothesize that a model combining tuned RoBERTa encoder with a frozen LM head for un- certainty estimation tends detect parts of text that are essential for the answer by reducing their uncertainty (see also Appendix B). To test this hypothesis, we used a variable-size memory con- sisting of tokens with entropy lower than a given threshold (Low II in Equation 1). The fine-tuned model with constant entropy threshold ê outper- formed the highest performing baseline memory- free model by 1.6 joint Fi points for HP, 1.59 and 0.58 points for 2W answer and supporting evidence prediction, and 1.25 Fl points for MSQ supporting paragraphs prediction. We also tested a dynamical entropy threshold as a value of the fifth percentile of the document token entropy assuming that this might help to better cover the full supporting evi- dence (Low (H < 5th%) in Table 1). Such mem- ory selection rule showed the best answer Fl for MSQ and led to a slight improvement compared to the task-tuned RoBERTa baseline but was weaker than fixed for HP and 2W.
As baselines in our experiments we used RoBERTa fine-tuned on the task without memory and RoBERTa fine-tuned with memory bank of 200 YAKE! (Campos al., 2020) keyword tokens from the contextual document (see Table 1). We started memory experiments from an assumption that tokens for which a language model is the most uncertain might be the most useful in the global context. However, the distribution of entropy of a task-tuned baseline model over the document tokens (see Appendix E Fig. 5) showed that the majority of the context tokens have high entropy. and locations of low entropy tokens are closely aligned with answers and supporting facts within the document. This observation points to the hy- pothesis that low entropy tokens might be helpful for generation because they overlap with the con- text with information relevant to the answer. Also, the global memory of low entropy tokens tends to guide the answer model to focus on such tokens. To ensure the correctness of this observation, we examined if the low entropy tokens are not rare en- tities for which the model has less amount informa- tion about. Figure 6 in Appendix E shows that the
We also evaluated the 2hatGPT2 (gpt-3.5- turbo- 00-3 checkpoint) with memory generated
hhttps///penna..o///ll///httttt

---

by GEMFormer Low (H 0). The results are presented in Table 2. We tested the question-only inputs to assess the general QA ability of the model and the two variants of input document represen- tation (the ChatGPT prompts for our experiments are listed in Appendix C). In the first case, doc-
or general-purpose LM can be used for that as well. We tuned the model on stage 2 but froze the pre- trained RoBERTa for memory uncertainty estima- tion. This led to the joint FI points decrease. Thus, updating the model for uncertainty estima- tion is critical to make memory useful. Indeed, we
GEMFormer
Ang Fl Letd Supp Fl Estd
Joint iosd
Input
QA Ans Sup Jnt
2WikiMHQA
MuSiQue Ans | Sup
Ans Sup
Low (H < 0.3)
73.13
83.8 = 043
64.77= 666
No Q Doc only
74.33
83.671 035
64.09 0.28
Q
19.89|
15.46]
4.98|
No fine-tune
73.88=0.2
80.77 0.12
61.72+ 034
Q+R
24.35/47.53
25.36|42.65
13.08 40.559
Random memory
72.92=021
81.76=01 0.1
61.58 025
Q+M+R
23.97|43.5
|14.44
26.08/41.7
13.05|38.04
Q+C
20,64
111.7
43.22
30.51 33.6
Table 3: Ablation study on HotpotQA dataset. No Q/ Doc only means excluding question from the memory population stage. No fine-tune uses a frozen pre-trained checkpoint for memory population and tuned with such memory on MHQA task. Random memory means memory of tokens randomly chosen from the document.
Q+M+C
57.34 20.9
112.94
33.97
35.98 (40.33
Table 2: Global explicit memory improves ChatGPT performance. Fl scores for dev sets of HotpotQA, Musique and 2500 dev samples subset of 2WikiMHQA. Q denotes question in model input, R - retrieved context, M memory, C full context document.
measured average per token entropy during train- ing of the GEMFormer Low (II < 0.3) model and found the growing difference in uncertainty between evidence and distractor facts (Fig. 2). The Random memory ablation was to fill memory with tokens selected randomly from the document. It showed that global memory with the arbitrary- selected content can not only fail to improve pre- dictions but can actually degrade the model perfor- mance. We also provide the comparison of our best-
ument sentences were stored in a vector database and ranked by relevance to the question via the basument-based QA pipeline from the Langchain library. The retrieved most relevant sentences were concatenated to the question and used for answer prediction. As a result, the ChatGPT performance was highly dependent on the effectiveness of the evidence retrieval process. If the retriever failed to accurately extract the evidence, the subsequent memory augmentation did not rectify this short- coming. In the second case, fed model with concatenation of question and contextual doc- ument with and without global memory augmen- tation. Although the answer Fl scores were sig- nificantly improved compared to the retrieval set- ting, they still fell short of the fine-tuned RoBERTa scores. However, in this setting, we observed no- table improvements in answer F1 scores (+6 F1 for HP, +7.94 Fl for 2W, and +5.47 Fl for MSQ), signifying the efficacy of memory augmentation in enhancing answer generation.
Answer
aweroee entooy
Supporting facts
Distractar facts
0.40
0.35
0.30
spach
Figure 2: Difference in uncertainty between distrac- tor and supporting facts grows with training. The plot shows the average per token entropy with standard deviation for answer spans, supporting evidence, and distractor facts on the HotporQA validation set during training of the GEMFormer Low (H < 0.3) model.
Ablation study To verify the proposed memory augmentation, we conducted an ablation study on HP. Results are shown in Table 3. To test the im- portance of the question input for memory filling, we trained No Q/Doc only model with memory selection rule H < 0.3 and stage 1 entropies cal- culated from the document-only context. The re- sulting joint F'l degraded by 0.68 points indicating how question affects the memory quality. With No fine-tune configuration we checked if the second stage MHQA task tuning affects memory content
performing GEMFormer Low (H < 0) model with existing base-sized MHQA models in Appendix A.
Memory analysis The results shown above demonstrate that global memory of supporting facts improves MHQA performance, and it is natural expect that a larger memory size should give bet- ter results. We analyzed the dev set predictions of GEMFormer Low (H < 0.3) trained on HP with three different random seeds (Table 4) and found

---

that instances, where the model utilized larger aver- age memory sizes, corresponded to higher joint Fl scores, thereby reinforcing our hypothesis. Specif- ically, a five-fold expansion in memory is linked to an increase of 1 1.32 points in the joint F1 score. Notably, runs with larger memory sizes indicate a higher coverage of evidence, albeit it does not exceed 30% The average size of supporting facts
scores (Fig, 4a). And the amount of evidence stored in memory has almost no effect for low coverage (up to 30%) and leads to the answer FI decrease for higher coverage values (Fig. 4b).
F2 85, $5.39
0.8
F1
BA.06
0.8
$4.07
0.8
6.5-
GEMFormer (H < 0.3)
Avg mem size
% of tokens from supporting facts stored in memory=ad ann
8 0..4
F1 65.19
0.3-
F1 64.26
F1 64.07
joint
1.6
0.0
65.39
50+ 36
30+ 13
31+
30+ 13
fraction
fraction
64.86
24 14
21 10
214 10
21 to
(a)
(b)
64.07
1119
10
1128
10
Figure 4: Higher occupation of memory with evi- dence tokens results in better scores. Panels show dependencies of answer Fl on (a) portion of memory tokens related to supporting evidence w.r.t. memory size, (b) portion of memory tokens related to supporting idence w.r.t. supporting evidence length for three runs of GEMFormer Low (H 0.3) HotpotQA with varying joint Fl scores.
Table 4: Larger average memory size and evidence coverage have higher model performance scores. The table shows three HotpotQA runs with average memory size and the percentage of tokens from supporting facts w.r.t. supporting facts length stored in memory in total, and for samples with correct (+) and wrong (-) answers.
is 83 tokens which means that only about half of the memory is occupied by useful information with the rest of the memory slots filled with tokens from distractor paragraphs. This occupation is the same for correct and wrong answer predictions.
Conclusions
In this study, we demonstrated how utilizing uncertainty-based global explicit memory can en- hance the model performance on MHQA tasks. Our findings indicate that utilizing low entropy context tokens can aid the model in MHQA reason- ing, but only when the entropy estimation model is specifically fine-tuned to the target task. Exper- iments show that higher- higher-performing models use larger memory sizes with better coverage of sup- porting facts.
Counterintuitively, samples with larger memory sizes for the same model have lower answer pre- diction quality (Fig. 3). This could potentially be attributed to the distracting influence of irrelevant tokens stored within the memory. The larger mem-
0.8
0.6
Limitations
65.39
0.2
64.86
64.07
There are several limitations this work. First, the global explicit memory augmentation of the input sequence may increase the training time by shorten- ing the context chunk lengths. Second, the current implementation of memory token selection results in storing significant fraction of irrelevant tokens which interferes with the calculation of correct pre- dictions. We will work methods to improve the relevance of information stored in memory.
0.0
200
memory size
Figure 3: Larger than average memory sizes have lower scores. The answer Fl moving average per memory length (solid lines) and the overall answer F1 (dashed lines) for three runs of GEMForrer Low (H < 0.3) on HotpotQA with different joint Fl.
ory size should have more unrelated tokens so the training should optimize the trade-off between en- larging memory for better evidence coverage and decreasing it remove noise. This is supported by the results of the deeper analysis in Fig. 4. Samples with the memory containing more tokens from evi- dence and less irrelevant ones tend to have better
Ethics Statement
This work is a fundamental research work that fo- cuses on technical improvement, thus we have not applied additional filtering techniques to the textual data we used, beyond what has been performed on

---

the original datasets. The textual data we used may have information naming or uniquely identifying individual people or offensive content that we have not been able to identify, as those are out of the focus of this work.
Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Pupers), pages 845-855, Melbourne, Australia. Association for Computational Linguisties.
Ruiliu Fu, Han Wang. Xuejun Zhang, Jun Zhou, and Yonghong Yan. 2021. Decomposing complex ques- tions makes multi-hop QA easier and more inter- pretable. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pages 169-180, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Acknowledgements
AS was supported by a grant for research cen- ters in the field of artificial intelligence, pro- vided by the Analytical Center for the Govern- ment of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000073032115000000 and the agreement with the Moscow Institute of Physies and Technology dated November 1, 2021 No. 70-2021-00138. Authors are thankful to SberDevices for granting access to additional computational resources.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014.
Neural turing machines.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adria Puigdomènech Badia, Karl Moritz Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Ko- ray Kavukcuoglu, and Demis Hassabis. 2016. Hybrid computing using a neural network with dynamic ex- ternal memory. Nature, 538(77666)441-4776
Hermann,
References
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Va- clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 268-284, Online. Association for Computational I Linguistics.
Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabharwal. 2020. A simple yet strong pipeline for HotpotQA. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pages 8839-8845, Online. Association for Computational Linguistics.
Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. ArXiv, abs/2006.03274.
Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. a5X5v:200.05150.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
G Shrivatsa Bhargav, Michael Glass, Dinesh Garg, Shirish Shevade, Saswati Dana, Dinesh Khandelwal, L Venkata Subramaniam, and Alfio Gliozzo. 2020. Translucent answer predictions in multi-hop reading comprehension. Proceedings of AAAI Conference on Artificial Intelligence, 34005):7700-7707.
Sepp Hochreiter and Jüüren Schmidhuber. 1997. Long short-term memory. Neural Computation, 978) 1780.
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. 2022. Recurrent memory transformer. In NeurIPS 2022.
Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Balcen: Robust multi--hop reasoning at scale via condensed retrieval. CoRR, abs/2 101466.36.
Mikhail S. Burtsev and Grigory V. Sapunov. 2020. Memory transformer. ArXiv, Abs/2006.11122.
Tianyang Lin, Yuxin Wang. Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. Al Open, 3:111-132.
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020. Yake! keyword extraction from single documents using multiple local features. Information Sciences, 509:257-289.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, a1691.
Sarath Chandar. Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Bengio. 2016. Hierarchical memory networks.
Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2022. A survey on multi-hop question answering and gen- eration. ArXiv, abs/2204.091400
Christopher Clark and Matt Gardner. 2018 Simple and effective parti-pp reading comprehension. In

---

Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. 2019. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction, In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 2335-2345, Florence, Italy. Association for Computational Linguistics.
pages 1655-1665, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Jason Weston, Sumit Chopra, and Antoine Bordes. 2015.
Memory networks.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Al- borz Geramifard, and Zhou Yu. 2022. Memformer: A memory- auggented transformer for sequence mod- eling. In Findings of the Association for Computa- tional Linguistics: AACI-IUCNT 2022, pages 308- 318, Online only, Association for Computational Lin- guistics.
Kosuke Nishida, Kyosuke Nishida, Itsumi Saito, and Sen Yoshida. 2021. Towards interpretable and reli- able reading comprehension: A pipeline model with unanswerability prediction. CoRR, abs/211 11 1.09029.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6140- 61550, Florence, Italy. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages Brussels, Belgium. Association for Com- putational Linguistics.
Alsu Sagirova and Mikhail Burtsev. 2022a. Complexity of symbolic representation in working memory of transformer correlates with the complexity of task. Cognitive Systems Research, 75:16-24, 24.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Alsu Sagirova and Mikhail Burtsev. 2022b. Extend- ing transformer decoder with working memory for sequence to sequence tasks. In Advances in Neu- ral Computation, Machine Learning, and Cognitive Research V, pages 253-260, Cham. Springer Interna- tional Publishing.
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, llya Eckstein, and Fei Sha. 2021. Read- Twice: Reading very large documents with memories. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5189-5195, Online, Association for Computa- tional Linguistics.
Artyom Sorokin, Nazar Buzun, Leonid Pugachev, and Mikhail Burtsev. 2022. Explain my surprise: Learn- ing efficient long-term memory by predicting uncer- tain outcomes.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks.
Wenting Zhao, Justin T. Chiu, Claire Cardie, and Alexander M. Rush. 2023. Hop, union, generate: Explainable multi-hop reasoning without rationale supervision.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics.
A Comparison with other MHQA methods
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. 2019. Select, an- swer and explain: Interpretable multi-hop reading comprehension over multiple documents. CoRR, 0bs/1911.00444.
We also compared our best-performing GEM- Former Low (H < Ô) model with existing base- sized MHQA models in Table 5, The proposed model shows the best results in MSQ answer pre- diction and 2W supporting evidence prediction. On HotpotQA our method achieves answer Fl compa- rable to ETC, BigBird, and ReadTwice and joint F1 higher than Longformer with the smaller-sized model.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Proceedings of the 31t International Conference on Neural Information Processing Sys- tems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
B Datasets and Training Details
Siyuan Wang, Zhongyu Wei, Zhihao Fan, Qi Zhang. and Xuanjing Huang. 2022. Locate then ask: In- terpretable stepwise reasoning for multi-hop ques- tion answering. In Proceedings of the 29th Inter- national Conference on Computational Linguistics,
All datasets' contexts are based on Wikipedia. 2WikiMultHHopA also has additional evidence re- lations tuples that were not used in our experiments.

---

Model
HotpotQA
2WHLMMHQAA MuSiQue
rest are cross-entropy losses for the classification of question types, supporting paragraphs, and ev- idence sentences. Weightenings a, were used to keep each loss term in a similar range". For Hot- potQA we used 01 = 10, 02,3,4 = 1 to reproduce the RoBERTa baseline scores from the Longformer paper. We also tested on of 1, and 5 because of the lower values of the question type loss compared to the other loss terms during training but on 10 showed the best results. For 2WikiMultiHoPOA we did not observe the significant differences be- tween loss terms values and used unit values for all alphas.
Ams
Sup
Jut
Ans
| Sup
Ans Sup
GEMFormer (H < 8)
125M
75.13
164.77
67.14/95.67
63.56
IRC (2021)
72.9 1798
"DEGN (2019)
110M
69.23
38.49/57.79
Step&casoner (2022)
110M
73.03/91.21
RAG Small (2023)
62.8
HUG Small (2023)
140M
66.8
[67.1
25.1
SAE (2019)
110M
74.81
(85.27
66.45
Longformer-bau buse (2020)
149M
661.4
ETC (2020)
75.1
BigBird-ITC (2020)
132M
75.7
186.8
167.7
Big (2020)
132M
75.5
[87.1
167.8
ReadTwice (2021)
140M
75.9
Table 5: Comparison with existing methods. Related works' Fl scores taken from the corresponding pa- pers. The mark means scores taken from Fu et al. (2021).
MuSiQue preprocessing and training were car- ried out following the Trivedi et al. (2022) End2End model setting. The dataset has no yes/no type of questions, only document spans. The ev- idence is presented with golden supporting para- graphs. So we added a paragraph start indicator token to the RoBERTa vocabulary for supporting evidence prediction. It was randomly initialized be- fore fine-tuning. The answer span targets were se- lected as the last occurrence of the span in the sup- porting paragraphs. The multi-task training objec- tive was sum of an answer span cross-entropy loss E'ppa and supporting paragraphs binary cross- entropy loss Eparai
The number of context paragraphs, the number of reasoning hops, sources, and sizes of train and dev sets for each dataset used in our experiments are presented in Table 6.
Dataset
para
Hops
Train
Dev
HotpoiQA
10
90447
7405
2WikiMuliiHopQA*
10
2,4
167454
12576
MuSiQue-Ans
20
24
19938
2417
Table 6: MHQA datasets statistics.
Preprocessing and Objective HotpotQA and 2WikiMuliiHopQA have questions with yes/no answers and context-span answers. Also, both datasets have golden targets for supporting evi- dence sentences and paragraphs. We prepared Hot- potQA and 2WikiMultiHopQA inputs and targets following the Longformer (Beltagy et al., 2020) MHQA approach. To prepare the input sequence, we added special tokens to indicate the question start and end, paragraph title start and end, and sentence end. The special tokens were added to the RoBERTa vocabulary and randomly initial- ized before fine-tuning. The training was carried out in a multi-task way to predict question types (yes/no//saan), answer spans, relevant paragraphs, and evidence sentences jointly. We used the fol- lowing loss function proposed by the Longformer paper authors:
L = C + BCE para.
(3)
Training and Hyperparameters For all datasets, the long input sequence was split into segments to be processed by the pre-trained RoBERTa model with 512 tokens input length. To avoid partitioning answer spans, we applied 20 token length overlaps between consecutive input chunks. The maximum memory size was limited to 200 tokens to balance the lengths of memory and context in the model input. On the memory population stage, before we collected the information from the context the memory, we filtered out special tokens related to the supporting evidence prediction, all possi- ble punctuation-relater tokens, and stop words? to reduce the amount of noninformative memory content.
To start training used the public RoBERTa- base checkpoint (125M parameters) and the Hug-
L= on Cagtueee + Orzor CFppn
0g agpaaa az Esents
? 1432Lssueccmment-773888222 Saakeddduusstttnns--cseeeeeerooo
where or or_CFF Espon is the noisy labels handling loss function (Clark and Gardner, 2018) to account for all possible occurrences of answer span and the
ioo)
o- remmve-sttooord---tthhrtttt http://ggaaaaccoooorrrrrrttaaaaa

---

Parameter
HotpotQA ZWikiMuliHHpQQA MuSiQue
distributions with the task-tuned RoBERTa encoder followed by pre-trained LM head. By this, we combined the reasoning skills of the tuned encoder and the language structure knowledge the LM head. The RoBERTa pre-training data contains Wikipedia corpus (Liu et al., 2019) and the datasets we used are also based on Wikipedia. This assures that pre-trained RoBERTa LM head weights possess information about the language distribution of the context documents in the training set. The frozen LM head is able map the output representations while preserving the uncertainty of predictions. As the encoder is progressively fine-tuned for MHQA task, its output representations become more certain for tokens related to the answer and supporting facts. During our primary experiments, also tested the LM head jointly fine-tuned with the encoder model. It led the same results as a baseline and did not yield any significant differences in uncertainty values for answer and supporting facts compared to other parts of the text. This observation can be attributed to the inherent nature of the LM task, which aims to predict all tokens in a sequence without any specific focus the MHQA reasoning.
Batch size
12
Epochs
Optimizer
AdamW
Adam 10% train steps
Adam
Warmup
1000
1000
Learning rate
3e-5
3c-5
2c-5
Compute time
7h
40min
Table 7: Training hyperparameters for all MHQA datasets
gingface Transformers RoBERTa model implemen- tation?. The HotpotQA and 2WikiMullHHoQAA training and evaluation consisted of four subtasks: question type prediction using the question clas- sification head over the first [CLS] token, answer span prediction, and evidence paragraphs and sen- tences prediction. The MuSiQue training and eval- uation consisted of two subtasks: answer span pre- diction and supporting paragraphs prediction. To predict supporting evidence, used two-layer feedforward networks with GeLU activation be- tween layers applied over the paragraph title end and sentence end tokens for HotpotQA and 2Wiki- MultiHopQA and over the paragraph start tokens for MuSiQue. For the fixed entropy threshold- based memory experiments, we tested number of threshold values and reported results with the best Ô choices to detect supporting evidence-related tokens.
the encoder
C ChatGPT evaluation prompts
During inference, the given model checkpoint and pre-trained LM head were used generate a global memory. To compute evaluation metrics, we collected predictions across document segments and took the most probable answer as the final pre- diction. For HotpotQA we ensured that predicted evidence came from the two most probable para- graphs according to the dataset setup (Groeneveld al., 2020).
The prompt for experiments with the retrieved con- text was the following: System: You are a world-class algorithm to answer questions in specific format. Human: Answer the question using the following context <context>. Question: <question>. Tips: Make sure to answer in the correct format.
To combine the question and the full contextual document in the model input (Question+ Contexi in Table 2) we used the following prompt: System: You are a world-class algorithm to answer questions based on the given text. Store
Training hyperparameters are listed in Table 7. All models were trained on NVIDIA 100 GPUs.
The linear decay schedule without a warmup, with 1000 steps and with 10% of training steps linear warmup and constant schedule with 1000 steps warmup were tested. Linear warmup with linear decay scheduler that performed best all datasets used in experiments. We also tested learn- ing rates of 2e-5, 3e-5, and 5e-5 and epochs of 3, and for hyperparameter search.
the answer and the supporting evidence from the text in the following structure: { a'aswew':: 'answer string', "supporting evidence': ['list of supporting evidence sentences' Human: Text: <context> Question: <question>.
Finally, the memory-augmented input (Ques- tion+ Memory+Coootnn in Table 2) prompt was the following: System: You are a world-class algorithm to answer questions based on the given text and memory.
Uncertainty Estimation Procedure GEM- Former stage I includes predicting token

---

Use the provided memory to detect question-related information in text. Store the answer and the supporting evidence from the text in the following structure: ('answer': 'answer string', ['list of
"supporting evidence': supporting evidence sentences' ]]. Human: Memory: <mem> Text: <context> Question: <question>.
D Rare tokens entropy
We collected rare (occurring in a context less than 5 times) tokens from each contextual document of the validation set for each dataset compare the characteristics of the entropy distributions of rare tokens to the overall context entropy distributions. The descriptive statisties averaged over the valida- tion set samples are presented in Table As result, we can see that the mean, median, mode and maximum values of rare tokens distributions match overall context distributions up to two deci- mal places for all three datasets. Also, both distri- bution types have skewness coefficients lower than for each dataset, which means the distributions are skewed the left. This observation confirms that rare tokens most frequently have high entropy values. We also calculated how many global mem- ory tokens are rare: for HotpotQA 61 21% of memory tokens are rare ones, for 2WikiMHQA 9 9 9%, and for MuSiQue - 16 + 15%. These ob- servations imply that, in practice, rare tokens tend to have high entropy relative to the overall context entropy, and for two out of three datasets rare tities are infrequent in global memory. The rare tokens analysis suggests that during fine-tuning, the entropy of question-reet related rare tokens decreases compared to the entropy of irrelevant tokens.
E Entropy over document distribution heatmaps
this section, we listed per token entropy heatmaps for one validation context example. Heatmaps illustrate that the pre-trained model's uncertainty is almost uniformly low except for newly-added tokens (see Fig. 6). Looking at the entropy difference after the first epoch and before fine-tuning, we can see how the uncertainty of title and sentence indicator tokens decreases. and the supporting facts have tokens with unchanged uncer- tainty, while the entropy the rest of the document increases (see Fig. 7).

---

Dataset
Tokens
Min
Mean
Median
Modc
Max
Skewness
Kurtosis
HoipotQA
Rare
0.043 0.400
0.465 0.34
0.487 0.039
0.495 noss
0.621 0017
-1.
2.42+ 2066 2.51+ -13
Overall
0.017 0.021
0.460 owe
0.487 0042
0.489 vor
0.625 o016
1722
2WikiMHQA
Rare
0,101 a: 0006
0,503 a02s
0,530 0,066
0,556 0.03
0.628 0015
2.73 =1447 2.58
Overall
0.081 + 0.029
0,503 0.125
0.529 0,27
0,553+ 0.045
0.635 0016
-1.42+ 0.5
2.51+ 2.40
MuSiQue
Rare 0.006 0.012
0.410 0.012
0,430 0,012
0.450.: 0.013
0.586 0015
-15
3.33 1.9
Overall
0.003 0.006
0.409 0.014
0,431 0.033
0.452+ 0013
0.590 0015
-1.59+
3.37+ 1.26
Table 8: Descriptive statistics for all context tokens' distribution and rare tokens distribution. The values are averaged over the validation set samples for HotpotQA, 2WikiMHQA, and MuSiQue datasets. Skewness is calculated with Peaherr-e coefficient and for kurtosis calculation the Fisher's definition was used.
.t-Ed E/
1994
[sent]
MIS MiS-- FAlEs WOLL re aaroonshi WITH actor sola Logosi, played by Martin Landuu [/eent] 52
rah Je31ic Parker, Patricia ArquOTto, Jaffeey Jones, Lisa forio, and HILL Purray are
ENe supporsing cast
EnE KEESSCOEE CErrraCSoo////SSSC
berricksOn (born July 15, 1966 IE S AmBrican dLractor, Scresnwriter and produce [/sent] He Lives in Los Angeles, California [/sent] He is best known for OLrecting horror filmo such as "Siniste
E", "The Exorcisn of Enity Rose and Deliver UE From EvEl aS will as the 2016 Farvel CIoeaaTI Universe Instal
FOnE, "Boctor Strange. 7750n SE-R00d50n,
IS 3 COnSUS designated piace (COP) in POLaski County,
Arkansas, in the United States (/Sent) Its populssiion WOS 183 SE the 2010 consus [/Sent) IT 15 part Of the Little
Nck-North Little CorNay Netropoliign Lton Statastical Area. I/sent] Woodson and 155 accomsanyinn hoosson Lake and VO
HollOw
PORDSORO for Wood ST., proniOmnt Plantation Owner, trader, and bust BusToessemn
The turn of
contury, [/SOGE) Woodson adiscont The Plantation,
Targest The plantations DUT
Nood
[758nt] KEXTyLBr SSTGE//E2yyLer BSESS (B07S JUnO 5, 1955) 15 30 Aserican musiciss, RUZIC prosicer, and conposor
film. Television. video games 1/sent Narshall Punch, Friedkin.
his [[/eet|
horror gences,
directors
[/Sent] hignest
Films:
Guardians the Galary" the
2014, and ItS 2017
[/sent] asoltion, he
the Amer
Karllyn Kanson, and produced ITS albunt The Pale Emperor" and "Reaven Ups 18 Down [/sent IeEaEd Nond E3-
T Davis wood 3r. (UCTober 18 H/seen Upside BOMI [/ent MEXEd WOODK/ Ensoward LIVIS Wood Jr. (dcrober 16, 1924
secenser 15, 1975) wOs American FALrEaKA actor, WrIter producer, and ciractor, (Sentt KENOOTI fver Us free Evil
12014 film)6/1500l lver UA 1rom EVEL IS 2014 American supernatural horror film directed by Scott Derrickson and pr
adlied by Jarny Braek KheiEee [/sent] The Mil IS afficially baved on 7001 non TETTION book anl Eee "leeatre The Ni
RhE by Ra128 Sarcnie an0 Li5a COLLEET COOL, and 1t5 Bornoting Eappaion hig Lighes that IE WOS inspired by actusl
3.0
accounts . [/sent] The filn Stars EriC Bana. Edgar Rarires, Sean Norris, D Olivia Nunn, and Joel Mchale in the rain ro
Les and was released on July 2. 2014 [/sent KTEAAO CLLLSS//SGN Collis 15 an Aperican Filmaker and actor, 1/se
Angoles Troe [75ent]] Ho also Studies Cinea at University Southorn California From 1991 TO 1997, [/56n]] COLLEs first vo was The 3isistant director for The SCOEE BerFIcASOn'S short 'LOVE The Ruins" (1995) [Z5ESt] 1998, be playo "Crankshaft" in Koyanagl's
No attended The Buke University From 1965
and The university Califoria,
0.8
Percent" [/sent kessinister Filnns/tssSiiister IS stars directed Scolt Berrickson Wiitlen Derrickson
supernatural
Cargill,
(exconns Brookse/
Heddsyyc Janvary 1991 BolEiFors, Varyiar Brooks Colifornis 1948
[Zsent] his start In novins Ropaar In EJ Wood TILMS such 7 Trom Ontar Space", Melen Glandoy, and "Mll 5
1Ol HoLLywoo.
I/sent
LT. [/SAnE] P0 1OOK break Trom acting durang ENe 1969 and 1976S but due The angping Interest The FILES of
R00d, he resserged in the 1988 and has DECOmD 3. PrOLIFIC aCTOr, [/SenE] PE aLEo has FInce gone
Write, proc
uce and direct several Files, //eent Det-botor Strange (2016 filn)=/tslnctor Strange IS 2016 Arerican superheroe//
: directt several Tiles. [/sent /centtccsslottto sesloctor Strange (2010 (inis/Lbbictrr Strange 15 à. 2016 American Superhera fi1m bas
ed on the Norvel COFLES character of the zane nane, prosuced by Marvel StusLos and distributed by KOLE Disney Studi
25 Notion Pictures (/Sent) It 15 the nourtesnt FiLm of the Narvel Cineratic Universe (AKU), (ZSent) The film WO5 di
rocted by Scott Derrickson, who wrote It with Jon Spathts and C. Robert Cargill, and stars Denedict Curberastch 55
Stophim Strange, atong CRiNITo! Elrofor, Rackol MEAGUES, Binedier King, Michaol Stuntbarg, Bonjariin Bratt,
Adkins, Hads Sen, and 1i1da SWInEon, [/SBnE "bactor Strange" Surgpon Strange Teares The RyStic arTS
Figure 5: Supporting facts appear in the areas of the low entropy of the context. The fine-tuned RoBERTa bascline entropy distribution heatmap. The presented text is a sample context from the HotpotQA validation set with the following supporting evidence: "Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer.". " "Edward Davis Wood Jr. (October 10, 1924- December 10, 1978) was American filmmaker, actor, writer, producer, and director."

---

<Ea:d (file I//aed Wood 1964
period conedy-dras Fila di directed produced Ti
and
Jainny Bopo cult Filemaker Ed Wood. [/ann]] The Film conceros USe paried in Aund life when
he rade his best known film vell his relationship with actor Bela Ligesi, played by Martin Landry, [sentt 5t
rah lessice Parcer, Patricia Arquette, leffrey Jones, 1ise Narte. and Bill Murray are srong The supporting ing cast 1/9
ent : derFAotsonK/eeSS Exscort berrickson (born July 16, 19651 15 an Aserican director, sereener-tor and produce
1/500: ho Lives In Los Angolss, Lortorri [/Eont] KS 15 bost krown for directiog iog Maror Filns such 55 Sinisto
The Expreise of Erily Rose", and 'ellvvr Us From Evil", 25 well a5 the 2016 Narvel Cieentic Universe instal
sunt, "boctor Strangw. [/Amt]- -LWodd Arkam wtt.a////////oeeee EMoodoon is cumsuss www.- Led place (C5P) Poleski County,
Arkansas, the United States, [/sent] Ifs popul in was 403 st The 2010 consis, 1/ [/Sent IT 15 part The
ock: Littlo Corsy Notropo [ian Statistical Area. [/eent Moodson and ITS accsepany Wordzon Lake aid ho prominent po owner, trader, and Bustressean
Hollow are nanesake
century. [/amt]
Largust
Bsteek/ eetk//e bates Ibom
1965)
prosucer, with fills Filn
Ta
television,
Punch, Much his
the der, Not Zonbie, Nell Porshall, Mil Villinn Friedkin, Scott berricioon and Janes Cunnn (Zenn) WIth Guin, Te has scores
"John '[/sett He colleborrt with dlrecters
Zact
Every one of the director's : Tilns; inc uding "cuarcians the to talay",, which bocano one of the highest grossing de
1e31l1 novies of 2011, and 1ts 2017 sequel. (ScenE to co0ltion, lis 15 0.50 the Lesd pustorist of the Ans rica rock
Band Parllyn Newson, and produced IES alburs "The File Engerar" and "heaver lips lde live 1 [heni RESEd Woode/ Earthol
Davis Mood Jr. (October 10, H8/v/e Upside Down [ZAent] NESEd Woody/ Roody/E-tiaar Eatinard BavIs Mood Jr. (OLtober 36, 1124
Decerber 10, 1978) vas an Aserican filuwaher, actor, writer, producer, and director. [/ent St Liveiver Us from tvil
(2014 Tiln/e/Es0n Tver US froe fyll In 2014 Averican supernatural barror FILM directed by SCotE Berrickson and pr oduced by Jerry Bruckheiner, [/5ent] The 1iln IS officialy based DE 2001 nOR Fiction book entitled "ewa'e the T. DIE by Raton Sarohic Lisa Cottiei COO1 and I6 BarActing carpase DEGhERITG EIat IT
Lispared actual
1.0
accounts [/Sent] 1ilh stars Bana, Edgar Parirez,
Horris, OLivin Hunn, Jost Motale the nain
and WaS COORE un July
Toent SERaan COLLES/EEA COLLES Aeer Filrmaker and actor 1990 for the Scott Derrickson's short "Love
Ehe
Califoriis Ruins 11951
0.0
the
1998
"cranEshaTe" EFIC Koyanagl 5 RRuddred Porcent //sent KEssmister (TILE) R/DSSIIISt
2011 matural h
Fror Tilm directed by Scolt Berrickson and wrillen Derricksen
C. MOcerT Gagill [/Set IE stars Ethan Baoke Fictional LEUR-EE weiter ILLISON Dswalt who eiscovers a TOX OF home nOVIES In hIS DELIC That pots hIS 1ar
in danger, [ssent] stacconrdd Trookss/tssCoreee Brooks Ibom Conrac Biedreyyck on January 3, 1931 in Baltinore. Marylar
() IS on Mericnn actor, [/sent] be Rovel To Hollwwed, Califorria In 1966 To pursue a carner In acting [/sent] he
DE 115 Start in Tovies appearing in ES hood FALNS such 35 "Plar 3 Tron duter Space Meen OF bienda and 2011 83
IE. [/5ent] He took Break 1700 acting durang the 19065 and 17705 but due to the ongoing interest in the Tilms of
al3o has STnce gune on To wrile, pro ure and diract several T1.0. [/sent REXOccor Strarge 12016 71 Strange
kd Vond.
reseerged in lhe 1995 and has become prolific actor [/sent
2015 Aserican supar harooar /
direct several 111.6. 1/t.1///6/]]/00
Strange (2816 (1tn)=/Ex0cC/ Strange à 2015 Amarican superTero Stuclos and distri by
the Marvel Contca character
some nene, produced
DS Pictures. [
the Tourteenth Chivetel Eilafor, Machel McAdSEs, Eenerict Wons. [/sent] "Doctor Straage" surrgeon
the Narvel Cinenat:ic Universe [/sent Robert Cargil]; Stuhlbarg,
Steahan
Bratt,
Adh11s, Rads Mikkelsen Tilda
Figure 6: The pre-trained model has uniformly low uncertainty about context except for sentence and paragraph marker tokens, added to the vocabulary before fine-tuning, The pre-trained RoBERTa entropy heatmap of an example from the HotpotQA validation set.
Wood / Mood is 1984 American biog pagraphical persod cosedy d drana film di directed and produced by Tin
Burtony and starring Johnny Deep CU Elrrakar Ed hood [/Sent] The Hile concern: the pertoc 1f Wood Stc when
madi his best Knnve Filmt at w #s his re Jonship with actor Dels Lugoss played by Nartin Landane //sent
Fab lessita Parkery Patricin Angietie Jelf Janes Lise Parteh and BILT Nurray are arong the Support ing cast
ent kibScel Dnj Dejrickim
bockk/////boott booo In iniricke@ (torn my TEA 1968) is on ican directorj screenwriter and
T. [/Sena] lig Lives in Los Angele Lottornia [/sent] lie is best Monnl for Lingesiing hor fortor FAL such as "simsie
T". The Esorcise of Enily Roze", and Deliier Us Froe Evily, a5 well as tho1 Narvel Cinenatic Universe install
Fanll Poctor Strange /seii KEAIRSsnn, Arkensave/ Eihisiin IS censes dees place [OP] In Polaski Countyy
Arkansas, 10 United States [/snt]]
DE 2010 censos, [/Sent] Ock LiTTle RochaCoraay Not Iaan Statistiol Area [/sent] woodion and IES accompanylng I woodson Lake and
popolat ion was
part The LiTFLe
nanorako for Wood ST. 3 prosinent plantotion Oaner, trador, and susinezsran Largest the plantationn own T. [[ Lotes//teyyer Bates (bora June 1965) 15 on Imericin FUs3cian, MUSIC prsducer, and consoser [[/eett]
certury. [/sent] Woodson anjacent the Plantation,
genres)
Kunch,"
callaborate
2oebici
Killian Frieckin
Serridison shich [/eent] With
of The highest the pustorist of Aner Etwar Band Maralyn Nanson 2nd produced tn atum The MO ENATON and Reevnn 197100 Down 1AS
The director's Tilss loding Umardians The
bestiic
2010 and Its 2017 sequel
oiittoon be
bavis Mcod 3r (Octobor I. s/5tinvven Upsioc DoME LECOE SPE todeer lavis NoOd 3r (Octobor 19 1924
Decomber 10 198 was Anorican 11 Falmekkrr actor, writer, producer, and director //snnt] EIST Ua free EvEL
22014 fitel=/1 1/L---lliv Eivr Us from Pyill
2014 Ln Ii mnrror Filn dirwelm of by kolll bei Fiickoo wnd pI
Mduced by Jerry Bridkte [/ent]] The AI IS officially based oF 2001 ne Lictionnn book entil le Heware the N
ERT by Palpn Sarchte 20d Liso Lotter CODl and IS prrsting Ambaign Lonter Ehst 1T WEs Insplred red by actusl
Sccouncs [/Sent] The file stars Erid Sana, Eogar Rantret, Sean isrris) Olivis Hamn, and Joel Pelale in the rain
1.00
LOS and wts ro Bloased on Joly A 2014. (oond stzid oitis/iii collis 15 34 Arerican filmneke and actor |/sen
3 Ta attanded The bake University froe NE 1940 and the Iniversity of formiaa Los Angelm Trea 2607
6.75
[/sone] Re also Stadied cindal at
Untwerity of Soatiem (armorra Trod 1991
[/sen]] collis 1irst
TK wis assistant director for The Scott Berricksnn's shor! Love In huins [196] [/sent] in 1995, playe (Crankstaft
Kofeles ' lumrres Percent d Ja
(111)1// (/5//iiinssn
2017 sxpernatoral
0.9
Fror film directed Scott Derrickson Written Derrickson
Robert Carpil [[/sett] stars Ethon lavke Kallywood, California 2yck January 1931
fictioral Trink Kinn Teer Ellise
who discovers
his
0.5
dangel [[/ent]]
broils//t
in [/sent) in Ed Wood Filas mich 1plan T2o3 Space" 6lenn or dl Glendy
actor
acting
0.00
[[/een] lee took break Troe
the 1969ls and 1970s
[/seit] the has interes: the files
te reeserjed the 1900'
prolific Cnneeti Dilootor Strange IS 2015 Are Iean superlie
write
-0.25
Uce and direct several 7iles [/sent]
Strang (491 finn
direct several films, [/sent]<t> loctor Strange (2616 Hitto/ Strangg & 2015 Meericon supornere filr 135
ed on the Harvel (omic) character di the same namel produced by Narvel Studiod and distributed by Malt Disney Studh
-0.50
2s Pottonn Pictured [/Sent]] It the four Leenth fir of the harvel Cinesiti ld Universe (20) [f/een/ The Fil C
rectel by Scott ber Mirkickoon, who wrold 1tvith Jan Spaiin and L Ribert Caritll, and stars Benedict Cunberbatch a5 Stephen Strangth 3loM with Chivatol 5 5106or Rachal Mosson Bonediet Vong Rechaol Stihitsry Bon SALD E7S11 500
-0.55
31 Malns Mods EiHialeen and d1 Swimzon. (Sent In looctor Strange surgeon Strange Leams the mystic arts SF carterfensing
ter carce Pensing COF sccidest. [//ett
-1.00
Figure 7: After one fine-tuning epoch with GEMFormer Low (H < 0.3), supporting facts tokens include elements with unchanged uncertainty values, while the entropy of the rest of the document has increased. The HotpotQA validation set example heatmap depicting the difference in entropy values after the first epoch of fine-tuning and before fine-tuning.


Reviews:
[{"review_id": "36QvYmPMcS", "reviewer": "Reviewer_yjSH", "summary": "", "questions": "\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your thoughtful feedback and constructive critique of our paper.**\n\n**Clarifications on Marginal Improvements:**\n\nWe acknowledge your concern about the marginal absolute improvements over the strong RoBERTa baseline. The complex nature of MHQA tasks, which require reasoning across diverse parts of lengthy documents, often poses challenges for even incremental enhancements. GEMFormer improvement in performance is modest but consistent. It outperforms not only the baseline but also the number of previous MHQA methods such as Longformer, IRC, DFGN, RAG, and HUG and demonstrates comparable answer F1 scores to ETC, BigBird, and ReadTwice while utilizing a smaller number of parameters (see Appendix A). From a practical point of view, compared to the other models with similar performance, our method avoids additional pre-training after warmstarting from a public RoBERTa checkpoint, contributing to enhanced resource efficiency. For example, the ETC paper provides an ablation study showing that removing the pre-training task degrades the model's performance in HotpotQA. We believe that this underscores the efficacy of GEMFormer as a competitive solution for MHQA tasks.\n\nOnce again, thank you for your thoughtful evaluation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. The proposed GEMFormer method consistently surpasses the RoBERTa baselines across three MHQA datasets. Extensive ablation studies analyze the impact of the question context, fine-tuning, and memory content on performance, providing insights into why the proposed approach works.\n", "reasons_to_reject": "While results are consistently better than the baseline, absolute improvements are marginal - 1-3 F1 points over a strong RoBERTa baseline. \n", "questions_to_authors": "\n", "paper_topic_and_main_contributions": "This paper introduces GEMFormer (Global Explicit Memory Transformer), a method to augment the input to a pre-trained transformer model like RoBERTa with a global memory sequence consisting of tokens from the document that are important for answering the question. This leads to improvement in multi-hop question answering (MHQA) which involves reasoning over multiple parts of a long document to answer a question. The method has two stages - First, the uncertainty of each token is estimated using the pre-trained language model head of RoBERTa. Counterintuitively, tokens with low uncertainty are found to be important and are added to the memory. In the second stage, the memory-augmented input is fed to RoBERTa to make final MHQA predictions. Experiments on three MHQA datasets show improvements over the RoBERTa baseline. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xfyZV0kqXE", "reviewer": "Reviewer_LDqA", "summary": "", "questions": "1. It would be interesting to see whether the answer model really focuses on the GM tokens or it inversely guide model not to focus on them. Entropy will be lower when model have less amount information about a token, which implies that rare entities should have lower entropy. It is not clear those rare entities are really important to generate target answer. \n2. If entropy value based extraction method can identify the important tokens, then why GPT3.5 model performance degrades.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We would like to express our gratitude for your in-depth review of our paper. We have carefully considered each of your comments and concerns.**\n\n**Addressing the Importance of Low Entropy Tokens for Answer Generation:**\n\nWe have included an entropy heatmap as Figure 5 in Appendix D to visually demonstrate that the locations of low entropy tokens are closely aligned with supporting facts within the document. This observation points to the hypothesis that low entropy tokens might be helpful for generation because they overlap with the context with information relevant to the answer. This is supported by the analysis showing that performance is positively affected by the increased fraction of memory tokens retrieved from evidence (see Figure 4a in the main text). Thus, our current understanding here is that the inclusion of low entropy tokens in the global memory serves to increase the model's attention towards evidence sentences, facilitating a more informed answer generation process.\nAlso, the general motivation behind the entropy-based memory population is the following. We feed the model with a question and a context and compute entropy for each contextual token. This entropy is conditional with respect to the question and token-surrounding context. Such entropy measures how much entropy a token has remaining if we have already learned the question and the context. In other words, how easily the token can be predicted given the question and the context. Taking into account that a document is a collection of question-relevant and distractor paragraphs, the entropy of a task-relevant token should be lower than the entropy of the irrelevant one.\n\n\n**Addressing the Baseline with a Memory Bank Consisting of Extracted Keywords:**\n\nFollowing your suggestion to evaluate GEMFormer against a simple but relevant baseline, we have performed experiments using the YAKE! [1] method for the extraction of keywords from the context to populate a memory bank of 200 tokens for each data sample. The results (see Table 1 below) consistently show that the model's performance with the GEMFormer's memory augmentation (Table 1 of the main text) surpasses that achieved by the memory bank approach using extracted keywords. This underlines the effectiveness of our proposed method in capturing and utilizing relevant information for improved question answering. Moreover, keywords in memory lead to a performance inferior to the memoryless RoBERTa baseline.\n\n| Dataset   | Ans F1$_{\\pm std}$ | Supp F1$_{\\pm std}$ | Joint F1$_{\\pm std}$ |\n|-----------|:---------------------:|:-------------------:|:--------------------:|\n| HotpotQA  |     $72.9_{\\pm 0.29}$    |   $81.67_{\\pm 0.17}$   |     $61.44_{\\pm 0.33} $   |\n| 2WikiMHQA |    $63.98_{\\pm 1.01}$    |   $63.13_{\\pm 0.27}$   |           -          |\n| MuSiQue   |     $30.30_{\\pm 0.6}$    |   $63.43_{\\pm 0.64}$   |           -          |\nTable 1. Results of fine-tuning Roberta with global memory of YAKE! keywords.\n\n**Answers to Questions:**\n\nQuestion 1:  It would be interesting to see whether the answer model really focuses on the GM tokens or it inversely guide model not to focus on them. Entropy will be lower when model has less amount information about a token, which implies that rare entities should have lower entropy. It is not clear those rare entities are really important to generate target answer.\n\nAnswer: 1) The observation you made regarding the lower entropy of rare entities implying their importance is astute. However, upon examining Figure 6 in Appendix D, we find that the entropy of the pre-trained model associated with special tokens such as <t>, </t>, and [/sent] (which are new to the model's vocabulary) is notably high. This observation indicates that the model lacks information about special tokens, contrary to the expectation that rare entities should have low entropy. Consequently, our model indeed concentrates on GM tokens, and the inclusion of such tokens in the memory serves to enhance the model's focus on supporting facts.\n2) We collected rare (occurring in a context less than 5 times) tokens from each contextual document of the validation set for each dataset to compare the characteristics of the entropy distributions of rare tokens to the overall context entropy distributions. The descriptive statistics averaged over the validation set samples are presented in Table 2. As a result, we can see that the mean, median, mode and maximum values of rare tokens distributions match overall context distributions up to two decimal places for all three datasets. Also, both distribution types have skewness coefficients lower than $-1$ for each dataset, which means the distributions are skewed to the left. This observation confirms that rare tokens most frequently have high entropy values. \nWe also calculated how many global memory tokens are rare: for HotpotQA $61\\pm 21\\\\%$ of memory tokens are rare ones, for 2WikiMHQA – $9\\pm 9\\\\%,$ and for MuSiQue – $16\\pm 15 \\\\%$. These observations imply that, in practice, rare tokens tend to have high entropy relative to the overall context entropy, and for two out of three datasets rare entities are infrequent in global memory. The rare tokens analysis suggests that during fine-tuning, the entropy of rare tokens related to the question becomes lower compared to irrelevant ones. This allows the model to preferentially store in the memory more relevant entities.\n\n|    | Tokens            |        Min       |       Mean       |      Median      |       Mode       |        Max       | Skewness (Fisher-Pearson coef.) | Kurtosis (Fisher) |\n|-----------|-----------------------|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:-------------------------------:|:-----------------:|\n| HotpotQA  | Rare | $0.043_{\\pm 0.040}$ | $0.465_{\\pm 0.034}$ | $0.487_{\\pm 0.039}$ | $0.495_{\\pm 0.065}$ | $0.621_{\\pm 0.017}$ |         $-1.30_{\\pm 0.56}$         |   $2.42_{\\pm 2.66}$  |\n|           | Overall          | $0.017_{\\pm 0.021}$ | $0.460_{\\pm 0.038}$ | $0.487_{\\pm 0.042}$ | $0.489_{\\pm 0.076}$ | $0.625_{\\pm 0.016}$ |         $-1.38_{\\pm 0.59}$         |   $2.51_{\\pm 2.72}$  |\n| 2WikiMHQA | Rare | $0.101_{\\pm 0.065}$ | $0.503_{\\pm 0.025}$ | $0.530_{\\pm 0.026}$ | $0.556_{\\pm 0.033}$ | $0.628_{\\pm 0.015}$ |         $-1.47_{\\pm 0.48}$         |   $2.73_{\\pm 2.58}$  |\n|           | Overall         | $0.081_{\\pm 0.059}$ | $0.503_{\\pm 0.025}$ | $0.529_{\\pm 0.027}$ | $0.553_{\\pm 0.045}$ | $0.635_{\\pm 0.016}$ |         $-1.42_{\\pm 0.45}$         |   $2.51_{\\pm 2.40}$  |\n| MuSiQue   | Rare | $0.006_{\\pm 0.012}$ | $0.410_{\\pm 0.012}$ | $0.430_{\\pm 0.012}$ | $0.450_{\\pm 0.013}$ | $0.586_{\\pm 0.015}$ |         $-1.55_{\\pm 0.23}$         |   $3.33_{\\pm 1.09}$  |\n|           | Overall          | $0.003_{\\pm 0.006}$ | $0.409_{\\pm 0.014}$ | $0.431_{\\pm 0.013}$ | $0.452_{\\pm 0.013}$ | $0.590_{\\pm 0.015}$ |         $-1.59_{\\pm 0.23}$         |   $3.37_{\\pm 1.26}$  |\n\nTable 2. Descriptive statistics for all context tokens’ distribution and rare (less than 5 occurrences in the context document) tokens distribution averaged over the validation set samples (with standard deviation values) for HotpotQA, 2WikiMHQA, and MuSiQue datasets.\n\nQuestion 2: If entropy value based extraction method can identify the important tokens, then why GPT3.5 model performance degrades.\n\nAnswer: GPT-3.5 scores provided in the paper were obtained following the common practice of storing a contextual document in a vector database as a collection of document sentences and retrieving sentences by relevance to the question. In experiments, we used the document-based QA pipeline from the Langchain library. The most relevant sentences were concatenated to the question and sent to ChatGPT for answer prediction. In this case, we found that the final performance of GPT-3.5 is highly dependent on the effectiveness of the evidence retrieval process. If the retriever fails to accurately extract evidence, the subsequent memory augmentation does not rectify this shortcoming. Furthermore, we conducted additional experiments where we fed a model with a concatenation of a question and a contextual document with and without global memory augmentation. The detailed results are presented in Table 3.\n\n| ChatGPT Input                         || HotpotQA || 2WikiMHQA || MuSiQue||\n|---------------------------------------|:---------:|:--------:|:--------:|:---------:|:-------:|:---------:|:-------:|\n|                                       | Ans F1 |  Supp F1 | Joint F1 | Ans F1 | Supp F1 | Ans F1 | Supp F1 |\n| Question                              |   $19.89$   |     -    |     -    |   $15.46$   |    -    |    $4.98$   |    -    |\n| Question + Retrieved Context          |   $24.75$   |   $47.53$  |  $ 16.43$  |   $25.36$   |  $42.65  |  $ 13.08 $  |  $40.59$  |\n| Question + Memory + Retrieved Context |   $21.97$   |  $ 43.5 $  |   $14.84$  |  $ 26.08 $  |  $ 41.7$  |   $13.05$   |  $38.04$  |\n| Question + Context                    |  $ 51.34$   |  $ 20.6 $  |   $11.7$   |   $43.22$   |  $30.55$  |  $ 30.51 $  |  $ 38.6$  |\n| Question + Memory + Context           |   $57.34$   |   $20.9$   |   $12.94$  |   $51.27$   |  $33.94$  |   $35.98$   |  $40.33$  |\n\nTable 3. F1 scores for dev sets of HotpotQA, Musique, and 2500 samples subset of 2WikiMHQA subset for GPT-3.5-turbo checkpoint.\n\nThe prompt for question+context inputs was the following: \n\n*System: You are a world-class algorithm to answer questions based on the given text. Store the answer and the supporting evidence from the text in the following structure: {‘answer’: ‘answer string’, supporting evidence’: [“list of supporting evidence sentences”]\nHuman: Text: <context> Question: <question>*\n\nThe memory-augmented input prompt was the following:\n\n*System: You are a world-class algorithm to answer questions based on the given text and memory. Use the provided memory to detect question-related information in text. Store the answer and the supporting evidence from the text in the following structure: {‘answer’: ‘answer string’, ‘supporting evidence’: [“list of supporting evidence sentences”]}\nHuman: Memory: <mem> Text: <context> Question: <question>*\n\nAlthough the answer prediction scores were significantly improved compared to the retrieval setting of GPT-3.5, they still fell short of the fine-tuned RoBERTa scores. However, in this setting, we observed notable improvements in answer F1 scores ($+6$ F1 for HotpotQA, $+7.94$ F1 for 2WikiMHQA, and $+5.47$ F1 for MuSiQue), signifying the efficacy of memory augmentation in enhancing answer generation.\n\nWe greatly appreciate your attention to these critical aspects of our work. Your evaluation and questions have guided us to provide more comprehensive explanations and supporting evidence for our manuscript. We are confident that these clarifications further underscore the significance and validity of our proposed GEMFormer method for multi-hop question answering. We hope that our efforts in addressing your concerns will lead to a more compelling paper that merits acceptance to EMNLP.\n\nThank you once again for your time and invaluable feedback.\n\n[1] Campos R., Mangaravite V., Pasquali A., Jorge A.M., Nunes C., and Jatowt A. (2018). A Text Feature Based Automatic Keyword Extraction Method for Single Documents. In: Pasi G., Piwowarski B., Azzopardi L., Hanbury A. (eds). Advances in Information Retrieval. ECIR 2018 (Grenoble, France. March 26 – 29). Lecture Notes in Computer Science, vol 10772, pp. 684 - 691.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Entropy-guided token extraction is a novel way to build a global memory.", "reasons_to_reject": "1. It is not clear that how tokens with low entropy can be important to generate answer. It needs elaborate discussion on what type of tokens are being extracted in the global memory. \n2. Important baseline is missing. There is no comparision with very obvious method of building memory bank using existing keywords extraction methods as the main purpose of this paper is preparing a set of tokens that would be helpful for answer. Without further experimental results, it is hard to evaluate the significance of this method.", "questions_to_authors": "1. It would be interesting to see whether the answer model really focuses on the GM tokens or it inversely guide model not to focus on them. Entropy will be lower when model have less amount information about a token, which implies that rare entities should have lower entropy. It is not clear those rare entities are really important to generate target answer. \n2. If entropy value based extraction method can identify the important tokens, then why GPT3.5 model performance degrades.", "paper_topic_and_main_contributions": "The paper focus on multi-hop question answering task. A new way of using entropy values of entities to extract a set of tokens named as global memory has been presented in this paper. It computes the entropy for each token from the probabilities of tokens returned by RoBERTa language modeling head layer.  Afterward, the constructed memory is used along with given context to generate answer for question.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mNZobUFU1t", "reviewer": "Reviewer_Wfje", "summary": "", "questions": "A: How to define uncertainty threshold on new multi-hop dataset?\nB: Is GEMFormer better than other multi-hop QA models? (https://arxiv.org/abs/2204.09140)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your valuable feedback and constructive insights. Below, we address each of your points.**\n\n**Addressing the Dataset-Specific Uncertainty Threshold Values:**\n\nWe acknowledge your concern regarding the dataset-specific uncertainty threshold values used in the first stage of the memory population. In this regard, our method is not different from any other method of fine-tuning or trainable model augmentation (i.e. LoRA) that assumes training on the target dataset. Designing a method that can generalize to other tasks without fine-tuning is not the goal of the presented study and achieving such generalization is not claimed in the text. This places GEMFormer in the research area where pre-trained LMs are augmented and fine-tuned for specific tasks to achieve better quality compared to more universal models. We believe that the research in this area still has significant value for the NLP community and deserves publication.\n\nWe understand the importance of reproducibility in research findings. As part of our code release, we will include detailed explanations of how these thresholds were chosen, allowing fellow researchers to adapt our approach to the new QA and multi-hop QA datasets.  The datasets we used are all publicly available. We are committed to ensuring that our work is as reproducible as possible. If you have specific concerns about reproducibility, we would be happy to address them.\n\n**Clarifications on the Roberta Usage and the Comparison with Previous MHQA LMs:**\n\nThe pre-trained RoBERTa-base was selected as a model starting point following the Longformer, BigBird, and ETC architectures that represent base-sized generative Transformers from the top of the HotpotQA leaderboard to the moment of the manuscript submission. We compare the performance of our approach not only to basic RoBERTa but also to other relevant MHQA models such as Longformer, ETC, BigBird, and ReadTwice. Please, refer to Table 4 in Appendix A for a comparison with 11 other models. This table underscores the improvements our approach brings to this specific task.\n\n We also include in comparison GPT-3.5 as a universal generalizable LLM alternative for answering multi-hop questions. ChatGPT evaluation results can be found in Table 7 of Appendix C and the extended data are provided in the following Table 1.\n\n\n**Answers to Questions:**\n\nQuestion A: How to define uncertainty threshold on new multi-hop dataset?\n\nAnswer: In our experiments, we conducted a systematic grid search within a range from the minimal value to the mode (not included) of entropy distributions averaged across train samples. We used the fine-tuned baseline entropy values. The optimal threshold value for memory augmentation was determined through experimentation. This approach ensures that the threshold value is adaptable to new multi-hop datasets while maintaining the performance gains achieved through our method.\n\nQuestion B: Is GEMFormer better than other multi-hop QA models? (https://arxiv.org/abs/2204.09140)\n\nAnswer: The comparison between GEMFormer and other multi-hop QA models is thoroughly addressed in Appendix A (Table 4) of our paper. DFGN, SAE, and ETC architectures mentioned in the suggested review paper are presented in Table 4.\n\nNotably, we have also extended our evaluation to include GPT-3.5, as detailed in Appendix C (Table 7) and in the revised version of our GPT-3.5 experimental results in  Table 1 below. \n\n| ChatGPT Input                         || HotpotQA || 2WikiMHQA || MuSiQue||\n|---------------------------------------|:---------:|:--------:|:--------:|:---------:|:-------:|:---------:|:-------:|\n|                                       | Ans F1 |  Supp F1 | Joint F1 | Ans F1 | Supp F1 | Ans F1 | Supp F1 |\n| Question                              |   $19.89$   |     -    |     -    |   $15.46$   |    -    |    $4.98$   |    -    |\n| Question + Retrieved Context          |   $24.75$   |   $47.53$  |  $ 16.43$  |   $25.36$   |  $42.65  |  $ 13.08 $  |  $40.59$  |\n| Question + Memory + Retrieved Context |   $21.97$   |  $ 43.5 $  |   $14.84$  |  $ 26.08 $  |  $ 41.7$  |   $13.05$   |  $38.04$  |\n| Question + Context                    |  $ 51.34$   |  $ 20.6 $  |   $11.7$   |   $43.22$   |  $30.55$  |  $ 30.51 $  |  $ 38.6$  |\n| Question + Memory + Context           |   $57.34$   |   $20.9$   |   $12.94$  |   $51.27$   |  $33.94$  |   $35.98$   |  $40.33$  |\n\nTable 1. F1 scores for dev sets of HotpotQA, Musique, and 2500 samples subset of 2WikiMHQA subset for GPT-3.5-turbo checkpoint.\n\n\nWe would like to thank you for your review once again. We hope that our clarifications and additional data clarify the contribution and significance of our study for QA tasks. We are confident that the revisions made in response to your feedback will strengthen the quality of our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed global memory population mechanism is indeed an evidence collection mechanism with heuristic threshold rules, which requires no additional training.", "reasons_to_reject": "(1) The heuristic uncertainty threshold used in first-stage of memory population requires specific values for different dataset, which is lack of generalization and reproducibility.\n(2) The overall framework perhaps works on general long document question answering and more LLMs. The authors only use Roberta and multi-hop QA for testing. In contrast, if the authors believe they made special progress on multi-hop QA, then analysis on hop-depend questions should be provided. For example, how GEMFormer is superior than other previous LLMs that can deal with multi-hop QA as well. There're only basic Roberta model be listed as a baseline.", "questions_to_authors": "A: How to define uncertainty threshold on new multi-hop dataset?\nB: Is GEMFormer better than other multi-hop QA models? (https://arxiv.org/abs/2204.09140)", "paper_topic_and_main_contributions": "The paper proposed a two-stage transformer-based framework GEMFormer towards multi-hop question answering task. In specific, GEMFormer first generate global memory based on given question and multiple document segments, then reuse question, document segments and newly produced global memory to generate answers. In first stage, the global memory is populated by ranking an entropy value over all tokens in the input sequence.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please refer to quite some models listed in: https://arxiv.org/abs/2204.09140"}, "extra_text": {}}, {"review_id": "5bKdeDBsnQ", "reviewer": "Reviewer_E7Ta", "summary": "", "questions": "Line 063: Is there previous literature on defining token importance as language model uncertainty? Some more background on this idea would be handy in the related work section. ", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are thankful for your feedback and thoughtful examination of our paper. Your comments provide valuable guidance for improving the clarity and overall appeal of our work.**\n\n**Addressing Clarity in the Results and Discussion Section:**\n\nWe genuinely thank you for raising this concern. The Results and Discussion section indeed encompasses a range of evaluations, including experimental results, ablation studies, and analyses of the impact of memory content on model performance. We grouped these assessments within a single section due to space constraints. The intention behind this approach is to present a comprehensive overview of our methodology's effectiveness. To enhance clarity, we will ensure that the motivations behind the ablation studies and analysis methods are clearly articulated in the camera-ready version of the paper.\n\n**Responses to Queries:**\n\nQuery: Line 063: Is there previous literature on defining token importance as language model uncertainty? Some more background on this idea would be handy in the related work section.\nResponse: Similar to our usage of language model predictions, in [1, 2] authors use a language model (LM) probability in machine translation task.\nA machine translation model probability and LM probability are used to measure the pointwise mutual information (MI) between each target token and its source context under the condition of the previous target context. The estimated values are considered token importance and are used as weightings to rescale the training loss. Such a training approach makes the model pay more attention to more “informative\" tokens.\nTo the best of our knowledge, there is no literature specifically discussing token importance based on language model uncertainty. Nonetheless, our motivation behind this approach was the following. Language model entropy for a given token is a conditional entropy with respect to the question and the contextual document. It shows the amount of information needed to describe the token given the question with the task-defining information and the source context. So lower uncertainty values should reflect tokens that are more relevant to the task.\n\n**Refinements in Language and Presentation:**\n\nWe sincerely appreciate your observation regarding sentence length and punctuation. We have revised the sentences in question as follows:\n\nLine 006: The landscape of MHQA approaches can be classified into two primary categories. The first group focuses on extracting supporting evidence, thereby constraining the QA model's context to predicted facts. Conversely, the second group relies on the attention mechanism of the long input encoding model to facilitate multi-hop reasoning.\n\nLine 093: GEMFormer adapts the idea of interpretable memory by storing tokens from the input sequence. This augmentation of memory enriches long inputs with essential global contextual information important for the task.\n\n**Low Reproducibility Score:**\n\nWe understand the importance of reproducibility in research findings. As part of our code release, we will include detailed explanations of how the global memory was populated and how the baseline and the models with memory augmentation were fine-tuned, allowing fellow researchers to adapt our approach to the new QA and multi-hop QA datasets. The datasets we used are all publicly available. We are committed to ensuring that our work is as reproducible as possible. If you have specific concerns about reproducibility, we would be happy to address them.\n\n\nOnce again, we thank you for your invaluable feedback, and we are committed to refining our paper in alignment with your suggestions.\n\n\n[1] Songming Zhang, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jian Liu, and Jie Zhou. 2022. Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2377–2389, Dublin, Ireland. Association for Computational Linguistics.\n\n[2] Aditi Jain, Nishant Kambhatla, and Anoop Sarkar. 2023. Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 341–356, Toronto, Canada (in-person and online). Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting new model for memory augmentation. ", "reasons_to_reject": "The paper is not clearly structured. For instance, a lot of content in the Results and Discussion section describes the methods used which have not yet been discussed, making it difficult to follow. ", "questions_to_authors": "Line 063: Is there previous literature on defining token importance as language model uncertainty? Some more background on this idea would be handy in the related work section. ", "paper_topic_and_main_contributions": "The authors present a novel method for solving the multi-hop question answering task, namely GEMFormer, which is a pre-trained model that is fine-tuned with memory augmented input. The results are improved over the baseline on three datasets. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 006: The sentence is too long and not clearly punctuated\n\nLine 093: Same comment as line 006, too many conjuncts in a sentence make it difficult to read\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting short paper, proposing a memory mechanism to enhance the multihop question answering (MHQA) capabilities of transformer models.

While it seems like reviewers agree the idea is interesting in general, review scores range from 2 to 4. I have read the reviews and rebuttals carefully, and have given the paper an in-depth read to try and give it a fair assessment.

## Summary
This work introduces GEMFormer (Global Explicit Memory Transformer) to address the task of MHQA. One significant challenge of MHQA is the fact that information relevant for answering the question is scattered across long, disparate documents, which might not fit into the underlying transformer's context window; even when using models with large window sizes, the attention mechanism might lead to interference of local and global information in the token representations of relevant pieces of information, potentially harming model performance.

The proposed MHQA mechanism is very simple:
- memory construction:
  - Long background documents are split into segments fitting into the transformer's context window.
  - the question is combined with each segment, and put through the transformer and LM head in a forward pass.
  - segment tokens with either highest (top-k) or lowest (below a threshold) LM entropy are selected and copied into the global memory.
- MHQA training
  - each segment is combined with the question and global memory, and the MHQA task is trained in standard fashion.

The authors find that of the two proposed entropy-based memory selection methods, selecting low-entropy tokens works best, with the threshold tuned as a hyperparameter on the MHQA dataset.

The method is trained and tested on 3 MHQA datasets, HotpotQA, 2Wiki, and MuSiQue, with a RoBERTa-base model, and is able to outperform the memory-free baseline on each. While the main body does not contain other -- more relevant -- baselines, these are evaluated against in the appendix in order to make room for some ablation studies that provide some interesting insights into GEMFormer behaviour.

The models chosen for comparison in the appendix seem relevant and sensible, using transformer networks of roughly comparable sizes to GEMFormer. GEMFormer compares favourably overall, even though it is not able to outperform some of the more involved baselines.

## Strengths:
- the presented memory construction method is novel, simple, yet seems effective
- the methodology is clearly presented and easy to follow
- GEMFormer is evaluated on a variety of different datasets.
- the conducted experiments are extensive, and seem to support the author's claims that (i) memory can help MHQA performance; and (ii) selecting low-entropy tokens is required to fill the memory with relevant information.

## Weaknesses:
- the entropy threshold for the (better performaing) low-entropy memory selection is a hyperparameter requiring per-dataset tuning.
- results are consistently higher than the baseline, but not to a high degree
- some parts, especially the discussion, can be hard to follow, as new concepts and ablations are introduced unexpectedly.

## Conclusion
In my opinion, the authors' rebuttals to reviews were satisfactory, and cleared up relevant questions with convincing answers.\
I find myself agreeing with the authors that a per-dataset hyperparameter such as the proposed entropy threshold can be regarded as part of dataset-specific finetuning, which is very much standard -- I am not aware of an MHQA method that is able to be trained on one, and then performs its best on all other MHQA datasets.\
Even though no new state of the art is established (SOTA is not an acceptance criterion in itself), I believe the proposed method shows enough novelty with its simple yet effective approach to memory construction and utilisation for the MHQA task.\
Overall, the work seems sound and merits acceptance; excitement is dampened somewhat by the relatively small improvements achieved, but this is a subjective measure separate from soundness.

While the writing could be slightly improved, such as better splitting of the discussion and ablation parts, as well as clarifying why low entropy works (low entropy = high certainty of tokens, conditioned on the question; changing "least uncertain" to "most certain" to avoid a potentially confusing double negative), this should be no problem for a camera-ready version, and the paper is otherwise very easy to follow.
Decision: Accept-Main