Divide & Conquer for Entailment-aware Multi-hop Evidence Retrieval

Anonymous ACL submission

Abstract
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041

Question: What nationality was James Henry Miller’s wife?
Answer: American

Lexical and semantic matches are commonlyused as relevance measurements for information retrieval. Together they estimate the semantic equivalence between the query and thecandidates. However, semantic equivalence is
not the only relevance signal that needs to beconsidered when retrieving evidences for multihop questions. In this work, we demonstratethat textual entailment relation is another important relevance dimension that should be considered. To retrieve evidences that are either
semantically equivalent to or entailed by thequestion simultaneously, we divide the task ofevidence retrieval for multi-hop question answering (QA) into two sub-tasks, i.e., semantic textual similarity and inference similarity
retrieval. We propose two ensemble models,EAR and EARnest, which tackle each of the
sub-tasks separately with off-the-shelf retrieval
models, and jointly retrieve sentences with theconsideration of the diverse relevance signals.
Experimental results on HotpotQA verify that
our models not only significantly outperform
all the single retrieval models it is based on,
but is also more effective than two intuitiveensemble baseline models.

1.

Supporting Evidences
Ewan MacColl(1) James Henry Miller (25 January 1915 – 22 October1989), better known by James Henry Miller stage name
Ewan MacColl, was an English folk singer, songwriter,
communist, labour activist, actor, poet, playwright andrecord producer .
Peggy Seeger(2) Margaret "Peggy" Seeger (born June 17 , 1935) is an
American folksinger.
(3) She is also well known in Britain, where she has lived
for more than 30 years, and was married to the singer andsongwriter Ewan MacColl until his death in 1989.
Semantic Equality
James Henry Miller (Q) ≈ James Henry Miller (25 January
1915 – 22 October 1989), better known by James Henry
Miller stage name Ewan MacColl (1)Textual Entailment
American (2) ⊢ nationality (Q)was married to (3) ⊢ wife (Q)Figure 1: An example from the HotpotQA dataset(Yang et al., 2018) showing the two different dimensionsof relevance between the question and its supporting
evidences.

includes semantic equivalence, referential equality,
and textual entailment. While referential equality can be mostly solved by coreference resolution
and entity linking, semantic similarity and textual
entailment require deep semantic understanding
between question and context. Most researches
use semantic similarity as a shorthand to refer tothe well-known Semantic Textual Similarity (STS)tasks, where semantic similarity is operationallydefined by the annotation guidelines, which fallaround the notion of semantic equivalence, i.e.,“Do these two sentences mean the same thing?”(Lin et al., 2021b) While textual entailment is aframework that captures semantic inference. Textual Entailment (TE) of two text fragments can be
defined as the task of deciding whether the meaning

Introduction

Widely adopted QA approaches use a two-stage
pipeline, i.e., a retriever module followed by areader module (Chen et al., 2017). The retriever is
responsible for collecting relevant contextual evidence fragments; then the reader module combines
the relevant information from the retriever module
to infer the answer. While it is common to utilize
an inference model as a reader to infer the correct
answer from the retrieved context, most existing
retrievers only focus on lexical and/or semantic
matches, ignoring the inference relations between
question and context.
According to formal semantic notions, the semantic relationship between two text fragments
1

042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057

058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106

2.

of one text fragment can be inferred from anothertext fragment. That is, a premise T entails a hypothesis H if, typically, a human reading T would inferthat H is most likely true. For example, T: Jack
sold the house to Peter. H: Peter owns the house.
Here H can be inferred from T, so T entails H.

2.1.

Most of the evidence retrieval works only measure the semantic textual similarity between thequestion and candidate corpus to determine the relevance. This works for single-hop questions, inwhich relevant information usually share the same
entity mentions with the question. However, it is
not sufficient for multi-hop questions. For multihop questions, the relevant relationship between
question and evidence(s) is beyond the lexical or
semantic similarity that is targeted by most retrievers, especially for secondary hops.
The evidence retrieval for the multi-hop QA task
broadly involves two different dimensions of relevance to measure: semantic equivalence and textual
entailment. Thus, we divide the multi-hop QA evidence retrieval task into two separate retrieval tasks,
where each aims to retrieve a subset of sentencesthat score highly on one of the relevance dimension
respectively, and then combine them to output the
final ranking with an ensemble model. To build
a retrieval method that does not rely on a large
training set with evidence annotations, we use bothoff-the-shelf statistical models and pre-trained language models as base models to capture the diverse
relevance signals.

Related Work

Text Retrieval

2

108

Traditional retrieval models such as TF-IDF and
BM25 (Trotman et al., 2014) use sparse bag-ofwords representations to collect lexical matching
signals (e.g., term frequency). Such sparse retrieval
models are mostly limited to exact matches. Dense
Retrieval models move away from sparse signals
to dense representations, which help address the
vocabulary mismatch problem. These models canbe categorized into two types according to theirmodel architecture, representation-based (Huang
et al., 2013; Shen et al., 2014) and interactionbased models (Pang et al., 2016; Lu and Li, 2013;
Guo et al., 2016; Mitra et al., 2017). Hybrid methods (Lin et al., 2021a; Gao et al., 2020; Karpukhin
et al., 2020; Shan et al., 2020) aggregate the dense
retrieval with sparse retrieval methods to bettermodel relevance. The entailment-aware retrieval
models we propose are also hybrid methods thatcombine sparse and dense retrieval methods, but
our method is unsupervised. Further, we combine asparse model with multiple dense models in order
to consider diverse relevance signals, i.e., textual
entailment in addition to semantic equivalence.

109

2.2.

132

Multi-hop Evidence Retrieval

Research works on multi-hop evidence retrieval
can be broadly categorized into two directions: (1)Question decomposition: (Min et al., 2019; Jiang
and Bansal, 2019; Fu et al., 2021; Perez et al., 2020;
Talmor and Berant, 2018) decompose multi-hop
questions into multiple single-hop sub-questions.
Instead of training a decomposer, our entailmentaware retrieval models tackle this task in the opposite direction. That is, we assemble candidate
sentences pairs that carry different relevance signals to match against the question. (2) Iterative
evidence retrieval: Feldman and El-Yaniv (2019)
proposed a method to iteratively retrieve supporting paragraphs, using the paragraphs retrieved inprevious iteration to reformulate the search vector.
(Asai et al., 2020) iteratively retrieve a subsequentpassage in the reasoning chain with an RNN. Qi
et al. (2019) trained a retriever to generate a queryfrom the question and the available context at eachstep. While iterative retrieval considers evidence
retrieval as a sequence process, so that the accuracy of subsequent retrieval steps highly depends
on previous decisions, our method jointly consider
high potential sentences pairs simultaneously.

Our contributions in this work include: (1) We
call attention to the fine-grained aspects of relevance for multi-hop QA evidence retrieval. In particular, textual entailment relation should be taken
into consideration along with semantical equivalence in order to cover a more accurate relevant
context. (2) We propose two ensemble models
that combine diverse relevance signals captured bythree off-the-shelf base models. Our experimental
results demonstrate that not only are the individual base retrieval models necessary in evidence
retrieval but cooperate advantageously to produce a
better ranking for multi-hop QA evidence retrieval
when used together. (3) We empirically show theeffectiveness of the proposed ensemble retrieval
models by evaluating on the HotpotQA dataset,and show they outperform all the base models andalso several ensemble baselines.

107

110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131

133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156

3.

157

Methodology

163

In this section, we introduce two ensemble models
for entailment-aware multi-hop QA evidence retrieval. At a high-level, we model diverse relevance
relationships with three base models, and combine
the relevance signals they capture to jointly retrieve
candidate evidences for multi-hop questions.

164

3.1.

165

190

Given a multi-hop question Q and a corpus C ={P1 , P2 , . . . , Pm } containing a set of documentsor paragraphs, the evidence retrieval task is to rank
the candidate text sentences (the “unit of indexing")from C and return a list of top N most relevant ones
that provide sufficient and less distracting information for answering Q. Estimating the degree of
relevance of each candidate sentence to the question is clearly an integral part of the task. To build
a better retriever for multi-hop questions, two dimensions of relevance (i.e., semantic equivalenceand entailment) need to be both considered in orderto provide a more accurate estimation of relevancefor each candidate sentence.
We divide this multi-criterial task into two separate ranking subtasks, which include semantic
equivalence as well as textual entailment. 1 Bothtasks require comparing information between thequestion and candidates, but the objectives of thecomparison are different. In this work, we propose to capture the entailment relations in parallelwith the semantic equivalence with separate models, which produce different and potentially conflicting rankings. The goal is to combine them tofigure out an aggregated ranking that promote gold
evidence sentences to the top of the list.

191

3.2.

192

We chose three off-the-shelf base models to capture diverse relevance patterns. To better estimate
semantic equivalence, we use both a sparse model(i.e., BM25) and a dense model (i.e., transformers
pre-trained for semantic search) to examine exact
match and semantic match respectively. In addition,
we utilized another dense model pre-trained on
QNLI dataset for capturing entailment relation. For
the dense models, we choose two pre-trained crossencoders (CE)2 (Reimers and Gurevych, 2019),

158
159
160
161
162

166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189

193
194
195
196
197
198
199
200
201

which are trained by taking concatenated questionand candidate sentence as a single input sequenceand generate an estimate of relevance score directly.
The two pre-trained cross-encoders we choose are:

Task

202
203
204
205

MSMARCO Passage Cross-Encoder3 istrained on the MS Marco Passage Ranking dataset(Bajaj et al., 2016) for information retrieval.
MS MARCO (Microsoft Machine Reading
Comprehension) is a large scale corpus consistsof about 500k real search queries from the Bing
search engine with 1000 most relevant passages.
The model is trained to rank the most relevant
passage that answers the query labeled by humanas high as possible.

206

QNLI Cross-Encoder4 is a pre-trained modelobtained using the Question Natural Language
Inference (QNLI) dataset introduced by GLUE
Benchmark (Wang et al., 2018). QNLI was automatically derived from SQuAD, with the processing target of question-answer entailment.

216

BM25

MSMARCO
CE

QNLI
CE

% Ques(k=3)% Ques
(k=5)

✓
✗
✗

✗
✗
✓

✗
✓
✗

14
25
20

10
22
16

✗
✗

✓
✓
✗

✓
✗
✓

33
38
64
35

30
35
62
33

✗

✗

44

29

✗

207
208
209
210
211
212
213
214
215

217
218
219
220
221

Table 1: Each line shows the percentage of questions
that have at least one evidence ranked within top-kby the model marked with a ‘✓’ but beyond top-k bymodel(s) marked with a ‘✗’. For example, there are 14%
questions with at least one evidence sentence is ranked
within top-3 by BM255 , but ranked beyond top-3 by
MSMARCO CE and QNLI CE; 35% questions withat least one evidence sentence ranked within top-3 by
QNLI CE but ranked beyond top-3 by MSMARCO CE.

Base Models

3.3.

Ensemble Models

Since the three base models aforementioned independently capture diverse relevance signals and arecomplementing each other as shown in table 3.2,
candidate sentence at each transformer encoder layer.
3
https://www.sbert.net/docs/
pretrained-models/ce-msmarco.html
4https://www.sbert.net/docs/
pretrained_cross-encoders.html#
squad-qnli
5
https://pypi.org/project/rank-bm25

1

For the focus of this work, we conduct coreference resolution within each paragraph before the retrieval task.
2
While usually much faster, bi-encoders are less effective
than cross-encoder models because the latter can exploit relevance signals derived from attention between the query and

3

222
223
224
225

228

an ensemble model should potentially improve thefinal retrieval performance if an appropriate aggregation strategy designed to combine them.

229

3.3.1.

230

Average ranking (AR) is a simple ensemble
ranking model which combines the ranking outputs from the multiple base models that ranks all
candidate sentences independently. A rank of acandidate sentence obtained by each base model iswith respect to the relevance signal the base modeltargets on. Thus, each sentence has M ranks (where
M is the number of base models). The final ranking
is obtained by sorting the sum of all M rankingsthat each sentence received.

226
227

231
232
233
234
235
236
237
238
239

Ensemble Baselines249

Similarity Combination (SimCom) calculates
hybrid retrieval scores through a linear combination of sparse and dense scores. For a given question, we aggregate the scores produced by the basemodels through a weighted average for each candidate sentence, called Question Evidence Relevance(QER) (see the equation in Appendix A). QER arethen used to rank the candidate evidence sentences,
so that candidate sentences with high relevance tothe question are promoted to the top of the list.

250

3.3.2.

251

In this work, we propose an entailment-aware retrieval (EAR) method to jointly consider pairs ofcandidate sentences that potentially contain complementary relevance signals. We form such sentence pairs using the Cartesian product of two setsof top ranked candidate sentences with respect
to semantic equivalence and textual entailment
correspondingly. While BM25 and MSMARCO
Cross-Encoder capture exact and semantic matches,
respectively, they both aim for estimating STS.
Thus, we take the union of top ranked sentences by
BM25 and MSMARCO Cross-Encoder as a unified set A = {Sa1 , Sa2 , Sa3 , Sa4 }, and top ranked
sentences by QNLI Cross-Encoder as another set
B = {Sb1 , Sb2 , Sb3 }. The pairs we consider are
P = A × B = {(a, b) | a ∈ A ∧ b ∈ B}.
We then concatenate the two sentences of eachpair as a sequence to score against the question witha reranker6 , such that the top scored sentence pair
(Sai , Sbj ) is most likely to form a compositional
relevant context covering both semantic equivalence and entailment relevance signals. When Sai

240
241
242
243
244
245
246
247
248

252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272

Entailment-Aware Retrieval

6
We use the MSMARCO Cross-Encoder as the reranker
since it is the best performing base model.

4and Sbj are examined individually, there is highchance that Sai receives a low IS score from the
QNLI cross-encoder and ranked down to the list,
Sbj can be scored and ranked low by BM25 and
MSMARCO cross-encoder. Thus, either using individual base models or aggregating ranking orscores with the ensemble baseline models, Sai andSbj are unlikely to be both promoted to the top of
the ranking list. Finding the best combination fromthe top-ranked subsets with respect to both semantic equivalence and textual entailment efficiently
takes the compositional requirement into consideration. We further concatenate the question q withthe pair Sai and Sbj as a new query to re-rank therest of the candidate sentences.

273

3.3.3.

288

EARnest

274
275
276
277
278
279
280
281
282
283
284
285
286
287

Evidences for a multi-hop question should be intuitively related, and often logically connected via
a shared named entity that would allow a humanreader to connect the information they contain. The
presence of a shared named entity between two candidate sentences often indicates the likelihood thatthe sentence pairs relate to each other and, thus,
they can be connected to form a coherent context
for the question.
To leverage such connection as an additional cue,
we add a named entity similarity term (NEST) to
the scoring function of the reranker in EAR whenestimate the top scored sentence pairs as

289

QEREarnest = (1 + N EST ) ∗ Sim(q, si ∥ sj )302

where Sim() is the scoring function of the reranker,
which scores the concatenation of sentence pair Si
and Sj against the question. N EST is a binary
switch, that is, if the two sentences share one ormore named entity, the promotion mechanism isactivated; otherwise it is deactivated.

303

Named Entity Similarity Term Besides using
SpaCy (Honnibal et al., 2020) to recognize namedentities with common entity types (such as namesof people, places, organizations), we also consider
titles of documents and phrases between a pair of
single or double quotes. When comparing whether
two sentence share an entity, we apply basic normalization (i.e., lower case, removing articles andspecial punctuations) and fuzzy match to toleratetypo, variations, and inclusive match.

309

290
291
292
293
294
295
296
297
298
299
300
301

304
305
306
307
308

310
311
312
313
314
315
316
317
318

4.

319

Evaluation and Results

320

4.1.

321

336

We conduct our evaluation using HotpotQA dataset(Yang et al., 2018). HotpotQA contains two question categories: bridge-type questions, in which
an intermediate entity is needed to be retrieved
before inferring the answer; and comparison-type
questions, which compare two provided entities.
Given the focus of this work, we use solely thebridge-type questions in our evaluation.7 We conduct the evaluation of our proposed methods on the5918 bridge-type questions out of the 7,405 examples from the development partition of HotpotQA
dataset in the distractor setting. Each question in
HotpotQA is supported by two documents, andprovided with ground-truth supporting sentences,which enables us to evaluate the evidence retrieval
performance of the various models.

337

4.2.

338

Table 2 reports the evidence retrieval performanceof all models discussed. All three base models thattarget either semantic equivalence or inference do
not yield optimal performance. As expected, the
MSMARCO CE achieves the highest performance
among the base models, as it is a strong baselinethat is commonly used for retrieval tasks. However, it only considers the semantic matching between question and individual candidate sentences,ignoring the other important relevance matching
characteristics such as exact matching signals, textual entailment, and relatedness between candidate
evidence sentences.
For the baseline ensemble models, AR performs worse than the MSMARCO CE, while beingslightly better than BM25 and the QNLI CE. Its
retrieval performance is essentially a compromiseamong the performances of the three base models,
because it directly averages the individual ranking
results. In contrast, SimCom8 does take advantage
of complementary relevance signals from the basemodels, so to perform better than any of the individual base model. However, it fails to deliver thebest overall performance because it simply combines the final output scores from the base models

322
323
324
325
326
327
328
329
330
331
332
333
334
335

339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362

Models P@3

DatasetBM25 0.43
MSmarco 0.47
QNLI
0.33
AR
0.43
SimCom 0.5
EAR
0.53
EARnest 0.55

P@5 MAP R@3
Base models
0.31
0.59
0.54
0.33
0.64
0.59
0.25
0.46
0.43
Ensemble Baselines
0.31
0.61
0.55
0.36
0.68
0.63
Our Approach
0.36
0.71
0.66
0.38
0.74
0.7

R@5

R@10

0.65
0.69
0.52

0.78
0.81
0.65

0.66
0.74

0.83
0.86

0.76
0.78

0.86
0.87

Table 2: Evidence retrieval results of base models, baseline ensembles, and our methods on HotpotQA. As canbe seen, the performance of our proposed ensemble
methods (EAR and EARnest) are effective for improving the retrieval performance in terms of all the metrics.
Our best model EARnest achieves the highest MAP
performance, outperforming all the base models andensemble baselines.

Results

without exploiting the interactions between the relevance signals behind.
Lastly, our approaches (i.e., EAR and EARnest)
not only outperform the base models, but also exceed the order-based and score-based ensemble
models on all metrics. They both jointly consider
diverse relevance signals simultaneously, and therefore achieve greater improvements on the retrieval
performances. EARnest further considers the relatedness between evidence sentences, becoming
our best model. It achieves the highest MAP, andhigher than the MSMARCO CE by 10%.

5.

Conclusion

In this work, we showed that successful relevance
matching for evidence retrieval in multi-hop QA
requires considering diverse signals including exact matching, semantic textual similarity, and textual entailment between question and candidate
sentences, and relatedness between candidate evidence sentences. We applied off-the-shelf statistical models and transformers to capture different
dimensions of relevance and effectively combined
them to jointly retrieve candidate evidences thatcover diverse and most relevant information for thequestion when concatenated. Experimental resultson HotpotQA reveal that our models are effective
for improving the retrieval performance for multihop questions, comparing to all the single retrieval
models they based on, also the order-based andscore-based ensemble baseline models.

7
On average, comparison-type questions are easier to answer because the necessary information (i.e., the two entities
to be compared) tends to be present in the question.
8
The result of the SimCom model uses α = 3 and β = 1,
which achieves the highest performance according to the gridsearch results on 10% of the full dataset.

5

363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392

393

References

394
395
396
397

Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi,
Richard Socher, and Caiming Xiong. 2020. Learning
to retrieve reasoning paths over wikipedia graph forquestion answering.

398
399
400
401
402
403

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng,
Jianfeng Gao, Xiaodong Liu, Rangan Majumder,Andrew McNamara, Bhaskar Mitra, Tri Nguyen,
et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint
arXiv:1611.09268.

404
405
406

Danqi Chen, Adam Fisch, Jason Weston, and Antoine
Bordes. 2017. Reading wikipedia to answer opendomain questions.

407
408

Zhengdong Lu and Hang Li. 2013. A deep architecture for matching short texts. Advances in neural
information processing systems, 26.

448
449
450

Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Multi-hop reading comprehension through question decomposition and rescoring.

451
452
453
454

Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017.
Learning to match using local and distributed representations of text for web search. In Proceedingsof the 26th International Conference on World Wide
Web, pages 1291–1299.

455
456
457
458
459

Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering.

Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu,
and Xueqi Cheng. 2016. A study of matchpyramid models on ad-hoc retrieval. arXiv preprint
arXiv:1606.04648.

460
461
462
463

409
410
411
412

Ruiliu Fu, Han Wang, Xuejun Zhang, Jun Zhou, andYonghong Yan. 2021. Decomposing complex questions makes multi-hop qa easier and more interpretable. arXiv preprint arXiv:2110.13472.

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun
Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. arXiv
preprint arXiv:2002.09758.

464
465
466
467

413
414
415
416

Luyu Gao, Zhuyun Dai, Zhen Fan, and Jamie Callan.
2020. Complementing lexical retrieval with semantic residual embedding. corr abs/2004.13969 (2020).
arXiv preprint arXiv:2004.13969.

Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, andChristopher D Manning. 2019. Answering complex
open-domain questions through iterative query generation. arXiv preprint arXiv:1910.07000.

468
469
470
471

417
418
419
420
421

Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce
Croft. 2016. A deep relevance matching model for
ad-hoc retrieval. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 55–64.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert:
Sentence embeddings using siamese bert-networks.
arXiv preprint arXiv:1908.10084.

472
473
474

422
423
424

Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrialstrength Natural Language Processing in Python.

Xuan Shan, Chuanjie Liu, Yiqian Xia, Qi Chen,
Yusi Zhang, Angen Luo, and Yuxiang Luo. 2020.
Bison: Bm25-weighted selfattention framework
for multi-fields document search. arXiv preprint
arXiv:2007.05186.

475
476
477
478
479

425
426
427
428
429
430

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng,
Alex Acero, and Larry Heck. 2013. Learning deep
structured semantic models for web search using
clickthrough data. In Proceedings of the 22nd ACM
international conference on Information & Knowledge Management, pages 2333–2338.

Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng,and Grégoire Mesnil. 2014. Learning semantic representations using convolutional neural networks forweb search. In Proceedings of the 23rd international
conference on world wide web, pages 373–374.

480
481
482
483
484

431
432
433

Yichen Jiang and Mohit Bansal. 2019. Self-assembling
modular networks for interpretable multi-hop reasoning. arXiv preprint arXiv:1909.05803.

Alon Talmor and Jonathan Berant. 2018. The web asa knowledge-base for answering complex questions.
arXiv preprint arXiv:1803.06643.

485
486
487

434
435
436
437
438

Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick
Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and
Wen-tau Yih. 2020. Dense passage retrieval for
open-domain question answering. arXiv preprint
arXiv:2004.04906.

Andrew Trotman, Antti Puurula, and Blake Burgess.
2014. Improvements to bm25 and language models
examined. In Proceedings of the 2014 AustralasianDocument Computing Symposium, pages 58–65.

488
489
490
491

439
440
441
442
443

Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, JhengHong Yang, Ronak Pradeep, and Rodrigo Nogueira.
2021a. Pyserini: An easy-to-use python toolkit to
support replicable ir research with sparse and dense
representations. arXiv preprint arXiv:2102.10073.

Alex Wang, Amanpreet Singh, Julian Michael, Felix
Hill, Omer Levy, and Samuel R Bowman. 2018.
Glue: A multi-task benchmark and analysis platform
for natural language understanding. arXiv preprint
arXiv:1804.07461.

492
493
494
495
496

444
445
446
447

Jimmy Lin, Rodrigo Nogueira, and Andrew Yates.
2021b. Pretrained transformers for text ranking: Bert
and beyond. Synthesis Lectures on Human Language
Technologies, 14(4):1–325.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, andChristopher D. Manning. 2018. Hotpotqa: A dataset
for diverse, explainable multi-hop question answering.

497
498
499
500
501

6

A.

502

503

QER of Similarity Combination η(BM25 )+α·η(ST Sq,sjq,sj )+β·η(IS q,sj )







3


QERq,sj =α·η(ST S q,sj )+β·η(IS q,sj )ModelsBM25
MSmarcoQNLI
EARnestEARnest - QNLI

if BM25q,sj >0

Otherwise

2

504
505
506
507
508

where semantic textual similarity (STS) and inference similarity (IS) are scores from MSMARCO
CE and QNLI CE. It first normalizes the scoreswith η 9 , and then combines the normalized scores
using the weights α and β.

B.

509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529

C.

531
532
533
534
535
536
537
538
539
540
541
542

P@5
0.32
0.34
0.24
0.38
0.37

MAP
0.6
0.65
0.45
0.75
0.7

R@3
0.55
0.59
0.43
0.71
0.66

R@5
0.66
0.71
0.52
0.8
0.77

R@10
0.79
0.82
0.64
0.88
0.86

Table 3: With the EARnest ensemble model framework,
we replace QNLI CE with Ms Marco CE, and the retrieval performance significantly decreased. Comparing
to the full EARnest model, MAP drops 5% without
exploting the QNLI CE model to capture the textual
entailment relevance signal.

Impact of K

EAR and EARnest both jointly consider pairs ofcandidate sentences top ranked by the base models.
The cut-off parameter K is used to partition sentences considered as top-ranked by individual basemodel or not. The larger K is, the more exhaustive combination of candidate sentence pairs would
be considered. However, the number of pairs is
quadratic in the number of K, so it becomes much
more computational costly when K is too large.
Thus, we test on 600 randomly sampled questions(about 10% of full dataset) to compare the impact
when changing the value of K. The resulting retrieval performance is exact same when changing
K from 3 to 5, while the number of pairs compared
increases from 12 to 33.5 on average. This is expected, because we only consider the top pair toscored against the question, and sentences in thepairs are often more likely to be ranked closer tothe top of lists by base models respectfully since
they contain stronger relevance signals.

530

P@3
0.44
0.47
0.34
0.56
0.52

than the full EARnest model, which confirms that
textual entailment is a very important relevancesignal to the multi-hop QA evidence retrieval task
and should be considered along with the semantic
equivalence.

Necessity of Inference model

To further demonstrate the benefits brought by theinference model, we conduct an ablation experiment by replacing the QNLI CE in EARnest with
MSMARCO CE while keep everything else thesame. We also compare the difference on retrieval
performance with the randomly sampled 600 questions. The result is shown in table C. Without
the QNLI CE capturing the entailment relation topromote evidences that are can be inferred by thequestions to the top, BM25 and MSMARCO CE
might miss them according to lexical and semantic
matches. Therefore, the result is significantly lower
8

https://pypi.org/project/rank-bm25
η performs normalization to scale inputs to unit norms
with Scikit-learn’s normalizer:https://scikit-learn.
org/stable/modules/generated/sklearn.
preprocessing.Normalizer.html
9

7

543
544
545
546
547