Machine Reading Comprehension using Case-based Reasoning
Authors: Dung Ngoc Thai
Dhruv Agarwal
Mudit Chaudhary
Wenlong Zhao
Rajarshi Das
Jay-Yoon Lee
Hannaneh Hajishirzi
Manzil Zaheer
Andrew McCallum
Venue: EMNLP 2023 Findings (2023)
Keywords: question answering
case-based reasoning

Abstract: We present an accurate and interpretable method for answer extraction in machine reading comprehension that is reminiscent of case-based reasoning (CBR) from classical AI. Our method (CBR-MRC) builds upon the hypothesis that contextualized answers to similar questions share semantic similarities with each other. Given a test question, CBR-MRC first retrieves a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers in the retrieved cases. The semi-parametric nature of our approach allows it to attribute a prediction to the specific set of evidence cases, making it a desirable choice for building reliable and debuggable QA systems. We show that CBR-MRC provides high accuracy comparable with large reader models and outperforms baselines by 11.5 and 8.4 EM on NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability of CBR-MRC in identifying not just the correct answer tokens but also the span with the most relevant supporting evidence. Lastly, we observe that contexts for certain question types show higher lexical diversity than others and find that CBR-MRC is robust to these variations while performance using fully-parametric methods drops.

Machine Reading Comprehension using Case-based Reasoning
Dung Thai', Dhruv Agarwal', Mudit Chaudhary', Wenlong Zhao', Rajarshi Das Manzil Zaheer", Jay- Yoon Lee", Hannaneh Hajishirzi', Andrew McCallum'
Universitty of Massachusetts Amherst
AWS Al Labs, 'Google Research, 'Scoul National University, University of Washington {dthai, dagarwal, mwwaudhary, wenlongzhao, mccal (orcallumm]n @oms umass. edu
Abstract
Test Question and Context
Q: Who invented the computer' A:
We present an accurate and interpretable method for answer extraction in machine read- ing comprehension that is reminiscent of case- based reasoning (CBR) from classical AL. Our method (CBR-MRC) builds upon the hypothe- sis that contextualized answers to similar ques- tions share semantic similarities with each other. Given a test question, CBR-MRC first retrieves a set of similar cases from a non- parametric memory and then predicts answer by selecting the span in the test context that is most similar to contextualized representa- tions of answers in the retrieved cases. The semi- parametric nature of our approach allows it to attribute prediction to the specific set of evidence cases, making desirable choice for building reliable and debuggable QA systems. We show that CBR-MRC provides high accu- racy comparable with large reader models and outperforms baselines by 11.5 and 8.4 EM on NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability of CBR- MRC identifying not just the correct answer tokens but also the span with the most rele- vant supporting evidence. Lastly, we observe that contexts for certain question types show higher lexical diversity than others and find that CBR-MRC robust to these variations while performance using parametri methods drops.
Context: Charles Babbagg
cradited for inwenting the mechanical
sonat datee Raids
canclae booe
Engine, Grndinte Aendeneeeeeeee
eventually
Top-k Case
Who poodtaed inventing the tolograph? David Alter
Context: In 1839 Elderion
David Aller invented the electric telegraph
deoreasing Similarity
Q: Who the inventor of the telephone? A: Graham Beil
Context: Graham credited patenting the first talephone
Ineuar
Context:
Cars
were invented in 1866, whan inventor Kari Bera
Scoring Candidate Spans Using Top-k Case Spans
score
Charles Uabbage
0.9
Score
Computer
0.5
ecoce
Engine
0.2
Figure 1: Overview of CBR-MRC. Top: given test question, CBR-MRC first retrieves similar cases from memory of known (question, context, answer) triples. Bottom: CBR-MRC scores each candidate answer span by comparing its contextualized representation with the answer spans the retrieved cases to output score. The candidate with the highest aggregate score with the answer spans from the cases is output as the prediction ("Charles Babhage").
1 Introduction
Machine reading comprehension (MRC) aims to measure the ability of models to understand and reason over a specified sequence of text. In the ex- tractive setting, the task requires model to answer question by reading set of one referred as the context, and identifying span of text from that context as the answer.
Current state-of-the -f-te-of-the-att machine readers (De- vlin et al., 2019; Choi et al., 2018; Liu et al., 2019; Lewis et al., 2020; Izacard and Grave, 2021) use tune-tuned transformer-based models (Vaswani et al., 2017). However, the prediction mechanisms in these models are largely opaque humans, limit- ing their interpretability and maintainability in real- world scenarios (Ribeiro et al., 2016; Yang et al., 2018; Camburu et al., 2018). This becomes particu- larly relevant when practitioners need to determine
passages,
'Work done while at University Washington.
!Our code is available at https:/ thub. com/dungtn/ cbr-txt

---

the cause of erroneous predictions, for instance. patch models deployed in production systems, Moreover, to build reliable QA systems, is es- sential for models to not only predict the correct answer but also provide evidence to support the predictions (Thayaparan et al., 2020; Zeng et al., 2020).
common MRC datasets, and find our method out- performs strong baselines by up to 11.53 EM. The inference procedure of CBR-MRC is not just accu- rate but also interpretable - we know which cases the model uses for its prediction and how much each case contributes. Further, the semi paramerrttrrii nature of CBR-MRC allows for efficient and accu- rate domain adaptation by simply adding new cases to the casebase without the need modify any pa- rameters." Lastly, we also find that our method shows higher gains compared to fully-parametric models when the training set contains lcxically di- verse contexts for questions with similar latent rela- tions, demonstrating the robustness of CBR-MRC when questions may be answered using several dif- ferent passages.
To address these limitations, we introduce CBR- MRC a remi-parametric approach for MRC that makes predictions by explicitly reasoning over set of retrieved answers for similar questions and their contexts from a non-parametric memory. Our key idea that contextualized answers to similar ques- tions share representational similarities. We posit that a (question, passage, answer) triple captures a set of latent relations, which can be understood to represent the types or intents present within a question given the accompanying context. We use the intersection of latent relation sets between ques- tions learn a similarity function that can then be used to predict answers to questions with similar in- tents. More concretely, our approach first compares the contextualized representations of candidate an- swer spans in the target context with gold answer spans of similar questions from memory. Then, the candidate with the highest aggregate similarity with the gold case answer spans is selected as the predicted answer. For example, in Figure I, the question "Who invented the computer?" shares the same latent relations with the question "Who the inventor of the telephone?"" and the contextualized span similarity with the case answer "Graham Bell" is maximized when compared with the correct an- swer "Charles Babbage" and low when compared with other candidate spans such as "mechanical computer".
In summary, our contributions are as follows:
1. We propose CBR-MRC - novel semi- parametric approach for MRC that outper- forms strong baselines on multiple datasets and settings by up 11.53 EM on Natu- ralQuestions and 8.4 EM on NewsQA.
2. We show that CBR-MRC makes predictions by identifying the correct supporting evidence. When presented with gold and noisy evidence sentences, CBR-MRC predicts the correct an- swer span with 83.2 Span-F1 and 74.6 Span- EM compared to BLANC (Sconwoo et al., 2020a) with 79.0 Span-Fl and 66.8 Span-EM on NaturalQuestions.
3. We show the ability of CBR-MRC for effi- cient and accurate few-shot domain adapta- tion. Compared to a strong baseline (Fried- man et al., 2021), we achieve a 5.5 EM gain on RelEx (Levy et al., 2017) and at-par perfor- mance on BioASQ (Tsatsaronis et al., 2015).
CBR-MRC adapts the case-based reasoning (Schank, 1983) paradigm from elassical AI (Kolod- ner, 1983; Rissland, 1983; Leake, 1996), which typically consists of four steps retrieve, reuse, revise, and retain. CBR-MRC uses the first two retrieval from a memory of cases, or casebase, us- ing the embedding of the test question followed by reusing reasoning patterns encoded in the contextu- alized answer embeddings of the retrieved cases to select the most similar span from the test context as the predicted answer. To the best of our knowledge, we are the first to propose a novel adaptation of the CBR framework for the task of question answer- ing over unstructured text. We empirically verify CBR-MRC on QuestraQQestions and NewsQA, two
4. We show that questions with different sets of latent relations can be expressed with varying degrees of lexical diversity in their accompa- nying contexts and analyze model behavior on such questions. As lexical diversity in- creases, fully-parametric methods drop by up to 15.4 F1, while CBR-MRC shows robust performance with drops of only up 4.1 F1.
"Case augmentation can be extended to make point-fixes for specific inference emors without expensive re-training.

---

Test question and context
Similar question pairs and their (cached) contexts
1. Case retrieval
who Lavented the conputer?
talephone'
telegraph?
K: Grahar Rell
A: David Alter
invencing Eizot conoutes 2532.
necharicol
C: Grahaan Bell is Bredited
Dr. Dovid Alter invented electric telegraph in
practical
Contextualized answer spans for inner-product
Candidate answer span extraction
Graham Ball
David Altar
Charles Babbage
3. Similarity matching and score aggregation to find similar spans
nechanicel conputer
enbaading
Curry antiddry
Cbarles Babbage
Compunat
Figure 2: Inference with CBR-MRC. For a given target query, (1) other similar queries (and their contexts) are retrieved: (2) candidate answer spans are extracted from target context; (3) the candidate spans in the target context are ranked w.r.t. answer spans of the retrieved queries by comparing their contextualized embeddings. Finally, the span with the highest inner-product similarity is selected as the prediction. Note that the casebase questions and their corresponding answer span embeddings pre-computed and cached.
2 Related Work
Case-based Reasoning, Previous work (Das et al., 2020a.b, 2022; Thai al., 2022; Yang al., 2023) has demonstrated strong performance using CBR on knowledge graph reasoning tasks. CBR has also been successfully applied to the task semantic parsing (Das et al., 2021; Pasupat et al., 2021; Awasthi et al., 2023), where methods typi- cally retrieve the logical forms of similar cases and pass them through seq2seg model as input along with target query. Both settings, however, operate over structured spaces, where similarities between instances can easily be determined by collecting and comparing symbolic patterns in the data, e.g., paths between nodes in KG, or syntactic similari- ties in semantic parsing. this work, however, we adapt CBR to operate over unstructured text, and it is not obvious how previous approaches can be re-used to work in this setting out-of-the-box.
Machine Reading Comprehension. MRC evalu- ates machine ability to reason over natural language by training it to answer questions on given pas- sage. Several variations of this task exist, such as cloze-style (Hermann et al., 2015; Yagcioglu et al., 2018), multiple-choice (Richardson et al., 2013;
Lai et al., 2017), extractive (Yang et al., 2015; Trischler al., 2016), and generative question- answering (Bajaj et al., 2016; Kočiský et al., 2017). Our work focuses on the extractive MRC setting, which aims to identify the correct answer span within a passage given test question.
Interpretable methods for MRC have also seen much recent interest, Thayaparan et al. (2020) high- lights the challenges facing MRC in terms of ex- plainability and its impact on MRC performance. Several methods (Yu al., 2014; Min al., 2018; Gravina et al., 2018) have attempted to improve interpretability MRC by identifying the support- ing sentence for the answer. However, these meth- ods rely on the availability of gold-supporting sen- tences, which may not always be present. Other methods utilize graph networks (Ye et al., 2019; Tu et al., 2019) and attention mechanisms to capture the attended parts of text or relations (Sco et al., 2016; Shao et al., 2020). In this work, we improve interpretability by providing a novel inference pro- cedure that allows to clearly attribute predictions to previously seen instances that were used in the decision-making process.
A concurrent work (Iyer ct al., 2023) performs answer sentence selection by using Graph Neu- ral Networks to model interactions among ques- tions and answers, assuming that for each question- answer, there exist similar question-answers in the training data. We instead focus on the more fine- grained task answer span prediction from con- texts and use transformer models to score question- question and answer-answer similarities between each retrieved case and a test example.
In-Context Learning (ICL.. ICL an emer- gent property observed in large language mod- els (LLMs) (Brown et al., 2020; Min et al.,

---

2022) showing remarkable success in several NLP tasks (Cheng et al., 2022; Lewkowycz al., 2022; Dunn et al., 2022). To make predictions, a model simply prompted with demonstrations for a target task and a test instance. Recent work (Rubin et al., 2021; Liu et al., 2021; Li et al., 2023; Khattab et al., 2022) has shown strong performance by retrieving relevant demonstrations. This setting can be under- stood specific instantiation of the case-based reasoning framework, where the "reuse" step uses ICL instead of a fine-tuned model. ICI., however, is sensitive to the prompt (Chen et al., 2022), leading to black-boxed inference and reducing reliability of predictions. Problem decomposition and itera- tive prompting methods, such as sfain-of-thougee (Wei et al., 2022) and least-to-most (Zhou et al., 2022; Drozdov al., 2022), attempt to address this issue by converting complex problems into inter- mediate steps. In contrast to methods reliant on emergent pre-training behavior, our work explicitly encodes the relational similarity hypothesis in a trained model to reuse exemplars for making pre- dictions.
is similar q, if the latent relations expressed in q and qi overlap, i.c., R2 n Rxy * 0. For the above example, similar question may be "Who was the inventor of the television?"" We hypothe- size that these latent relations are captured dense representations obtained from language models; thus, relational overlap can be assessed via vector similarity scores. In experiments, pre-trained BERT (Devlin al., 2019) model encode questions and use the representation of the [CLS] token for similarity calculations.
use a
To ensure that a question representation faith- fully captures the expressed relations and is not in- fluenced by the entities in the question, we replace mentions of all entities with [MASK] token, e.g., "who invented the telephone?"" ?" becomes "who in- vented the [MASK]?" ?" Previous work (Soares al., 2019) has shown this technique be useful in learning entity-independen relational representa- tions. In CBR-MRC, masking removes spurious similarities, where the presence of a common en- tity results in two relationally divergent questions being adjudged as similar, for e.g., "who invented the telephone?" ?" and "what is the telephone used for?" should be considered dissimilar in our setup. In practice, we use T-NER (Ushio and Camacho- Collados, 2021), an automatic entity recognizer, identify mentions of all entities in the question.
3 Method
We now formally describe CBR-MRC (Figure 2). A case in CBR consists of a problem and its solution. Formally, we define a case tuple {q, A. p}. representing question, its corresponding answer set², and the passage context that contains each element in the answer set. The context p is a span of text that not only contains all a A but also captures one or more latent relations between the question and the answer. We refer to collection of all cases as the casebase B, which is typically set to the full training data and may continually be augmented incorporate new data patterns. In the MRC setting, annotations for p for all B assumed be available.
The similarity score between two masked queries is computed as the inner product between their normalized vector representations (cosine sim- ilarity). The representations for all questions in the casebase are pre-computed and cached. During inference, retrieve the top-k similar cases for query by performing a fast nearest-neighbor search over the cached representations.
Case retrieval during training. To minimize the number of retrieved cases with a low overlap in latent relations during training, we additionally em- ploy two filtering heuristics (1) we imum threshold of 0.95 for the cosine similarity score to filter out dissimilar cases, and (2) we use a wh-filter to remove retrieved cases where the question keyword (c.g.. "who", "where", "when") ") present in the case question does not match the keyword present in the input query.
3.1
Case Retrieval
min-
Given an input question q, CRR-MRC retrieves sim- ilar cases from the cascbase, where similarity is de- termined by the overlap of latent relations encoded by the query and the casebase questions. Formally, let R. be the set of latent relations expressed in g. For example, for the question "who invented the telephone?", the latent relation expressed is the concept "invented by" A case CA := Ak, Pk)
3.2
Case Reuse
We hypothesize that contexts associated with simi- lar questions also share relational similarities. For
A is a singleton if every question has a unique answer. 'Note that CBR-MRC does not require explicit annota- tions of question relation types, e.g., mapping to KB schema
relations. It instead leverages such semantics implicitly.

---

instance, Figure 2 shows the context surrounding the answer span of the retrieved case "Graham Bell is credited with patenting... " and the target context surrounding the gold answer "Charles Babbage is credited with inventing..." In CBR-MRC, we use the contextualized embeddings of the answer spans from retrieved cases to find this semantic overlap within the target context. For each candidate an- swer span in the context, similarity score is plicitly computed with answer representations from the retrieved cases.
3.2.1 Training
The passage encoder must be fine-tuned to maxi- mize similarity scores between answer span em- beddings of similar questions. We fine-tune a pre-trained BERT model with a contrastive loss (Khosla et al., 2020) function using answer and non-answer spans.
Specifically, consider a train question 4. its set of answers A,, and its set of candidate spans Sand, S Sp, derived from the passage Pt. We the gold answer set A, as the set of positive candidate spans, and define the set of negative candidate spans as Seand, \ A,, Let sim(:,-) =- Enc(-) )) Enc(-) be the similarity function (dot prod- uct or cosine similarity) between vector representa- tions. For the top-k similar questions as described in the previous section, we fine-tune the passage encoder with the following objective derived from the soft nearest-neighhor loss (Salakhutdinov and Hinton, 2007; Frosst et al., 2019):
For efficient processing, we employ a dense re- triever that uses contextualized answer spans from past cases as queries and searches for the target an- swer over the input passage embeddings. Given an input question and an input passage p. we search for text span E Sy. the set of all possible spans in p, whose representation the most similar to representations of answer spans in similar question contexts. We employ a standard BERT-base model as the passage encoder of the dense retriever. Due to the combinatorial complexity of considering all s E Sp. we follow Seo et al. (2019) and consider only a subset of candidate spans in p likely to be the answer. In particular, we consider set of can- didates Stand C Sp such that Stand is composed of only entity mentions, date-time strings, numbers, and quoted strings that can be extracted from p. To increase span coverage, we include all n-grams up to length three. See A.1 for more details regarding the candidate span extraction process.
L. log Σ
max max (i), agEAr
SEAr
+ log
Σ
max max exp(sim(ax,s) a.EA,
86S 144
where 7 is temperature hyperparameter. The loss for the entire training set is simply the mean of L over all dt E Q, the training set. Among the top-k retrieved cases, we pick the spans with the highest similarity scores. This loss function encourages the similarity scores of the gold answer spans with re- spect to the answers in the top-k cases to be higher than the similarity of other spans of texts in the input context that are not the answer.
We now formulate the task to predict the answer à to the question y from context passage given its relevant past cases. Let C == (u, } be the k-th retrieved case for the given question q. For an answer ag A let Enc(as:) be the contex- tualized answer span embedding pooled from the embeddings of the answer tokens in Pk- Similarly, for all candidate answers C Seand, let Enc(s) be their contextualized embeddings pooled from the encoded tokens in context p. The predicted answer span based on the k-th retrieved case is given by
4 Experiments
We evaluate CBR-MRC on two machine reading comprehension tasks - supervised answer span extraction and few-shot domain adaptation. For three MRC datasets, we use the training data to both construct the casebase as well as train our model and report performance on the in-domain test sets. We report the few-shot domain adaptation performance two additional datasets.
à (k) = arg max max Enc(s) Enn((())) aES
a.C.A.
i.e., the candidate answer span most similar to the answer set A₁ based on similarity scores computed using inner products. At inference, the similarity scores of each candidate span are aggregated across the top-k retrieved cases by taking the maximum scores. The candidate span with the highest aggre- gated score is predicted as the answer.
4.1 Datasets
NaturalQuestions (NaturalQ) consists of over 323K questions and answers, representative of real queries people make on the Google search engine (Kwiatkowski et al., 2019). The answers are sourced from Wikipedia and annotated by

---

crowd workers, including both long and short version. Our experiments utilize short answers as the gold answers and use the long answers as context. NewsQA has 120K crowd-sourced ques- tions hased on CNN news articles (Trischler et al., 2016). Annotation is obtained by two sets of crowd workers: one creating questions based on article highlights and the other answering them using spans from the full article. BioASQ is large- scale biomedical dataset that features question- answer pairs crafted by domain experts (Tsatsa- ronis et al., 2015). Questions are linked to rele- vant scientific literature found on PubMed, with the abstracts of these articles serving as the context for MRC. RelationExtraction (RelEx) comprises question-answer pairs that express Wikidata rela- tions between entities mentioned in Wikipedia arti- cles (Levy al., 2017). The dataset is derived from the WikiReading (Hewlett et al., 2016) slot-filling dataset through predefined templates.
NewsQA
NaturalQ
Model
EM
F1
EM
F1
BERT
50.11
65.07
64.48
76.39
ALBERR
51.18
66.02
63.81
75.89
RoBERTa
52.36
67.28
66.33
78.54
MADE
56.55
72.12
68.86
80.09
SpanBERT
52.85
67.93
66.60
78.31
BLANC
55.52
70.31
68.33
80.04
SpanBERT LARGE
53.84
69.06
69.14
80.66
BLANC
57.40
72.36
70.59
81.99
CBR-MRC
64.95
69.17
82.12
83.10
Table MRC performance on NewsQA and Natu- ralQ. CBR-MRC outperforms all baselines in finding the exact match of the answers. On NewsQA, our model shows +8.4 EM over MADE but has lower FI which indicates CBR-MRC focuses getting the right answer rather than overfitting to the answer tokens.
surrounding context of the answer span (i.e., the supporting evidence), we also report Span-EM and Span-F1, following Seonwoo et al. (2020b), which consider overlapping indices between the predicted and gold answer spans and exclude answers with irrelevant supporting text.
We utilize the curated version of all the previ- ously mentioned datasets from the MRQA shared task (Fisch et al., 2019), including annotations for the gold answer spans within their context. The gold context for constructing case, described in Scction 3, is a short paragraph surrounding the gold answer span taken from the originally pro- vided text. Additionally, note that we use BioASQ and RelEx to only evaluate in the sho- setting and not train our models.
4.3 Results
We report our main MRC results NewsQA and NaturalQ in Table 1. We find that CBR-MRC consistently outperforms all baselines. On Nat- uralQ. CBR-MRC achieves the state-of-the-art, out- performing the next best model (BLANC,ARGR by 11.53 EM. Similarly, on NewsQA. CBR- MRC shows an improvement of 12.56 EM over BLANCLARGE. and 8.4 EM over the next best model (MADE). Note that the best baseline model (BLANCLArGe) has three times the number of pa- rameters as CBR-MRC. Furthermore, our model shows performance gains in EM but a lower Fl which indicates CBR-MRC focuses on getting the correct answer rather than over-fitting to the answer tokens.
4.2 Experiment Setup
We refer the reader to Appendix A.1 for more de- tails on the models used in this work.
Baselines. We compare CBR-MRR MRC to BERT (De- vlin et al., 2019), RoBERTa (Liu al., 2019), AL- BERT (Lan al., 2019), and SpanBERT (Joshi et al., 2020) as well as the SOTA baselines for our settings BLANC (Seonwoo et al., 2020b) and MADE (Friedman et al., 2021).
4.3.1 Supporting Evidence Identification
Evaluation Metries. We report exact match (EM) and Fl scores to measure model performance. EM calculates the percentage of predictions that match all tokens of any one of the correct answers for given question. Fl score measures the token overlap between the prediction and the correct an- swer. As these metries do not take into account the
becoming increasingly important to train mod- els to predict the right answer for the right rea- sons. Since CBR-MRC identifies the answer span by comparing it with multiple answer-containing contexts of similar cases, hypothesize that it can cut through the noise and identify the correct occurrence of the span that exactly answers the question. Our main baseline is BLANC (Seon- woo et al., 2020b), which trains an auxiliary model specifically for context prediction.
*(1) All reported numbers are from single rums on the test set unless noted otherwise. (2) Hyperparameters used by our models were tuned on a development set and the details are included in the appendix.

---

Dataset
Model
Span-EM
Span-F1
100
BERT
60.63
72.92
90
ALBERT
60.31
72.66
RoBERTa
62.59
75.07
80
NaturalQ
SpanBERT
62.71
75.16
70
BLANC
64.57
76.99
SpanBERT LARGE
65,28
77.62
60
CBR-MRC
BLANC
66.75
79.04
BLANC
50
MADE
CBR-MRC
74.64
83.16
40
BERT
45.53
59.18
100
200
300
ALBERT
46.54
60.12
Context Lexical Diversity
RoBERTa
47.43
61.36
Figure 3: Robustness to lexical diversity in passages.
NewsQA
SpunBERT
48.04
62.26
Lexical diversity latent relation clusters the num- ber of unique tokens in passages that latent relation cluster (or question type). We plot Fl performance (averaged over clusterings varying thresholds) 8 buckets of increasing lexical diver- sity for the latent relations seen at training. CBR-MRC shows a drop performance only up 4.10 points, while BLANC and MADE show drops of to 15.38 and 11.80 points, respectively.
BLANC
50.60
64.39
training for
SpunBERTLARC LARGE
49.03
63.43
BLANCLARGE
52.39
66.48
CBR-MRC
54.07
68.21
RoBERTa
60.12
65.99
SpanBERT
57.63
63.47
NaturalQ
BLANC
61.43
67.07
CBR-MRC
71.30
81.04
Table 2: Identifying the correct supporting evidence. We report Span-Fl and Span-EM scores for CBR-MRC and baseline models, indicating how often the span with the correct supporting evidence identified versus se- lecting either the incorrect answer spurious occur- rence of the correct answer. We report baseline numbers for BLANC as reported their paper. NaturalQ* is a containing questions with
two steps implicitly. These models, therefore, must rely only on the parameter values learned at train- ing in order to handle several relational semantics, which may each be expressed in contexts with vary- ing levels of diversity in language form. In practice, we find that some relations are indeed expressed with more lexical diversity than others (Appendix A.3). In Figure 3, investigate how CBR-MRC performs at inference under varying amounts of lexical diversity observed in passage contexts at training. We first cluster questions according to their embeddings and then compute n-gram statis- tics (Li et al., 2015; Dušck et al., 2020; Tevet and Berant, 2020) across passage contexts found in each cluster to measure diversity. We use Natu- ralQuestions as case study for our analysis and compare performance with BLANC and MADE, two fully-parametri baselines.
subset of Natural answers having at least two mentions within the context.
Table 2 highlights the performance gains of CBR- MRC compared to BLANC and several strong base- lines. Our method shows improvements on Natu- ralQ by 7.89 EM and 4.12 Span-F1 points in predicting the span indices. On NewsQA, it im- proves performance by 1.68 Span-EM and 1.73 Span-F1 points compared to BLANC. To deter- mine the gains due to predicting the answer span with the correct supporting evidence, we follow the approach in BLANC and evaluate performance on the NaturalQuestions (NaturalQ*) subset that contains questions with answers that have at least two mentions within the context. Our method out- performs BLANC on this subset by 6.73 and 4.05 points for Span-EM and Spa respectively.
Clustering questions by latent relations. We cluster questions in the training set based on the latent relations they encode using hierarchical agglomerative clustering (HAC) (Krishnamurthy et al., 2012) over similarity scores. For a fair eval- uation, we compute 6 different flat clusterings of varying cluster tightness, i.c., how similar each question is within cluster. For each clustering. compute the per-cluster lexical diversity the count of the number of unique tokens in the set of passage contexts across questions in the cluster, normalized by the cluster size. We then cluster the
4.3.2 Robustness to Lexical Diversity
CBR-MRC performs answer extraction by explic- itly comparing span similarities of the target con- text with the context spans in the filtered casebase. Purely parametric methods can also seen as fol- lowing a similar mechanism, albeit performing the

---

BioASQ
RelEx
Cases (k)
10
Model
EM
F1
EM
82.03
82.12
77.95
77.39
F1
82.13
83.10
78.00
78.45
MADE
43.3
62.9
72.0
84.9
MADEPT
62.5
74.5
82.3
90.0
Table 4: Performance with different number of case retrievals E {1,5. 10, 20 Both EM and Fl display the same trend. Both performance metries increase as k increases from 1 to 5. However, larger sets of retrievals (k > 10) reduce performance to levels worse than even which indicates the introduction of noise.
CBR-MRC
62.4
64.6
87.8
88.2
Table 3: Few-shot domain adaptation. We compare CBR-MRC with two variants of the MADE adapter networks, which are also trained on NaturalQuestions. MADEzs00 demonstrates zero-shot performance, while MADEfr is further fine-tuned 256 samples for each dataset. CBR-MRC is evaluated by simply adding the samples into the casebase without any mun-tuning,
the target dataset. We follow the same practice in (Friedman et al., 2021) and evaluate CBR-MRC on a held-out subset of 400 samples. For both datasets, we use the trained model obtained from the experiment on the NaturalQuestions dataset, and the casebase for each dataset is built from the same training set that MADE also uses.
test set by assigning each question to the cluster label of its top-1 similar train question for each of the 6 clusterings, resulting in 6 test clusterings (see Appendix A.4 for more details).
Table 3 presents the performance of CBR-MRC and MADE adapters. Our model outperforms the zero-shot adapter network (MADEZERO) by simply putting some samples from the target dataset into the casebase. When compared the fine-tuned adapter (MADEfr), our model gains 3.5 EM points on RelEx while being comparable to MADE on BioASQ. These results confirm that CBR-MRC can transfer to new domain by simply collecting a handful of sample cases, which is significantly less expensive than model fine-tuning.
Results. Figure shows the mean F1 scores be- tween CBR-MRC and the baselines on test queries for varying values of lexical diversity observed at training for the same latent relations as the test queries. We group clusters by lexical diversity scores into buckets and report the aggregated Fl scores for each bucket." On average, the perfor- mance gap between CBR-MRC and basclines tends increase lexical diversity increases. The min- max Fl difference, which captures the drop in performance as lexical diversity increases, is 4. 4.10 points for CBR-MRC while BLANC and MADE show significantly larger differences of 15.38 and
We notice drop in Fl scores both datasets. F1 scores evaluate the partial matching of answers. The inference procedure used by previous work, including our baselines, independently predicts the start and end spans of the answer, which encour- ages the inclusion of partial matches. In CBR-MRC, on the other hand, the entire candidate span is con- sidered when making prediction by comparing against the representations of gold spans from the casebase. This results in higher similarity scores for candidate spans that map to the same answer type, rather than matching the correct answer partially, which explains lower Fl scores for CBR-MRC.
11.80 points, respectively. We posit that as di- versity in contexts for the same latent relation in- creases, it is harder for fully parametric models learn parameters that are suitable for all lexical variations. On the other hand, the remi-parametric nature of CBR-MRC allows it to partially shift this burden to test time and ground itself using only a few retrieved cases that are explicitly compared.
4.3.3 Few-shot Domain Adaptation It is important for models to generalize to unseen datasets. We test CBR-MRC trained on NaturalQ on two diverse out-of-domain datasets BioASQ for MRC (Tsatsaronis et al., 2015) and RelEx for relation extraction (Levy et al., 2017). We compare our model with the MADE adapter networks (Fried- man et al., 2021) that are trained on NaturalQues- tions, and further fine-tuned on 256 samples from
4.3.4 Effect of Retrieval Quantity
We explore how the number of retrieved cases k affeets the reading comprehension performance, via varying kt the Natural dataset. We find that while increasing does enrich the set of contexts for referencing the reuse step, too many retrievals may introduce irrelevant cases that degrade performance.
((1) We use k-means over lexical diversity scores of all clusters to construct B = 8 buckets. (2) We report the Fl performance on all 6 clusterings in Appendix A.S.

---

4.3.5 Effect of Modeling
chine Reading Comprehension task. The datasets utilized in this study have been extensively em- ployed in previous research and, as far as our knowledge extends, are not associated with any privacy or ethical concerns.
We analyze CBR-MRC performance with different base models and show that CBR-MRC can improve upon them. We run the experiments on NQ with the same set of hyper-parameters.
Acknowledgements
Model
EM
F1
Span-EM
Span-F1
BERT
64.48
76.39
60.63
72.92
We thank members of UMass IESL for helpful dis- cussions and feedback. This work was supported in part by Amazon Digital Services, in part by IBM Research Al through the Al Horizons Network, in part by the Chan Zuckerberg Initiative under the project "Scientific Knowledge Base Construc- tion", and in part using high-perffrmancee comput- ing equipment obtained under grant from the Collaborative R&D Fund managed by the Mas- sachusetts Technology Collaborative. Any opin- ions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect those of the spon- sor(s).
CBR-MRCAERT
82.12
83.1
74.64
83.16
DeBERTa
43:11
74.07
36.03
63.7
CBR-MRCDMBERTA
80.30
82.22
76.09
80.39
Table 5: Performances of CBR-MRC variants on NQ. We compare two variants of CBR-MRC with BERT and DeBERTa as its base model.
5 Conclusion
We present CBR-MRC, a for machine reading comprehension that sim- ple, accurate, and interpretable. Our model stores a collection of cases, retrieves the most relevant cases for given test question, and then explicitly reuses the reasoning patterns encoded in the embed- dings of these cases to predict an answer. We show that our model performs well for both extracting answers and identifying supporting evidence on several MRC tasks compared to fully-parametric baselines. We also demonstrate the ability of our model to transfer to new domains with limited la- beled data. Finally, we analyze our model under varying conditions of lexical diversity and find that it is robust high lexical diversity, whereas fully- parametric models show a drop in performance.
model
References
Abhijeet Awasthi, Soumen Chakrabarti, and Sunita Sarawagi. 2023. Structured case-based reasoning for inference-time adaptation of text-to-sql parsers. arXiv preprint arXiv:2301.0m 1.000000.........
Payal Bajaj. Daniel Campos. Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. Ms marco: human gener- ated machine reading comprehension dataset.
6 Limitations
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-199...
CBR-MRC, and the CBR framework more gener- ally, relies on the existence of past cases to make predictions for a new problem. This can pose a challenge for composite questions, which require multi-hop reasoning, since the likelihood of enu- merating each algebraic combination of reasoning patterns in the casebase is impractical. Models, thus, may only be able to match a portion of the question, resulting in partially correct reasoning. To address these limitations, future work could explore methods to explicitly encourage such com- positional generalization, such as question decom- position with recursive application of CBR.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018 e-snli: Natu- ral language inference with natural language expla- nations. Advances in Neural Information Processing Systems, 31.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint urXiv:2209.0766:
Zhoujun Cheng. Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et 2022. Binding language models symbolic languages. arXiv preprint arXiv:2210.02875.
Ethics Statement
The objective of our research and the proposed methodology is to enhance performance on the Ma-

---

Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:/808.07036.
Nicholas Frosst, Nicolas Papernot, and Geoffrey E. Hin- ton. 2019. Analyzing and improving representations with the soft nearest neighbor loss. In International Conference on Machine Learning.
Rajarshi Das, Ameya Godbole, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2020a. A simple approach to case-based reasoning in knowl- edge bases. In AKBC.
Alessio Gravina, Federico Rossetto, Silvia Severini, and Giuseppe Attardi. 2018. Cross attention for selection- based question answering. In NLAAI@AI
John Hartigan and Manchek A Wong. 1979. Algo- rithm as 136: A k-means clustering algorithm. Jour- nal of the royal statistical society, series e (applied statistics). 28(1):100- 108.
Rajarshi Das, Ameya Godbole, Nicholas Monath, Manzil Zaheer, and Andrew McCallum. 2020b. Prob- abilistic case-based reasoning for open-world knowl- edge graph completion. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pages 4752-4765, Online. Association for Computa- tional Linguistics.
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend.
Rajarshi Das, Ameya Godbole. Ankita Naik, Elliot Tower, Robin Jia, Manzil Zaheer, Hannaneh Ha- jishirzi, and Andrew McCallum. 2022. Knowledge base question answering by case-based reasoning over subgraphs. In ICML.
Daniel Hewlett, Alexandre Lacoste, Llion Jones. Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. WikiReading: A novel large-scale language understanding task over Wikipedia. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535-1545, Berlin, Germany. Association for Computational Linguistics.
Rajarshi Das. Manzil Zaheer. Dung Ngoc Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum, 2021. Case-based reasoning for natural language queries over knowledge bases. In EMNLP.
Roshni Iyer, Thuy Vu, Alessandro Moschitti, and Yizhou Sun. 2023. Question-answer sentence graph for joint modeling answer selection. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 968-979, Dubrovnik, Croatia. Association for Com- putational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-tras tre-training of deep bidirectional transformers for language understand- ing. In NAACL.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACI.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. arXiv preprint arXiv: 20X.::2099.5000 15003.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. 2022. Struc- tured information extraction from complex scientific text with fine-tuned large language models. arXiv preprint 122.02221..
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-seanchh predict: Composing retrieval and language mod- els for knowledge-intensive nlp. arXiv preprint arXiv:2212. 14024.
Ondfej Dušck, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the-art of end-to-end natural language generation: The e2e nlg challenge. Computer Speech & Language, 59:123-151 156.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. 2020. Su- pervised contrastive learning. Advances in Neural Information Processing Systems, 116116111.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Fun- sol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading compre- hension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP.
Janet L. Kolodner. 1983. Maintaining organization in a dynamic long-term memory. Cognitive Science, 7(4):243-280.
Dan Friedman, Ben Dodge, and Danqi Chen. 2021. Single-dataset experts for multi-dataset qa. In Em- pirical Methods in Natural Language Processing (EMNLP).
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2017. The narrativeqa reading comprehension challenge.

---

Akshay Krishnamurthy, Sivaraman Balakrishnan, Min Xu, and Aarti Singh. 2012. Efficient active algo- rithms for hierarchical clustering. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, page 267-274, Madison, WI, USA. Omnipress.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini urXXvv1907.1
Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe. Mike Lewis, Hanmaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprimt arXiv: 12837.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: benchmark for question answering research. Transactions of the Association of Compu- lational Linguistics.
Sewon Min, Victor Zhong, Richard Socher, and Caim- ing Xiong. 2018. Efficient and robust question an- swering from minimal context over documents.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmen- tation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683-7698, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint (0X.x:1704.0450
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. 2019. ALBERT: A lite BERT for self- supervised learning of language representations. CoRR, ttos/1999.11944
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should trust you?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16 page 1135-1144, New York, NY, USA. Association for Computing Machinery.
David B Leake. 1996. Case-based reasoning: experi- ences, lessons, and future directions.
Omer Levy, Minjoon Seo, Funsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21 Conference on Computational Natural Language Learning [CoNIJ. pages 333-342, Vancouver, Canada. Association for Computational Linguisties.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open moen-domain machine comprehension of text. In Pro- ceedings of the 2013 Conference Empirical Meth- ods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computa- tional Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrievalauugmente generation for knowledge-intensive nlp tasks. In Neurips.
Edwina L Rissland. 1983. Examples in legal reasoning: Legal hypotheticals. In IJCAI, pages 90-93.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:21
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo, al. 2022. Solving quantitative reasoning problems with language models. arXiv preprint arXiv: 21X.:22066.1888...
Ruslan Salakhutdinov and Geoff Hinton. 2007. Learn ing a nonlinear embedding by preserving class neigh- bourhood structure. In Proceedings of the Fleeenth International Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pages 412-419, San Juan, Puerto Rico. PML.R
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A deversity-promotte objec- tive function for neural conversation models. arXiv preprint arXiv: 15 10.03055
Roger C. Schank. 1983. Dynamic Memory: A Theory of Reminding and Learning in Computers and People. Cambridge University Press, USA.
Tianle Li, Xueguang Ma. Alex Zhuang, Yu Gu, Su, and Wenhu Chen. 2023. Few-shot in-context learn- ing for knowledge base question answering. arXiv prepriat 2rXiv:2305.0/750
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiy preprint 200.:200000004.
Minjoon Sco, Jinhyuk Lee, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannanch Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In ACL.

---

Yeon Seonwoo, Ji-Hoon Kim, Jung- Woo Ha, and Al- ice Oh. 2020a. Context-aware answer extraction in question answering. In EMNIPP.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1 19900
Yeon Seonwoo, Ji-Hoon Kim, Jung-Woo Woo Ha, and Al- ice Oh. 2020b. Context-aware answer extraction in question answering. In EMNLP.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis, 2018. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes.
Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is graph structure necessary for multi-hop question answering? In Proceedings of the 2020 Conference Empirical Methods in Natu- ral Language Processing (EMNLP). Association for Computational Linguistics.
Yi Yang. WikiQA: challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 2013-2018, Lisbon, Portugal. As- sociation for Computational Linguistics.
Yih, and Christopher Meek. 2015.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: dataset for diverse, explainable multi-hop question answering, In Proceedings of the 2018 Conference Empiri- cal Methods Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
ACL.
Guy Tevet and Jonathan Berant. 2020. Evaluating the evaluation of diversity natural language generation. arXiv preprint arXiv:2004.02990
Dung Thai. Srinivas Ravishankar, Ibrahim Abdelaziz, Mudit Chaudhary. Nandana Mibiinuukulasooriy,, Tahira Naseem, Rajarshi Das, Pavan Kapanipathi, Achille Fokoue, and Andrew McCallum. 2022. Chr- ikb: A case-based reasoning approach for question answering over incomplete knowledge bases, arXiv preprint Xiix:2224.085555
Zonglin Yang. Xinya Du, Erik Cambria, and Claire Cardie. 2023. End-to-end case-based reasoning for commonsense knowledge base completion. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 3491-3504.
Mokanarangan Thayaparan, Marco Valentino, and An- dré Freitas. 2020. A survey on explainability in ma- chine reading comprehension.
Deming Ye, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, and Maosong Sun. 2019. Multi-paragrahh reasoning with knowledge-enhanced graph neural network.
Adam Trischler, Tong Wang, Xingdi (Eric) Yuan, Justin D. Harris, Alessandro Sordoni, Philip Bach- man, and Kaheer Suleman. 2016. Newsqa: A ma- chine comprehension dataset.
Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection.
George Tsatsaronis. Georgios Balikas, Prodromos Malakasiotis, Joannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, Yannis Almirantis, John Pavlopoulos, Nico- las Baskiotis, Patrick Gallinari, Thierry Artières, Axel-Cyrille Ngonga Ngomo, Norman Heino, Éric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ton Androutsopoulos, and Georgios Paliouras. 2015. An overview of the bioasq large-scale biomedical se- mantic indexing and question answering competition. BMC Bioinformatics, 16.
Changchang Zeng. Shaobo Li, Qin Li, Jie Hu, and Jian- jun Hu. 2020. A survey on machine reading compre- hension: Tasks, evaluation metrics and benchmark datasets.
Denny Zhou, Nathanacl Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Olivier Bousquet, Quoe Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint 2rXiv:2255.1080
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. 2019, Select, an- swer and explain: Interpretable multi-hop reading comprehension over multiple documents.
Asahi Ushio and Jose Camacho-Collados. Collados. 2021. T- NER: An all-round python library for transformer- based named entity recognition. In EACL (demo).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Newrips.

---

A Appendix
consideration for models address. A question in this set may contain up 4 different contexts that can answer the same exact question. In Table 8, we show examples from the test set of questions that can be answered using multiple, diverse contexts.
A.1 Experiments: Implementation Details
All CBR-MRC models employ the BERT-base ar- chitecture and the cased vocabulary. We use fixed set of hyperparameters to train all our models. an Adam optimizer with learning rate of 2e 5, epsilon 1e 8, and max gradient norm value 5.0. We set the number of warm-up steps to be 8% of the training data size. All models are trained for 10 epochs, which takes 2-4 days on a single RTX-8000 GPU each, depend- ing on the size of the dataset. The standard models we report have parameter counts of 108M (BERT, MADE, SpanBERT, BLANC), 17M (ALBERT), 124M (RoBERTa), and the large variants have parameter count of 333M.
More specifically,
A.4 Lexical Diversity Analysis: Clustering questions by latent relations
We cluster questions in the training set based on the latent relations they contain using hierarchical agglomerative clustering (HAC) (Krishnamurthy et al., 2012) over masked similarity scores?. Im- portantly, since we do not have ground-truth la- tent relations for questions, we run our analysis multiple flat clusterings, where each cluster in clustering should represent the same semantic relations. To obtain flat clusters in a fair manner, we cut the HAC tree using C' thresholds, computed using k-means (Hartigan and Wong, 1979) over similarity scores across all questions in the train- ing set. This yields C' clusterings with increasing levels of cluster tightness, i.e., how similar each question is in cluster. We set C' to 6 and lower- bound the similarity scores to 0.9 to reduce noise in the clusters. For each clustering, we then com- pute the per-cluster lexical diversity as the count of the number of unique tokens in the set of passage contexts across questions in the cluster, normalized by the cluster size. Finally, we cluster the test set by assigning each question to the cluster label of its top-1 similar train question for each of the C clusterings, resulting in 6 test clusterings. In our analysis, we retain only unique Aq p) triples re- sulting in 7,213 instances from an original set of 12,836. We additionally drop 98 questions that can- not be answered by any of the three models under consideration, resulting in a final set of 7,115 test questions.
Our approach follows prior QA work (Seo et al., 2019) and seleets a broad set of spans that may be answers to factoid-type questions, such as entity mentions, date-time strings, numbers, and quoted strings. We also include all n-grams up to three words. For date-time string extraction, the datefinda Python package, and for entity mentions, we use the T-NER library (Ushio and Camacho-Collados, 2021).
use
A.2 Experiments: Dataset Statisties
For our experiments, we use the curated versions of NaturalQuestions, NewsQA, BioASQ, and RelEx datasets from the MRQA shared task (Fisch et al., 2019). Table 6 presents the number of examples present in each of the datasets used in the paper.
Dataset
Train
Dev
NaturalQ
104,071
12,836
NewsQA
74,160
4,212
BioASQ
1,504
RelEx
2,948
Table 6: Dataset statistics
A.5 Lexical Diversity Analysis: F1 Performance
A.3
Lexical Diversity Analysis: Distribution in NaturalQuestions
Figure shows the absolute Fl performance CBR-MRC and the two baselines (BLANC and MADE) on each clustering separately. Note that Figure 3 is a reduction from this, in that it takes the average difference between the Fl score of CBR- MRC and the baselines across the 6 clusters.
We evaluate the distribution of lexical diversity in the NaturalQuestions dataset. Out of the set of test questions under consideration in our case study on lexical diversity, we find that 7.46%, or 504, unique (question, answer) pairs contain more than one context, which is used to extract the answer, highlighting the relevance of lexical diversity as a
"We restrict the number of neirest-neighhors to 20 for each question.

---

HAC threshold=0.0
HAC threshold=0.001
HAC threshold=00026
95
95
90
90
85
CBR MRC
85
CBR-MRC
BLANC
Test
BLANC
80
75
75
CBR-MRC MRC
BLANC
70
MADE
65
65
100
200
300
100
200
300
50
100
150
200
250
Lexical Diversity Train
Lexical Diversity in Train
Lexical Diversity Train
HAC threshold=0.05
HAC threshold=0.00m 0.084
HAC threshold=0.1
90
90
Test F1
80
Nest
70
CBR-MRC
CBR-MRC BLANC
BLANC
BLANC
MADE
MADE
MADE
100 Lexical Diversity in Train
100
200
100
200
Lexical Diversity in Train
Lexical Diversity in Train
Figure 4: Fl performance v/s lexical diversity in passage contexts clusterings with decreasing levels of cluster-tightness. We cluster questions by latent relations in the training set using HAC using cut thresholds and assign each test question to one these train clusters. We then bucket clusters by lexical diversity scores compute Fl performance of each bucket for each of the 6 clusterings shown.
A.6 Does CBR-MRC have the correct inductive bias?
Text representations obtained from pre-trained lan- guage models have shown the ability to capture the latent semanties of text spans. In this experi- ment, evaluate CBR-MRC when the parameters are set to the pre-trained BERT-base model with- out fine-tuning. As shown in Table 7, we find that our model with no fine-tuning performs surpris- ingly well, indicating that CBR-MRC does have the correct inductive bias. However, fine-tuning with our proposed contrastive loss leads to significant increase in performance.
Model
NewsQA
NaturalQ
BERT
50.11
64.48
SpanBERT TER-MRCGPE-TRANEE
52.85
66.60
18.63
46.07
CER-MRCFINETTNEED 64.95
82.12
Table 7: Pre-trained CBR-MRC performance. We report the performance (EM) CBR-MRC using both pre-trained passage encoder as well encoder fine- tuned with the contrastive loss described in $3. We additionally include scores from reasonable, finette baselines for comparison.

---

Question
when does scason 5 of ruby come
October 14, 2017
[..] Four scasons. referred to as Volumes" ", have been released with fifth careadl ongoing since its premiere October 2012017. As
[...I Volume premiered
2017
amsounced
the RTX Austin 2017 event. The episodes [..]]
[..] <Table> <T> <Th colspan='2'> Season </Th>> <Th colspan= 22"> Episodes </Th>> <Th colspan= "2'> Originally aired </h>>> [..] colspan= <fTd> <Td colspan= 14
</Td> <Td colspan= I'> October 14, 2017 2017 10- 14 </Td> <Id> [...]
who of the declaration indepen- dence
Thomas Jefferson
Committee of Five draft declaration, consisting of John Adams of Massachusetts. Renjamin Franklin of Pennsylvania Thhomas Jefferson Virginia. Robert R. Livingston of New York. and Roger Sherman of Connecticut. The committee [...]
[...] John Adams leader pushing independence. had persuaded the committee to select Thomas Jefferson to compose the original draft of the document which Congress edited to produce the final version [...]
The copy used for this printing has been lost and may have been capy in Thomas Jefferson 's hand Jefferson original draft is preserved at the Library Congress, complete with changes made John Adams [...]
who plays mom tv show mom
[-.]] working Bonnie Plunkett Allison Junney is also recovering drug and alcohol addict. Christy daughter [..]
attending Alcoholics Anonymous meetings
Junney
[..]] stars Anna Faris Allison Janney lead roles dysfunctional daughter mother duo Christy and Bonnie Plunkett [...]
[-..] her relationship with her mother Bonnie [..] <lib Allison Janney as Bonnie Plunkett: Christy 's mother. joyful cynical recovering addict now grateful tries [...]
depart- ment of bomeland secu- rity created
November 2002
<T>> <Th> Formed </Tie> <Td> November 2002 years ago 2002 </Td>> </>
colspan="2" Ageney overview </Th> <Tr>>> <T>> <Th> Formed </The <Td> November 25 2002 15 years ago 2002- -1 >//Td>>>> [..]
[...] The Department of Homeland Security was established on November 2002 by the Security
intended consolidate U.S. [...]
who has won the euro- vision song contest the most times
Ireland
[...] Ireland has finished first seven times, more than any other country [...]
[-.] <Tablss <Tr> <hh> Wins </Th>> <ho Country </Th> <Th> Years </Th>> </T>>><<T <Tr>
<Ti> 7 </Ti> <Td> Ireland </Ti> <Td> 1970, 1980, 1987. 1992. 1993, 1994. 1996 </Td> </Tr> <Tr> <Tds </Td> <Td> Sweden </Td> <Td> 1974, 1984, 1991, 1999.
2012, 2015 </Td> <TT <T>> <Td>
The country with the highest number of wins is Ireland, with The only person have won more than once [...]
Table Lexical Diversity in NaturalQuestions. We show here unique question-anseet -nswerr pairs that can answered using diverse contexts.


Reviews:
[{"review_id": "jFxGQtYLtA", "reviewer": "Reviewer_paQh", "summary": "", "questions": "How to extract the candidate answer span? Using existing parsing tool?\nHave there been any statistics about the accuracy of the question that the answer is in the answer of similar question pairs and not in the answer? If no similar answers exist in the fields used, is this method still very useful?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review! We appreciate that you found our work novel (echoed by 2R1W and Crp1), well-written (echoed by 2R1W), applicable in multiple settings, and interpretable (echoed by 2R1W and Crp1). Please find our detailed responses to your comments and questions below:\n\n> **Comment**: *I am not very sure about the most essential reason for the success of this method. Whether the golden answer appears in the retrieved answers is what makes the method effective.*\n\n**Response**: The main hypothesis underlying CBR-MRC is that *answers to similar questions share semantic similarities*. We take a semi-parametric approach to explicitly reason over a set of answer spans of questions similar to the test question, where similarity is computed using contextualized representations from a language model. We, thus, reduce the task of extractive QA to the selection of a span from the test passage that is most similar to the answer spans of similar questions. Crucially, we use *representations* of the retrieved answers to select the answer span for the test question and, therefore, **CBR-MRC does not require the gold answer to exist in the retrieved set**. \n\n---\n\n> **Comment**: *I think it would be better if more ablation studies can be conducted for a more solid conclusion: 1. Concatenating samples in training sets and original input to predict without reranking answers with similarity scores.*\n\n**Response**: The idea to augment the input with cases has indeed been explored in prior QA works, such as in KBQA [1], semantic parsing [2], and commonsense KBC [3]. While these methods similarly make use of a set of retrievals, the reasoning mechanism they employ is fully parametric and thus black-boxed. Our method takes a substantially different approach to using retrievals by providing an explicit algorithm for span selection that is transparent and interpretable. However, exploring a parametric method of combining retrievals to make predictions for MRC is a very interesting future direction that we could explore. Thank you for the suggestion!\n\n---\n\n> **Comment**: *2. Calculate the correlation between prediction accuracy and the answer overlap of retrieved answers and golden answers. I assume if the golden answer does not exist in retrieved answers the performance will decrease heavily.*\n\n> **Question**: *Have there been any statistics about the accuracy of the question that the answer is in the answer of similar question pairs and not in the answer? If no similar answers exist in the fields used, is this method still very useful?*\n\n**Response**: We briefly discuss the NQ dataset as evidence for our claim that the gold answer need not be in the retrievals. In NQ, there are 8,061 (~63%) test questions **with answers that are not seen in the training set**. The accuracy of our method on this subset is 71.8% (+16.7% compared to BLANC). In further analysis on CBR-MRC’s retrieve step, we find that only 18 out of 12,836 test questions (0.14%) in fact have the gold answer appearing in the answer set of the retrieved cases. These numbers verify that our method does not need the gold answer to appear within the retrieved cases. \n\n---\n\n> **Question**: *How to extract the candidate answer span? Using existing parsing tool?*\n\n**Response**: Our approach follows prior QA work [4] and selects a broad set of spans that may be answers to factoid-type questions, such as entity mentions, date-time strings, numbers, and quoted strings. We also include all n-grams up to three words. For date-time string extraction, we use the `datefinder` Python package, and for entity mentions, we use the T-NER library [5].\nWe present this discussion in L313-322 of Section 3.2 (“Case Reuse”), and will include these additional details in the camera-ready version. \n\n---\n\n**Missing References**: Thank you for pointing these out. We will add them in our final version.\n\n---\n\n[1] Das, R., Zaheer, M., Thai, D., Godbole, A., Perez, E., Lee, J. Y., Tan, L., Polymenakos, L., & McCallum, A. (2021). Case-based Reasoning for Natural Language Queries over Knowledge Bases. In EMNLP 2021\n\n[2] Pasupat, P., Zhang, Y., & Guu, K. (2021). Controllable Semantic Parsing via Retrieval Augmentation. In EMNLP 2021\n\n[3] Yang, Z., Du, X., Cambria, E., & Cardie, C. (2023). End-to-end Case-Based Reasoning for Commonsense Knowledge Base Completion. In EACL 2023\n\n[4] Seo, M., Lee, J., Kwiatkowski, T., Parikh, A. P., Farhadi, A., & Hajishirzi, H. (2019). Real-time open-domain question answering with dense-sparse phrase index. In ACL 2019.\n\n[5] Ushio, Asahi, and Jose Camacho-Collados. \"T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition.\" In EACL 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is novel in MRC. The authors propose a new way to retrieve questions that can ease spurious similarity and use a more explainable way to rerank retrieved answers instead of directly concatenating them together.\n2. The paper is well-organized and easy to read. \n3. The method is straightforward and works well in different settings. ", "reasons_to_reject": "I am not very sure about the most essential reason for the success of this method. Whether the golden answer appears in the retrieved answers is what makes the method effective. I think it would be better if more ablation studies can be conducted for a more solid conclusion:\n1. Concatenating samples in training sets and original input to predict without reranking answers with similarity scores.\n2. Calculate the correlation between prediction accuracy and the answer overlap of retrieved answers and golden answers. I assume if the golden answer does not exist in retrieved answers the performance will decrease heavily.", "questions_to_authors": "How to extract the candidate answer span? Using existing parsing tool?\nHave there been any statistics about the accuracy of the question that the answer is in the answer of similar question pairs and not in the answer? If no similar answers exist in the fields used, is this method still very useful?", "paper_topic_and_main_contributions": "This work proposes a novel approach using similar question-answer-context triples in training sets as context to solve machine reading comprehension, which is motivated by similar questions that can attribute to specific predictions. The authors use [cls] of BERT to calculate the similarity of questions. They mask the entities in question to remove spurious similarity caused by entities. Then they rerank the candidate answers according to their similarity scores with retrieved similar answers. The experiments are conducted on four MRC datasets and achieve the SOTA performance. The results also show the effectiveness of the few-shot setting. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 7, Line 487: albeit --> ALBERT\nSuggest sharing the source code."}, "extra_text": {"Missing_References": "1. Gao, Tianyu, Adam Fisch, and Danqi Chen. \"Making pre-trained language models better few-shot learners.\" arXiv preprint arXiv:2012.15723 (2020).\n2. Lewis, Patrick, Pontus Stenetorp, and Sebastian Riedel. \"Question and answer test-train overlap in open-domain question answering datasets.\" arXiv preprint arXiv:2008.02637 (2020).\n3. Wang, Cunxiang, Pai Liu, and Yue Zhang. \"Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n4. Zhang, Hongxin, et al. \"Robustness of Demonstration-based Learning Under Limited Data Scenario.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022."}}, {"review_id": "SnUsI0CDWq", "reviewer": "Reviewer_2R1W", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review! We appreciate that you found our work novel (echoed by paQh and Crp1), well-written (echoed by paQh), interpretable (echoed by paQh and Crp1), and generalizable to other tasks. Please find our detailed responses to your comments and questions below:\n\n> **Comment:** *(main) All the experiments using the proposed work use BERT-base encoder only.* \n\n**Response:** We acknowledge your valid concern and present new results with one of the suggested models (we will be sure to include more models in the camera-ready version). The following are the results of experiments using DeBERTa as the base model for CBR-MRC and for the baseline on the NQ dataset:\n\n| Score | CBR-MRC w/ DeBERTa  | DeBERTa baseline|\n|:---:|:---:|:---:|\n|EM| 0.8030 | 0.4311|\n|F1| 0.8222 | 0.7407|\n|Precision| 0.8625 |  0.8410 |\n|Recall| 0.8254| 0.7182|\n|Span EM| 0.7609| 0.0036|\n|Span F1| 0.8039 | 0.6370 |\n|Span Precision| 0.8450 | 0.8037 |\n|Span Recall| 0.8050| 0.5679|\n\nThese results confirm that our method similarly demonstrates strong empirical performance with more recent models. We will add these and a more expanded set of results with a selection of newer models in the final version of the paper. Thank you for raising this concern!\n\n---\n\n> **Comment:** *(main) Unclear if the proposed method is truly much better than baselines.*\n\n**Response:** We argue that vanilla EM and F1 are less sound metrics for MRC since they do not distinguish between cases when a model predicts the right answer tokens in the right v/s the wrong context. That is, a model may obtain high F1 but never produce answers for the right reasons. We, therefore, report Span-EM and Span-F1 (Table 2), which require a model to predict not only the correct answer tokens, but also their right occurrences in the passages, thus eliminating spurious hits. In our experiments, we find that CBR-MRC consistently outperforms all baselines on both Span-EM and Span-F1 on this stricter evaluation. We do also report EM and F1 (Table 1) due to their common usage in the literature, and observe that CBR-MRC based on BERT-base (1) outperforms not only BERT-base but also SpanBERT-large, and (2) is surpassed only in the evaluation of NewsQA F1. \n\nThe following examples illustrate the weakness of the vanilla F1 metric, where the baseline BLANC model is given partial credit despite predicting incorrect spans due to overlapping tokens with the ground truth answer; crucially, Span-F1 is robust to such cases. \n\n```\nquestion: where does the term jack mormon come from\ngroundtruth: “jackson county , missouri”\nprediction: “missouri , during the kirtland period of latter day saint history , circa 1834”\n```\n\n```\nquestion: which river separates the bronx in new york city from manhattan island\ngroundtruth: “harlem river”\nprediction: “the hudson river”\n```\n\nIn summary, on the more interpretable and challenging metrics of Span-EM and Span-F1 (Table 2), CBR-MRC consistently achieves state-of-the-art performance and exhibits both accuracy and better reliability than baseline methods.\n\n---\n\n> **Comment:** *(minor) There are many other RC datasets available, which would be easy to apply this method on.*\n\nThank you for the suggestion! Our decision to prioritize the evaluated datasets was motivated by the available room for improvement based on prior art. We, therefore, did not evaluate on datasets such as SQuAD, which are near saturation (>93% EM). However, we will be sure to include a selection of a few more challenging datasets in the camera-ready version.\n\n---\n\n> **Comment:** *(not weakness, suggestion) Some discussion of how this relates to nearest-neighbor models would be helpful.*\n\n**Response:** Thank you for the helpful suggestion! A canonical work that uses nearest-neighbor retrieval for question answering, probably also the paper being referred to in the review, is PAQ [1]. Here, for a test question, the most similar question from a large set of synthetically generated questions is retrieved and the answer corresponding to the retrieved question is simply output as the prediction. In contrast, our CBR-MRC method differs in the following crucial ways - (a) we retrieve not just one but a set of questions that are jointly reasoned over, (b) the retrieval aims to select questions with the same reasoning patterns but not necessarily the same entities as the test question, and (c) the actual answer tokens of the retrieved questions are not used as the prediction - we instead find the spans in the accompanying context/passage of the test question whose neural representations are the most similar to the representations of the answers of the retrieved questions. Crucially, nearest-neighbor methods, such as PAQ, retrieve questions and output their labels as predictions, and thus perform no reasoning. In contrast, our method uses an explicit reasoning mechanism over the retrieved set of QA pairs. \n\nWe will be sure to include this discussion in the final version of the paper. Please let us know in case you had a different work in mind - we would be happy to add more discussion. \n\n---\n\n[1] Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. \"PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them.\" 2021. arXiv preprint arXiv:2102.07033", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work introduces quite a novel approach to RC QA. The ideas beind this work can potentially be applied to other tasks as well.\n- The interpretability and no-training extensibility features of this approach are particularly appealing.\n- The proposed models gets good results on the datasets even using representations from small model like BERT.\n- The paper is well written.", "reasons_to_reject": "__(main) All the experiments using the proposed work use BERT-base encoder only.__\n\nBERT models are over 5 years old at this point and even among that, the model used BERT-base is a weaker one. Although the demonstrated gains are large, it's not clear if these gains would hold up if one uses the current generation of models. Using at least somewhat newer and larger models like T5 or DeBERTa would have significantly improved my confidence.\n\n\n__(main) Unclear if the proposed method is truly much better than baselines.__\n\nThe proposed method performs much better on EM (8-12) points but is within ~2 points (better or worse) on F1. For span-based evaluation where the same entity can occur multiple times referred to in different ways (President Obama vs Barack Obama), F1 seems a better metric than EM. F1 of course also has flaws though. Given the difference in the result is so large, it would be useful to manually evaluate and compare ~100 predictions or so by different systems.\n\n__(minor) There are many other RC datasets available, which would be easy to apply this method on.__\n\nIt would have been better to see the results on a couple of more datasets, at least common ones like Squad.\n\n__(not weakness, suggestion) Some discussion of how this relates to nearest-neighbor models would be helpful.__\n\nAlthough I'm not familiar with this line of work, and I cannot find paper right now, I think I've seen works where they find the best matching instance for the test instance from the training data and use its classification label for the test instance. I think I've only seen it for classification tasks though.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work introduces a new semi-parametric method for span-based reading comprehension QA where for the test instance, they find best matching K instances from the training data, and use alignment between the representations to predict the answer. Concretely, it works in the following way:\n\n- Make a collection of (passage, entity-masked question, answer) tuples from training data. For the test question, mask the entities and find the top K training tuples by comparing test and train entity-masked questions. Then extract BERT representations for candidate spans from the passage (all spans containing an entity) of the test instance, and find the best matching candidate span from all top K training tuples' passages. The span in the test instance passage that matches the best is the final answer.\n- For top K similarity function to work, the authors train use contrastive loss for ranking answer spans higher than non-answer spans.\n\nThis approach has 2 nice features:\n\n1. It provides additional interpretability as one can look at the training instances model uses to make the prediction.\n2. It provides a way for the model to use new training questions without having the need to retraining on it, like standard RC model pipelines.\n\nInterestingly, this approach also performs much better and shows improved robustness to lexical changes than baselines on NewsQA and NaturalQuestions.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8gXyK4VwAz", "reviewer": "Reviewer_Crp1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review! We appreciate that you found our work novel (echoed by paQh and 2R1W), accurate, interpretable (echoed by paQh and 2R1W), and robust. Please find our detailed responses to your comments and questions below:\n\n> **Comment:** *The proposed system might be too complex to use in real practice. The efficiency of the case retrieval and many heuristics construction might hinder the method from being scaled up. Further scalability analysis might be needed.*\n\n**Response:** Our proposed approach is a sum of extremely lightweight parts. In particular, our case retrieval uses an efficient dense retriever with a small index size of around **33MB**, providing retrieval times of **only ~2.6ms**. NER for question masking and answer span extraction takes about **only ~11.8ms**. Overall, our CBR-MRC inference procedure takes **~41ms**, which is almost identical to the ~38ms latency of the baseline method, BLANC. We report all latency numbers averaged over 12,836 questions from the NQ test set.\n\n---\n\n> **Comment:** *The case retrieval may make the model focus too much on the absolute answer span position while learning the general reasoning patterns, which makes the model hard to generalize.*\n\n**Response:** In CBR-MRC, the answer is predicted by reasoning over the contextualized representations of answer spans of similar questions. While positional information is encoded in these representations, we argue that the predominant information stored in these vectors are the semantics of the text. We, therefore, claim that **CBR-MRC does not rely on the absolute answer span positions to make predictions**. We quantitatively verify our claim by running the following analysis: \n\nFor each test sample in the NQ dataset, we obtain (1) the absolute start and end token positions of CBR-MRC’s predicted answer span, and (2) the start and end token positions of the answer span from the retrieved cases (the one that has the highest similarity with a candidate span in the test context as described in L334). Then, we compute the Spearman and Pearson coefficients between the start positions of the prediction and retrieval (we repeat this calculation for the end positions). The following are the results:\n\n| | Spearman correlation | Pearson correlation|\n|:---:|:---:|:---:|\n|Start positions of answer spans|  0.2873 |  0.3056|\n|End positions of answer spans| 0.2796| 0.2091|\n\nWe observe that for both the start and end position, a weak correlation is indicated by both metrics between the answer span positions of CBR-MRC's predictions and those of the retrieved cases. This confirms that our model does not overfit to absolute positions seen in the retrievals.\n\n---\n\n> **Comment:** *For example, in the main results, the model shows performance gain in EM but a lower F1 score. the author explains this by saying \"The model focuses on getting the written answer rather than over-fitting to the answer tokens.\" However, I feel such a case retrieval will lead to more over-fitting.*\n\nEM and F1 are less informative metrics and sometimes misleading since they do not distinguish cases when a model predicts the right answer tokens in the right v/s the wrong contexts. That is, a model may obtain high EM and F1, but never produce the answer tokens with the correct supporting evidence or for the right reasons. Nonetheless, we report EM and F1 (Table 1) due to their common usage in the literature and observe that CBR-MRC based on BERT-base (1) outperforms not only the baseline BERT-base but also SpanBERT-large, and (2) is surpassed only in the evaluation of NewsQA F1. Notably, on the stricter evaluation setting of Span-EM and Span-F1, CBR-MRC consistently outperforms all baselines, demonstrating its accuracy and reliability. \n\n---\n\n> **Comment:** *Impact of the casebase is unclear. The method utilizes training data as the casebase for retrieval. It is unclear how that casebase influences the performance of the model. And whether the utilization of training data as the casebase would lead to over-fitting?*\n\n**Response:** In our experiments, we find that using the training data as the casebase consistently helps test performance and does *not* cause overfitting. Note that our case retrieval is based on similarities in question *types*, and CBR-MRC is trained to be able to reuse generalizable reasoning patterns in the cases but not their case-specific entity information.\n\nOur choice to use the training data as the casebase in all experiments was made in order to construct a fair evaluation setting, since the baseline methods have access to the full training data as well. For in-domain evaluation on held-out test sets, we show in Table 1 and 2 that CBR-MRC (based on BERT) outperforms the BERT baseline and does not exhibit overfitting to the training data. Further, for few-shot domain adaptation, results in Section 4.3.3 demonstrate that despite using the NQ training data in the casebase, CBR-MRC is able to generalize well to a new domain by simply augmenting the casebase with a few target domain examples without any parameter updates. As for zero-shot domain adaptation, where no data from a new domain is available, we argue that this setting is similarly challenging for both fully-parametric models and CBR-MRC. Building out-of-domain generalization capabilities using our model represents an exciting avenue for future work.\n\nIn summary, using the training data as the casebase does not have an adverse effect on the generalization abilities of the proposed CBR-MRC method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) A new method for achieving high accuracy on MRC tasks.\n2) The proposed method can not only achieve better accuracy but also can identify the span with the most relevant supporting evidence, which provides the use of similar cases provides interpretability.\n3) The experimental results show the method's improvement on MRC datasets and the model is able to handle variations in question phrasing and still identify the correct answer.", "reasons_to_reject": "1) The proposed system might be too complex to use in real practice. The efficiency of the case retrieval and many heuristics construction might hinder the method from being scaled up. Further scalability analysis might be needed.\n2) The case retrieval may make the model focus too much on the absolute answer span position while learning the general reasoning patterns, which makes the model hard to generalize.  For example, in the main results, the model shows performance gain in EM but a lower F1 score. the author explains this by saying \"The model focuses on getting the written answer rather than over-fitting to the answer tokens.\" However, I feel such a case retrieval will lead to more over-fitting. \n3) Impact of the casebase is unclear. The method utilizes training data as the casebase for retrieval. It is unclear how that casebase influences the performance of the model. And whether the utilization of training data as the casebase would lead to over-fitting?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies machine reading comprehension (MRC) and proposes a method called Case-based Reasoning for MRC (CBR-MRC).  The main contribution of the paper is the introduction of a semi-parametric approach for MRC that makes a prediction by explicitly reasoning over a set of retrieved answers for similar questions and their contexts from memory. The paper shows that CBR-MRC consistently outperforms all baselines on two benchmark datasets, NewsQA and NaturalQ, achieving state-of-the-art results. The paper also demonstrates that CBR-MRC can identify not just the correct answer tokens, but also the most relevant supporting span.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a case based reasoning approach to reading comprehension question answering. They use BERT embeddings to identify similar questions in the training data to a given test example, and use these to help predict the final answer. The reviewers found the approach presented to be novel for these sorts of QA tasks, and that the paper was well-written. The results were also strong, and during rebuttal the authors provided additional results with DeBERTa that strengthened their contribution.

Unfortunately, the authors missed the following paper, which seems to be a generalization of their work:

Roshni G. Iyer, Thuy Vu, Alessandro Moschitti, and Yizhou Sun. Question-Answer Sentence Graph for Joint Modeling Answer Selection. The 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023)
Decision: Accept-Findings