RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering
Authors: Victor Zhong
Weijia Shi
Wen-tau Yih
Luke Zettlemoyer
Venue: EMNLP 2023 Findings (2023)
Keywords: Question answering

Abstract: We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zeroshot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.

RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question
Answering
Victor Zhong
Weijia Shi
Scott Wen-tau Yih
Luke Zettlemoyer
University of Washington
Meta AI (vzhong, swj0419, www./iggtsw.isnooooooooo edu
scottyihêmeta. com
Abstract
We introduce RoMQA, benchmark for Robust, Multi-evidence, multi-answer QA, that explicitly evaluates for robustness to small question pertur- bations. RoMQA, shown in Figure 1, differs from previous work in number of ways.
We introduce RoMQA, the first benehmark for robust, multi-evidence, multi-answer ques- tion answering (QA). RoMQA contains clus- of questions that are derived from re- lated constraints mined from the Wikidata knowledge graph. RoMQA evaluates robust- ness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human- wwrittnn questions that require reasoning over more ev- idence text and have, on average, many more correct answers. In addition, human annota- tors rate RoMQA questions as more natural likely be asked by people. We evalu- ate state-of-tte-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zero- shot and few-shot models perform similarly naive baselines, while supervised retrieval methods perform well below gold evidence up- per bounds. Moreover, existing models are not robust variations in question constraints, but can be made more robust by tuning on clus- ters of related questions. Our results show that RoMQA
Evaluates robustness to constraint variations. RoMQA contains clusters of related questions that are used measure robustness to varying implicit question constraints, For cach cluster, we compute a robustness score that the minimum score over the questions it contains. In order to perform well on RoMQA robustness evaluation, model must be able to understand many different combinations of the implicit constraints that define the cluster, such as what it means to be pianist, to be born in Paris, and to play Western classical music. To our knowledge, RoMQA the first QA benchmark that evaluates this type of robustness.
More complex questions. Natural questions of- ten have many answers and cannot be answered from a single text. When compared to existing datasets, RoMQA questions have more answers (mean 108.6, median 11, as shown in Figure 2), cover a wider range of diverse topics (as depicted in Figure 3), and necessitate a higher amount of ev- idence text (mean 41.6, median 24). RoMQA also contains entity-linked, relation-eat eextranee text that provide provenance for constraints, showing that the questions are answerable with multi-evidence reasoning from the text corpus.
challenging benchmark for large language models, and provides a quantifi- able test to build more robust QA methods.
1 Introduction
A high quality compositional question answering (QA) model should exhibit robustness to subtle variations in the underlying meaning of input ques- tions. For exmaple, consider the question "which pianists born in Paris play Western classical music?" To show robust understanding, QA model should not only be able to correctly answer this direct ques- tion, but also wide range of related queries that differ in only few constraints (e.g. who was pianist born in Paris?, who was Western classical pianist, not born in Paris?). Prior compositional QA datasets do not evaluate the robustness of QA models to variations in question constraints.
More natural human written questions. Con- trary to previous multi-answer compositional QA datasets, RoMQA provides significantly larger pool of 28k human-written questions. This rep- resents ten-fold increase when compared the previous leading dataset QAMParl (Amouyal al., 2022)), which features a mere 2k human-written questions. Human evaluations show that ques- tions in RoMQA are more natural, as gauged by

---

Question
Implicit constralints
Example evidence
Answers
Which planists play Western clossical music?
occupation pianist subj bern_in in Paris subj
Lily Maisky foorn July Poris) classical plantst.
Liy Musky Queffélec
Queffélec (bom
January 1948) French classical planist, born
genra weskern, classical subj
Who was pianist Paris?
occupasion pianist subj sub1
October 27, 2004) was particularly for his advocacc cellucyry mueie...Helller was born in Paris, and began piano lessons at the age of five and from the age of ten until the outbreak of World War he studied with Robert Casadesus...
(June
parist nated
Amy Claude Hellfer
Who was Westem clessical mucic pianist, not born Paris?
genne westernuc cossnntstscci subj accupatiin plantst sub] borngin Pariis subj
David Fray (born 24 May 1981) is French classical planist... Fray was born in Tarbes, near the Pyreniess
David Fray André Waits
André Walts (bom June 20, 1946) is an American classical pianist and professor the Jacoba School Music indlann University... in Nunembergg Germany,
son Hungarian mather...
Figure I: A cluster of related questions, implicit constraints, evidence text, and answers from RoMQA. Within a RoMQA cluster, related questions differ in implicit constraints. In addition to evaluating model performance across questions, RoMQA evaluates robustness to variations in question constraints by scoring worst-case performance among related questions.
how likely person is to ask the question (Fig- ure 4). Qualitatively, RoMQA questions contain fewer overly precise constraints, unusual attribute comparisons, and overly large numbers of referen- tial hops.
2 RoMQA
We describe RoMQA construction and how it dif- fers from prior compositional QA datasets.
2.1 Dataset construction
RoMQA construction has three goals. First, we want diverse selection of question topics. Sec- ond, these questions should require reasoning multiple pieces of evidence. Third, need understand what implicit constraints the questions contain order to evaluate robustness varying constraints. At a high level, RoMQA construction involves 1) sampling constraints from knowledge base (KB) triples, 2) clustering related constraints, 3) sampling implicit constraints that form logical queries, and 4) annotating language questions.
We evaluate state-of-the-att large language mod- els (LMs) on RoMQA in zero-shot prompting, few- shot in-context learning, and supervised learning settings. In the closed setting where the model selects among 100 candidate entities, zero-shot and few-shot LMs perform on par (e.g. 38.5 F1 by 8-shot OPT-175B: Zhang al. (2022)) with sim- ple baselines such as predicting all candidate enti- ties (33.5 F1). RoMQA also remains very chal- lenging for state-of-the-art supervised methods, with the best setrieve-then-clasiie model achiev- ing 63.8 Fl compared to gold-evidence upper bound of 95.0 Fl. The open setting, where no candidates are given, is even more challenging to existing methods - the state-of-the-at Instruct- Ouyang al. (2022)) obtains 12.6 Pr@ 10 (precision at 10) while super- vised eetrieve-then- generate obtains 58.6 Pr@ 10.
Sampling constraints from a KB. We cre- ate RoMQA questions from Wikidata (Vrandečić and Krötzsch, 2014) that are answerable given entity-linked and relation-extracted text (Elsa- har et al., 2018). Wikidata consists of subject- proposition-object triples such as Gilbert_Amy occupation pianist. We convert these triples into entity-relation constraints. For instance, the previous example is decomposed into constraints Gilbert_Amy occupation obj and pianist occupation subj.
GPT3
Finally, no test model is robust variations in question constraints. The best performing retrieval method obtains a worse-case related question test score of 37.9 F1 in the closed setting - a 25.9 Fl absolute drop compared to evaluating questions independently. Training on clusters of related ques- tions. such as RoMQA clusters, improves model ro- bustness over training on unrelated questions. How- ever, the robustness gap remains large- - closing this gap will likely require significant advances in natural language understanding. We open-source RoMQA at anonymous. ur.
Clustering related constraints. A cluster of re- lated constraints shares at least two answer en- tities. For example, occupation pianist subj and palcce_o__brtte Paris subj are in the same cluster because they share the same answers Gilbert_Amy and [laude_Helffer (Paris-born pi- anists). As Wikidata has a skewed proposition distribution, we resample cluster constraints with probability inversely proportional to their propo-

---

RoMQA
A film composed by S. Thaman and produced by Ganesh Babu.
Who did not play for the Carolina Panthers but was linebacker and was the Chicago Bears?
Which members of the Royal Society received the Order of Australia. but employed by the University of Oxford?
Sub-orbital spaceflight that launched from Cape Canaveral Air Force Station Launch Complex 5. Launched hy Mercury- Redstone Launch Vehicle
Who is athlete who participated diving, and was horn in Stockholm?
HAtpotQA
Are Random House Tower and 888 7th Avenue both used for real estate?
Which American singer and songwriter has mezzo- -oezzan vocal range, Tim Armstrong or Tori Amos?
WFMT FM radio transmits from the second tallest building in the United States, which is located where?
Who was the recipient of prize also given to a player for Chinese club Tiagji Quanjian? Which of Tara Strong major voice role in animated series is an American animated television series based on the DC Comics
fictional superhero team, the "Teen Titans"?
SomplexWccuuestionn
What university has more than 15,835 undergraduates and is the university Derek Fisher attended? Who influenced Whitman's 's poetry who was the public speaker who spoke about the American Civil War?
What the main train station called in the govermmental jurisdiction where the government includes the position Mayor of
San Francisco?
Which country that borders Russia has the smallest ISO?
What country that's neighbor of Russia is a governmental jurisciction where Erik Asanbayev holds a governmental office?
QAMParl
Where did Roman Catholic archbishop of San Francisco attend school?
At what institution did Bishop of Derby receive their education? For which movie did Mani Ratnam work the script and serve producer?
What Type VII C/41 and Type VII ships in both the vessel classes of German? Philip Kaufman was responsible for both writing and directing of which movie?
Table I: Randomly sampled examples from RoMQA and other compositional QA datasets, Human evaluations show that people are more likely to ask RoMQA questions than those from other compositional QA datasets. Quali- tatively, RoMQA questions exhibit fewer artifacts such as overly precise constraints (c.g. 15,835 undergraduates), overly numerous references (e.g. an American animated... based on... the "Teen Titans"), and unusual attribute comparisons (e.g. smallest ISO).
sition frequency in the KB (Appendix A). This down-samples over-represented propositions such as country. We keep clusters with >3 constraints to be able to generate many related questions from each cluster. We discard clusters of potentially spuriously related constraints with single shared answer. 10k clusters are randomly chosen for train- ing and 15k clusters for evaluation.
pianist subj AND born_in Paris subj. We dis- card overly general queries with 25000 answers.
Creating natural language questions. Mechan- ical Turk workers annotate logical queries marked with Wikidata titles, descriptions, and aliases into questions. Appendix B Figure 11 shows the interface. Two more annotators verify each annota- tion to confirm that it matches the logical query. We keep only annotations with 100% agreement, re- sulting in 11% being discarded. After verification, additionally discard clusters with <2 questions.
Sampling constraints to form logical queries. We generate <5 logical queries using each clus- ter. For each logical query, we copy the cluster and remove constraints with probability 0.1 and negate with 0.1. We negate sparingly because large numbers of negative constraints result in unnatural questions. We further remove redun- dant constraints (c.g. American presidents born in the US), and uniformly subsample up 4 con- straints. This constitutes a logical query with mul- tiple conjunctions and subtractions. For instance, the cluster (occupation pianist subj, born_in Paris subj) can form a logical query occupation
2.2 Dataset analyses and comparison
We compare RoMQA to prior compositional QA datasets: HotpotQA (Yang et al., 2018), Com- plex WlexuutQussssons (CWQ; Talmor and Berant, 2018), and QAMParl (Amouyal et al., 2022).
Dataset size and question complexity Ta- ble 2 shows that only RoMQA evaluates robustness to input variations. Moreover, only RoMQA and QAMParI are human-writte with multiple answers

---

Human written
Multi answer
Gold
Robustness evaluation
Dataset
Train
Dev
Test
evidence
RoMQA (Ours)
11k
7k
11k
Yes
Yes
Yes
Yes
HotpotQA
90k
7k
7k
Yes
No
Yes
No
CWQ
28k
3k
3k
Yes
Yes
No
No
QAMParl
64k
Ik
1k
Eval only
Yes
Yes
No
Table 2: Dataset size and question complexity.
10"
dataset
RoMQA
0.175
dataset
RaMQA
0.14
dataset
RoMQA
Proportion of examples
CWQ
Proportion examples
CWQ
Proportion of examples
0.12
CWQ
10
HotpotQA
HotpotQA
0.10
HotpotQA
0.125
10"
QAMParl
QAMPari
QAMPari
0.100
0.08
10
0.075
0.06
10"9
0.050
0.04
10
0.025
100
200
of answers
# of evidence
# of words in question
Figure Dataset comparison over question, evidence, and answer size distributions.
unique NPs in 500 questions
Proportion of votes "someone would ask this question
1500
1000
500
0.2
0
RoMQA
CWQ
HotpotQA
QAMParl
0.0
RoMQA
CWQ
HotpotQA
QAMParl
Figure Question diversity as measured unique noun- phrases in 500 randomly sampled questions from the development set of each dataset. The batches are ran- domly sampled 4 times compute standard deviation.
Figure 4: The distribution of questions naturalness rat- ings by 3 annotators on 1,000 randomly sampled ques- tions from the development annotator rates four questions shuffled random order, one from each dataset. he annotator is asked whether they would ask the question themselves, and if they think someone else would ask the question.
of each dataset. Each
and gold evidence. QAMParl provides 2k human- written evaluation questions while RoMQA pro- vides 1 1k training and 17k evaluation questions. Figure shows the distribution of answer, ev- idence, and question sizes. First, RoMQA ques- tions require finding many answers. On aver- age, RoMQA questions have 108 answers at least 10x larger than others. Second, RoMQA re- quires reasoning over much more evidence docu- ments. On average, entities in the RoMQA answer set combine for total of 52 evidence sentences. Third, RoMQA questions are longer with more words. Figure shows that, a random sample of 500 questions, RoMQA refers to more unique noun phrases apart from HotporQA.
in randomly sampled questions from HotpotQA, WonpllexWeuuusstonss and QAMParl. They in- clude unusual constraints such as IDs (e.g.... the smallest ISO?!), overly precise constraints (e.g.... is an American animated television series based on the DC Comics fictional superhero team, the "Teen Titans") and excessive number of ref- erential expressions (e.g. in the governmental jurisdiction where the government includes the po- sition Mayor of San Francisco). We compare the naturalness of 1,000 randomly sampled human writ- ten questions from each dataset. Each annotator is shown a randomly sampled question from each
Naturalness human evaluation Prior work sometimes sacrifice question naturalness in pur- suit of complexity. Table illustrates artifacts
'ISO codes are 2-3 character-long codes that represent names of countries and their subdivisions.

---

Model class
Setting
Input format
Zero-shot
Closed
Gilbert Amy [SEP] Who was pianist born Paris?
Open
Who pianist born in Paris?
Few-shot
Closed
Katie Bell [SEP] Who is athlete who participated in diving, and was born in Stockholm? [SEP) False [newline] ... Gilbert Amy (SEP) Who was pianist born in Paris? [SEP] True
Open
Who is an athlete who participated in diving. and was born in Stockholm? [SEP] Johan Jansson [newline] ... Who was pianist horn in Paris? [SEP]
Supervised
Closed
Gilbert Amy [SEP] Who was pianist born in Paris?
Open
Who was a pianist born in Paris?
Suptevidence
Closed
Gilbert Amy [SEP] Who was a pianist horn in Paris? [SEP] Gilbert Amy (born August 1936) is French composer and conductor ... Who was pianist born in Paris? [SEP] Gilbert Amy (born 29 August 1936) is French composer and conductor... ...
Open
Table 3: Input format for question "Who was pianist born in Paris". For closed setting, "Gilbert Amy" is used example candidate. Evidence includes retrieved sentences for retrieval methods or gold evidence for upperbound.
dataset, shuffled in random order. The annotator is asked "how likely would you ask this question?" and "how likely do you think another person would ask this question?" Each question is annotated by crowdworkers. The breakdown of ratings across questions is shown for each dataset in Fig- ure 4. On average, annotators consider RoMQA to be significantly more natural than HotpotQA and ComplexWebee WebQuestions.
of only seeing some examples (e.g. list British foot- ballers). For each score, we additionally have ro- bustness variant. Let Q {?1. Q2 -- --. denote cluster of related questions. The question qi has the corresponding predicted answer set pt and gold answer set Jr- robustness score is the worst-case score across the cluster. For instance, the robust Fí is F, (T(()) = min; (F (pr,g:)) We compute similar robustness scores for accuracy and preci- @
3 Experiments
3.2 Models
How do existing QA systems perform on RoMQA? Are they robust to variations in question con- straints? We answer these questions by experi- menting with state-of-the-art models in zero-shot, few-shot, and supervised learning settings.
We evaluate three classes of models. The input format for each class is shown in Table 3.
Zero-shot. We consider a naive closed setting baseline that predicts all candidates as answers. We also include state-of-the-arr prompting mod- els tk-instruct-38 (Wang al., 2022) and opt-instruct-1758 (Zhang al., 2022). In the closed setting, they generate yes/no given the ques- tion and a candidate. In the open setting, they generate answers given the question.
3.1 Evaluation
Open vs closed settings Given a question in RoMQA, a QA system should produce a set of answer entities. In the closed setting, the system is given 100 candidate entities and must identify which ones answer the question. Negative candi- dates are potentially difficult for a model because they can match any constraint in the question. In the open setting, candidates are not given.
Few-shot in-context learning. We eval- uate tk-instruct-38 (Wang et al., 2022). opt-instruct-1798 (Zhang et al., 2022), and GPT3 (text-davinci-oo0) Brown et al. (2020)) with as many in-context examples as the model allows (4, 8, and 8 respectively). Input format is similar to the zero-shot setting, with the addition of in-context examples. In closed setting, the context includes equal number of randomly sampled positive and negative examples. We compare the scores for the candidate answering vs. not answering the current question. These scores are calibrated using channel calibration (Min et al., 2022). open setting, the model context includes
Evaluation metrics RoMQA questions have many answers. For the closed setting, we evaluate predictions using F1 and accuracy. F1 measures set overlap between predicted and gold answers, while accuracy measures whether they match exactly. For the open setting, evaluate precision @ K (P1o) for two reasons. First, precision gives partial credit when it is too hard enumerate the full set. Sec- ond, the user may ask question with many an- swers (e.g. hundreds or thousands) with the intent

---

Dev
Test
Model
F1
ace
F1 R
acc
ace
Zero-shot
predict all
34.1
13.1
0.0
0.0
33.5
12.8
0.0
0.0
ik-instruct-3b-
34.5
13.2
0.0
0.0
34.0
12.8
0.0
0.0
opt-instruct-Oshot
36.0
14.0
0.0
0.0
36.0
14.0
0.0
0.0
Few-shot
tk-instruct:bb--shot
33.5
12.9
0.0
0.0
33.1
12.5
0.0
0.0
opt-8shot
38.9
16.1
0.0
0.0
38.5
15.5
0.0
0.0
Supervised
binary
35.9+0..7
10,9+0.6
2.3+0.1
0.320..1
35.5
10.2+1.3
2,5+0.3
0.3+0.1
bintry+tettrival
64.0#0.6
38.6
7.0+0.3
0.340.1
63.8 $
37.9+1.1
7.0+0.1
0.7+0.1
binary+gold evidence
95.3+t0.3
83.5+0.8
1.8
39.242.6
95.0+0.3
83.4+0.9
71,5+0.9
38.2=1.2
Table 4: Model performance closed setting RoMQA. Metrics set F1. set accuracy, and their robustness counterparts (i.e, worst case measure over cluster of related questions), Each model is given 100 candidate entities and must decide which entity belongs to answer set. The retrieval model additionally observes sentences retrieved via BM25 followed by DPR. Zero-shot and few-shot are binary-classffiers calibrated with channel calibration. Supervised models fine-tune BART- BART-large on the training data to classify the answer set on a per-entity basis.
10 subsampled answers for each example.
"Claude Helffer... was a French pianist", "Helf- fer was born in Paris", and "David Fray... is a French classical pianist". Because the gold evi- dence only contains sentences that provide prove- nance an implicit constraint, does not contain the sentence "David Fray was born in Tarbes, near the Pyrenees." " other words, given gold evidence, the QA model does not need to filter out incorrect answers in the candidate answers pool (c.g. David Fray) because it only has to verify that the pro- vided evidence supports all parts of the question (e.g. born in Tarbes instead of Paris). Instead, it only needs verify that the evidence references all implicit constraints. Consequently, the gold ev- idence setting overly optimistic in that part the reasoning is completed by perfect retriever. While no such retriever currently exists, this set- ting nevertheless provides upper bound estimate for RoMQA.
Supervised learning. We tune BART-large with and without retrieved evidence (Lewis al., 2020) and show standard deviation across 5 random seeds. For the closed setting which considers a candidate entity, we use a two-stage hybrid retrieval because dense retrievers under-perform sparse retrievers on rare, precise entities (Sciavolino et al., 2021). We first retrieve documents with BM25 using entity title the query. We then use DPR (Karpukhin al., 2020) to retrieve document sentences whose cosine similarity with the question exceed thresh- old (0.65). This threshold is tuned by maximizing the retrieval hit rate on the validation set. Finally, we fine-tune to classify whether each candidate belongs to the answer set.
In the open setting, we do not use classification models because it is computationally prohibitive to decide over all possible (2.9M) entities per ques- t Instead, we directly retrieve evidence with DPR and fine-tune the model to generate the an- swer set as a 1024-token sequence.
3.3 Results
Table 4 and Table 5 show performance on RoMQA closed and open settings. RoMQA is challenging for state-of-the-ar large-scale models. Moreover, these models are not robust to variations in question constraints. The best models signifi- cantly trail the gold evidence upper bound, showing there is significant room future work.
Upper bound with gold evidence. We provide performance upper bound by training supervised models with gold evidence sentences that pro- vide provenance to an implicit question constraint. For instance, consider the example Figure 1. Claude Hellfer is an answer to the question "Who was a pianist born in Paris?", whereas David Fray is not. In this case, the gold evidence includes
Zero-shot and few-shot models perform simi- larly to naive predict-all baseline. In the closed setting, each system given a set of 100 candidate entities and must identify which entity belong to the answer set. We find that ofate-of-the-an pre-
Forr reference, inference over the entire tast set with 100 candidate entities per example using the classification model requires 10 hours on a Volta 32GB GPU.

---

Dev
Test
Model
F1
F1
P10
P10
F1
P10
P10
Zero-shot
lb-instruct-3--oshot
0.2
0.0
1.7
0.0
0.3
0.0
1.8
0.0
opt-instrrut-Oshot
1.6
0.0
5.0
0.1
1.6
0.1
5.1
0.2
Few-shot
Ik-instruct-h--sshot
0.3
0.0
1.9
0.0
0.4
0.0
2.0
0.0
opt-Sshot
2.1
0.0
5.5
0.2
2.2
0.1
5.6
0.1
gpt3-8shot
4.3
0.4
13.3
1.2
4.4
0.4
12.6
1.0
Supervised
seq2seq
32.2+0.9
11,2+0.5
47.3#1.2
20.3+
32.6#1.00 1.0
11,5+0..
45.7
19.7=1.1
seq2seq+retrieval
41.640.8
19.1
58.6#1.5
38.342,4
41.080.5
18.420.6
56.8+1.4
36.0#2.3
Table 5: Model performance open setting RoMQA. Metrics counterparts (i.e. worst case measure over cluster of logically related questions). Each model is given the question and must generate the answer set sequence. The retrieval model additionally observes sentences retrieved by DPR. Supervised models fine-tune BART-large on the training data. All models generate the answer set sequence. We do not evaluate upperbound gold evidence method because it necessarily provides candidate entities and therefore is no longer open domain.
F1. precision 10. and their robustness
1.0
0.4
pearson F=04899, p=0.000
pearson r= 154, p=0.000
pearson
p=0.000
precision
recall
11
precision
recall
f1
precision
recall
0.0
0
10
20
30
40
60
0
10000 20000
30000
1.0
1.5
2.0
2.5
3.0
3.5
4.0
# answers
prop constraint generality
# positive constraints
Figure 5: Correlation with model performance (F on the closed setting. Imprecise questions with many answers are casier to answer (higher F,). Questions based on general propositions that co-occur with many different entities are easier answer. Questions with constraints more difficult answer.
trained instruction prompting models perform on par with the naive baseline of simply predicting that every candidate belongs to the answer set. This oc- curs both with instructing prompting and in-context learning models, and suggests that they can not ef- fectively reason about the types of compositional questions found in RoMQA.
more precise questions. First, when the question has many answers, the model has an easier time pro- ducing some correct answers. Second, the model performs better on more general propositions that co-occur with many different unique entities. Third, the model struggles with questions with more im- plicit constraints.
Methods are not robust to question constraint variations. All methods drop in performance when considering the worst-case performance among clusters of related questions. This suggests that large LM-based QA systems are not robust to variations in question constraints. Figure 6 shows what types of questions result in robustness drops. Compared other questions in the same cluster, a question is easier it contains more answers, and harder if specifies more implicit constraints.
Both closed and open settings remain challeng- ing with supervised training. When given 1 annotated examples, large retrieval models perform better than zero-shot and few-shot LMs. However, supervised performance also trails the gold evi- dence upper bound. This suggests that there is significant room for modeling improvements that retrieve and compose evidence in order to answer compositional questions.
What types of questions do the best-performing supervised systems struggle with? Figure 5 plots Pearson correlation with F1 in the closed setting, and shows that systems generally struggle with
Training on clusters of related questions
'Pearson correlation is measure of linear corelationn A Pearson coefficient of I or -1 implies positive or negative linear correlation, while 0 implics no linear dependency.

---

Training Closed
Open
0.4
questions Fí
Fi
AF,
Pio
Pio"
1o
0.2
Unrelated 56.2
28.2
-28.0
27.0
-25.4
Related 55.7
28.6
-27.1
26.3
11.7
-14.6
0.0
-02
Table 6: Training supervised retrieval models on related vs. unrelated questions. Unrelated questions training involves selecting question from each cluster, In contrast, related questions training includes entire clus- ters of related questions. While training on more diverse set of unrelated questions yields slightly higher overall performance, training on related questions pro- duces robust models.
pearson 0199,, p=0.000
pearson 0.344, p=0.000
-044
precision
recall
f1
precision
-40
-20
0
20
-2
-1
# answers
A# positive constraints
Figure 6: Correlation with robustness drop (F'1 cluster mean F1) in closed setting. The axes denote deviation from the cluster means. Among cluster of related questions, more precise question with more constraints or fewer answers tend to be harder for the model than related questions with more answers less constraints.
K Wwittkowski et al., 2019; Sciavolino et al., 2021) to answering over multiple passages (Yang et al., 2018; Welbl et al., 2018; Thorne et al., 2018). Re- cent work emphasize answering questions that have multiple answers (Min et al., 2020; Amouyal al., 2022). RoMQA combines the latter two settings in that it requires answering questions over multi- ple pieces of evidence to provide multiple answers. Compared to prior datasets, RoMQA questions re- quire robust reasoning over more pieces of evi- dence provide more answers.
(e.g. RoMQA clusters) is one way improve model robustness. Given clusters of questions with related implicit constraints, in the first setting we train on unrelated questions - one question from each cluster for total of K training examples. In the second setting, we train on related questions K consecutive examples from entire clusters. Ta- ble 6 shows that while the diversity from training unrelated questions marginally improves overall performance, training on clusters of related ques- tions results in more robust models. Nevertheless, the robustness drops remain significant. Consider- ing that variations in RoMQA questions are reason- able questions humans would write, as opposed to artificially created adversarial questions, our find- ings suggests that there is practical need for de- veloping more robust QA systems.
Robustness evaluation NLP systems have pre- vious been show to lack robustness. They are sus- ceptible to character based attacks that comprise of both nonsensical inputs (Jia and Liang, 2017), ran- dom sentence/word triggers (Jia and Liang, 2017; Wallace et al., 2019), and semantically equiva- lent inputs adversarially selected to disrupt sys- tem behaviour (Ribeiro et al., 2018; Zhao et al., 2018: Iyyer et al., 2018). contrast, the questions in RoMQA are not adversarial - they are written with reasonable information seekiitnio--eekinnggo intent.
Building context for open setting is very chal- lenging. While the closed setting RoMQA chal- lenges current state-of-the the-trr models, the open setting remains an even greater challenge. A key challenge in the open setting is that it is difficult to compute the evidence set required the answer the question. Consider Figure I's question "Who was Western classical music pianist, not born in Paris". The obvious way human would answer this question is substracting the set people born in Paris from the set of Western classical music pianists. However, both of these sets are very large. Our results show that an end-to-end large language model struggles in reasoning over such large sets.
Zero-shot/few-shot learning Recent work has also shown that large pretrained LMs can perform zero-shot and few-shot inference (Brown et al., 2020; Wang et al., 2022; Zhang al., 2022). For the former, the LM performs inference given a prompt or an instruction. For the latter, the LM is also given training examples as demonstration. We use both settings as baselines for RoMQA, and find that there is significant room for improvement in acage-scale pretraining to capture compositional reasoning over multiple pieces of evidence text.
Conclusion
4 Related Work
We presented RoMQA, the first benchmark for robust, multi-evidence, multi-answer QA.
Question answering datasets Existing QA datasets focus on answering from a single pas- sage (Rajpurkar et al., 2016; Joshi et al., 2017;
RoMQA evaluates robustness models to vary- ing question constraints by testing for worst-case

---

performance among clusters of related questions. RoMQA is a challenging dataset for state-of-the- art large LMs and provides quantifiable test for developing more robust QA methods.
Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
6 Limitations
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In LREC 2018.
Robustness definition A robust system needs to be robust towards all forms of input variations. The type of robustness to varying implicit con- straints measured in RoMQA is but one form. Other forms of robustness include adversarial character attacks, sentence injections, and word triggers. RoMQA limited English - future work should investigate robustness in other and multiple languages. The questions in RoMQA are based on Wikidata triples and contains limi- tations inherent to Wikidata and large-scale knowledge-grapha in general (e.g. staleness of facts, over-represented propositionssenniries,, in- completeness). The models evaluated in this work (e.g. opt-instruct-1755 text-davicinci-e00 are but a sample of state-of-the-arl language mod- els at the time of writing. As new models emerge, it is possible that they exhibit robustness beyond that of the evaluated models.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In NAACL.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: large scale distantly supervised challenge dataset for reading comprehen- sion. In ACL.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In EMNLP.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova. Llion Jones. Matthew Kelcey, Ming-Wii Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: benchmark for question answering research. TACL.
7 Ethics Statement
In order for Al systems to be used in critical appli- cations, we need to establish trust in these systems. One hallmark of trustworthiness is robust system behaviour. RoMQA is a preliminary step quanti- fying the robustness of question answering systems to natural variations in the question. It is our hope that the release of RoMQA will facilitate the de- velopment of more robust and more trustworthy NLP systems, and that future work will provide further quantifiable tests for the robustness of NLP systems.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In ACL.
pre-training
Sewon Min, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. In ACI.
References
Samuel Joseph Amouyal, Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Be- rant. 2022. QAMPARI: An open-domain question answering benchmark for questions with many an- swers from multiple paragraphs. arXiv 12665.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In EMNLP.
Long Ouyang, Jeff Wu, Jiang. Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller. Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow instructions with human feedback. arXiy 2203.02155.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Carbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens

---

Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNIP.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversar- ial rules for debugging NLP models. In ACL.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centri ques- tions challenge dense retrievers. EMNLP.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL.
James
Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In NAACL.
Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: A free collaborative knowledge base. Commu- nications of the ACM.
Eric Wallace, Shi Feng. Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial trig- gers for attacking and analyzing NLP. In EMNLP.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, et al. 2022. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv 2204.07705.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL.
Zhilin Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian L.i, Victoria Lin, Todor Mi- haylov, Myle Oit, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoer.. 2022. Opt: Open pre-trained transformer language models. arXiv 2205.01068.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. ICLR.

---

A
Subsampling propositions
C Dataset Statistics
Cluster
0..33
answer ents
76
154
237
0.02
333
10
433
574
0.01
748
# 985
16"
0.00
2
3
4
# constraints
10
Figure 8: Answer set size vs constraint count.
6
Figure 7: Number of constraints per cluster.
1.00
We want questions that cover diverse topics, however Wikidata has a very skewed proposition distribution, with a long tail of rare propositions. Hence, we down-sampi frequent propositions. Let Prrop(#) denote the percentage of triples that con- tain the proposition z. We define the average propo- sition probability Prrop' X En Prrop (1) Given constraint with proposition I, we remove
pos constraints
0.75
CV
0.50
0.25
0,00
1
# neg constraints
Figure 9: Positive vs negative constraint count.
1 min (1, Pprop' Pprop (2)
it with probability In particular, those with below average frequency are removed, and those with above average frequency are removed with increasing likelihood. After removing constraints based on propositions, we randomly sample up to 10 constraints using a distribution over their inverse proposition proba- bilities _ Figure 7 shows the distribution over cluster sizes after resampling. Figure 10 shows that resampling results in more diverse set of questions by emphasizing rarer propositions in the knowledge graph.
Cluster sizes. Figure shows the distribution cluster sizes in RoMQA. During the sampling pro- cedure, we remove small clusters of <3 questions and avoid large clusters of 27 questions.
Implicit constraint distribution. Figure shows the distribution of positive and negative implicit constraints in RoMQA questions. Most questions have positive constraints, and 0-1 negative con- straints. We limit questions to constraints during sampling. In practice, nonsensical questions with too many constraints are filtered out during verifica- tion. Figure shows the distribution of implicit con- straints vs. the size of the answer More precise questions with more implicit constraints typically have fewer answers. In general, RoMQA questions may have more than 1000 answers, though the vast majority contain less than 1000 answers. The out- lier questions with more than 1000 answers are not shown in the figure.
B Annotation
RoMQA questions are annotated by crowd-workers on Amazon Mechanical Turk from the US, Canada, UK. Australia, and New Zealand. We require that annotators have 295% approval rating and have done a minimum of 5000 HITs. Questions mitted for annotation in batches of 500. For each batch, a sample of 10 questions from each worker is inspected by the authors. If 22 annotations in the sample are incorrect, then response from the worker in that batch are marked for re-annotation. The final set of annotations are additionally veri- fied by 2 more crowd-workers to confirm that they correspond constraints. We keep only exam- ples with 100% agreement, which corresponds to 89% of the annotated data. Workers were paid minimum of 15 USD per hour, estimated using completion time per HIT.
sub-
D Experiment setup
For zero-shot and few-shot models, we use API services provided by the original authors. For su- pervised models, we fine-tune BART-large models with learning rate 1e 6 on V100 GPUs. For clas- sification in the closed setting, train with batch size 100 and evaluate with batch size 1000. For generation in the open setting, we train and eval- uate with batch size 2 and decode with beam size 3. Experiments are run on a Slurm cluster V100

---

Top 100 proposition types
country located the administrative territorial entity
instance country cilizenship occupation member of sports place of birth sport cast member laxon rank genre place of death performer country of origin participant educated
earent taxon languages spoken, written signed director shares border with part of record label position played on team speciality
screenwriter employer
prop
headquarters location
member political party
conflict
original language of film TV show
author
subclass
award received
operator
contains administrative territorial entily
platform
publisher
member
named after
child
producer
spouse
father
position held
military branch
owned by original broadcaster
instrument
orig
sibling location
final
0.00
0.02
0.04
0.06
0.08
0.10
0.12
prob
Figure 10: Most common propositions, before and after subsampling. Subsampling downsamples overly represented propositions in the knowledge graph and results in more diverse set of propositions and question topics.
GPUs for 3-5 days each. We default hyper- parameters for supervised models, with maximum batch size as allowed by our GPUs. We perform early stopping on the validation set (for supervised methods) and evaluate on the test set only once.
rank_bm22 Apache 2.0
spacy: MIT
centence_transfommee Apache 2.0
ray: Apache 2.0
License
wrangl: Apache 2.0
The majority of RoMQA is licensed under CC-BY- NC, however portions of the project available under separate license terms:
ujson: link
gwickidata: Apache 2.0
hydra-core: MIT
torch: link
tqdm: MIT

---

Write a paragraph summarizing these statements:
True
Damon Hill: British racing driver
pole position: person, who starts race at first row
this thing
False
1993 Formula One season: sports season
part of: object of which the subject part. AKA: in, chain,
this thing
contained within
Examples things that should meet the criteria described by your paragraph (don't copy them in your answer):
Argentine Grand Prix: Formula motor
1995 Australian Grand Prix: 581st Formula Championship Grand Prix
French Grand Prix: Formula motor held 1994
Mechanical Turk annotation interface for question writing. The annotator is shown collection positive and negative constraints in random order. Each constraints consists of entity, proposition, and direction. Entities and propositions are expanded with descriptions and aliases from Wikidata. A subset of answer entities listed to disambiguate answer types.
Figure II:


Reviews:
[{"review_id": "iFkfn0mUGo", "reviewer": "Reviewer_Tpkz", "summary": "", "questions": "1. When construct the question, how to consider the robustness evaluated examples and the adversarial examples (I suppose there are no consideration of the adversarial examples now), and what will happen if we add the adversarial examples when evaluate the robustness (from both the problem definition aspect and the evaluation aspect)?\n\n2. What are the performance of the non-LLM methods, in terms of the widely use performance metric (e.g. F1, EM) and the robustness metric?\n\n3. Since the datasets have already provided the evidences, what is the retrieval role of the retrieval model (in Table 5) and why we need retrieval here and how it can improve the QA results?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n> ***1. The construction of the datasets could contain some risks. For example, the scope is limited to a specific type of robustness based on constraint variations and does not address other aspects like adversarial examples.***\n\nWe agree that the definition of robustness is considered broader and contains different aspects. They are all important. We choose to focus on robustness to compositional constraints because a reliable compositional QA system should exhibit the ability to manage minor variations in constraints, demonstrating a deep understanding of the text and the relationships between different pieces of information. \nWe believe it would be valuable for future study to broaden the scope to include other dimensions of robustness. We appreciate the reviewer for highlighting this perspective and will include it into our discussion of potential avenues for future research. We will also clarify the scope of this work to compositional constraints.\n\n> ***2. Besides, the source highly relies exclusively on Wikipedia/Wikidata for sourcing facts which could limit diversity and lead to factual errors.***\n\nAs stated on the Wikidata page [1], the content in Wikidata is verifiable, supported by authoritative sources of information. References are incorporated into item pages, serving as pointers to specific sources that substantiate the data presented in a statement. While Wikidata may contain imperfections, we believe it to be a valuable source of factual knowledge for assessing compositional robustness, given its extensive coverage of relational facts. Moreover,  Additionally, it is worth noting that numerous previous studies have established their QA benchmarks based on Wikidata [2,3], indicating it is widely recognized in the research community. We appreciate your concern and will incorporate a discussion of it in the limitations section of our paper.\n\n\n> ***3. The evaluation is mostly focused on large pretrained language models which does not cover a wide range of QA methods and support the initial statement of the existing QA models cannot achieve better robustness ability. Besides, the problem definition could be more clear. What are the performance of the non-LLM methods, in terms of the widely use performance metric (e.g. F1, EM) and the robustness metric?***\n\nWe appreciate the reviewer's comment on the scope of our evaluation. To the best of our knowledge, the current state-of-the-art QA systems, such as FiD [4] and Atlas [5], are predominantly based on language models. If the reviewer has any suggestions regarding additional systems to evaluate, we would appreciate the suggestion and incorporate the evaluation of these systems into our final version.\n\n> ***4. When construct the question, how to consider the robustness evaluated examples and the adversarial examples***\n\nWe appreciate the reviewer bringing up the important aspect of adversarial examples. Our focus in this study was on evaluating the model's robustness to variations in constraints, which is one important dimension of robustness in compositional QA. We believe that adversarial examples and constraint variations are complementary dimensions of robustness that collectively provide a comprehensive evaluation of a model's capabilities. For the evaluation perspective, we will include two metrics: one is to evaluate robustness towards constraint variations and the other is to evaluate models’ ability to handle adversarial questions. \n\n> ***5. Since the datasets have already provided the evidences, what is the retrieval role of the retrieval model (in Table 5) and why we need retrieval here and how it can improve the QA results***\n\nIn practical scenarios, the evidence required for QA systems is not always readily available, requiring the QA systems to learn two key skills: (1) the retrieval of multiple pieces of evidence, and (2) reasoning across different pieces of information to construct an answer. Our open-book setting is designed to mimic this real-world process of answering compositional questions. Additionally, there are many prior work that focuses on multi-evidence retrieval for multi-hop questions, as illustrated by the following studies [6, 7, 8]\n\n\n[1] https://www.wikidata.org/wiki/Wikidata:Verifiability\n\n[2] QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs\n\n[3] Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases\n\n[4] Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering\n\n[5] Atlas: Few-shot Learning with Retrieval Augmented Language Models\n\n[6] 'Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval'\n\n[7] 'Multi-hop Evidence Retrieval for Cross-document Relation Extraction'\n\n[8] 'Multi-Hop Paragraph Retrieval for Open-Domain Question Answering'", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addresses an important issue that the robustness is a critical but under-studied area for QA systems. The proposed dataset provides a concrete way to measure it.\n2. The paper thoroughly evaluates many state-of-the-art models and sets challenging benchmarks and provides useful insights into what types of questions are more challenging in terms of robustness.\n3. It seems that the RoMQA dataset thoughtfully constructed based on sourcing constraints from a knowledge graph.", "reasons_to_reject": "1. The construction of the datasets could contain some risks. For example, the scope is limited to a specific type of robustness based on constraint variations and does not address other aspects like adversarial examples. Besides, the source highly relies exclusively on Wikipedia/Wikidata for sourcing facts which could limit diversity and lead to factual errors. \n\n2. The evaluation is mostly focused on large pretrained language models which does not cover a wide range of QA methods and support the initial statement of the exist QA models cannot achieve better robustness ability. Besides, the problem definition could be more clear.\n\n3. Lacks detailed error analysis explaining why models struggle on certain questions and the robustness metric may not correspond well to real-world robustness if question variations are not natural or representative. More qualitative examples can be added for both illustrating the dataset construction and evaluation.", "questions_to_authors": "1. When construct the question, how to consider the robustness evaluated examples and the adversarial examples (I suppose there are no consideration of the adversarial examples now), and what will happen if we add the adversarial examples when evaluate the robustness (from both the problem definition aspect and the evaluation aspect)?\n\n2. What are the performance of the non-LLM methods, in terms of the widely use performance metric (e.g. F1, EM) and the robustness metric?\n\n3. Since the datasets have already provided the evidences, what is the retrieval role of the retrieval model (in Table 5) and why we need retrieval here and how it can improve the QA results?", "paper_topic_and_main_contributions": "This paper introduces RoMQA, a new question answering dataset and benchmark for evaluating the robustness of QA systems. The paper claims that prior QA datasets do not explicitly test for model robustness to small variations in question phrasing or constraints and the authors aim to create a benchmark that measures QA robustness.\n\nThe main contributions are \n1. The proposed RoMQA dataset containing clusters of related questions that vary in their implicit constraints which aim to address the robustness evaluation for the QA models.\n2. The proposed RoMQA has more complex questions requiring reasoning over more evidence documents and providing more answers on average and the human evaluations show RoMQA questions are more natural than other compositional QA datasets.\n3. Experiments show state-of-the-art models have low robustness based on RoMQA evaluation and the analysis indicates some interesting results e.g. training on RoMQA clusters improves robustness compared to unrelated questions.\n\nIn summary, the paper introduces a new robustness-focused QA dataset and benchmark, and uses it to demonstrate that existing models lack robustness to subtle question variations. The dataset and robustness metric are the main contributions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mAZp5c7fkV", "reviewer": "Reviewer_EMCn", "summary": "", "questions": "- Line305: how was the 0.65 threshold determined\n- Line 300: It would still be interesting to evaluate the performance of a dense retriever.\n- Line 314: The section on \"upper bound with gold evidence\" is unclear.", "limitations": "", "rating": 3, "confidence": 1, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n>***1. The modeling part of the paper where the authors mention that the performance of prompting LLMs is on par with the naive baseline is not convincing.***\n\nWe ask the reviewer whether our finding is “not convincing” or “surprising”. If the reviewer finds our result not convincing, we welcome feedback on how to make our finding more convincing. If the reviewer finds our result surprising: our dataset comprises complex questions, each with multiple possible answers—averaging 108 answers per question. However, large-scale language models typically generate only a handful of answers, which inherently restricts their performance on our dataset. Moreover, the observed suboptimal performance of current state-of-the-art models highlights the complexity of the challenges posed by our dataset, thereby emphasizing the need for future research in the direction of compositional QA. We hope that this addresses the reviewer’s concerns.\n\n>***2. Line305: how was the 0.65 threshold determined.***\n\nThis threshold is tuned by maximizing the retrieval hit rate on the validation set\n\n>***3. Line 300: It would still be interesting to evaluate the performance of a dense retriever.***\n\nOur early experiments showed that dense retrieval underperforms hybrid retrieval on a subset of the corpus. This finding is in line with other works on sparse retrieval. \n\n>***4. Line 314: The section on \"upper bound with gold evidence\" is unclear.***\n\nIn that section, we aim to emphasize that gold evidence only includes sentences that support the answer, and not sentences that would contradict it or are irrelevant. This means that, when using gold evidence, the model does not need to filter out incorrect answers in the candidate answers pool because it only has to verify that the provided evidence supports all parts of the question. We appreciate your feedback in terms of writing and will clarify this section in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose a large dataset for question answering for entity-based questions which will be useful for other researchers.", "reasons_to_reject": "- The modeling part of the paper where the authors mention that the performance of prompting LLMs is on par with the naive baseline is not convincing.\n- Some parts of the write-up is unclear and some details are missing (please see questions below)", "questions_to_authors": "- Line305: how was the 0.65 threshold determined\n- Line 300: It would still be interesting to evaluate the performance of a dense retriever.\n- Line 314: The section on \"upper bound with gold evidence\" is unclear.", "paper_topic_and_main_contributions": "The paper proposes a new multi-evidence, multi-answer question answering (QA) dataset that was generated by mining entities and constraints from wikidata and having annotators convert the logical form of these entities and constraints to questions. The authors cluster related questions together and use a 'robustness' score by only considering the minimum performance of the questions in the same cluster. In addition, they explore the performance of zero-shot, few-shot (in-context learning) and fine-tuning of language models on the dataset and show that the dataset is challenging. While the dataset is useful, the modeling part of the paper needs further details in order to support the claims made in the paper.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h4hdxw6BVr", "reviewer": "Reviewer_zaFs", "summary": "", "questions": "1. In Table 1 under RoMQA, why there are two sentences that are not questions, are they included in the dataset?\n2. l. 243, in the closed setting, how are the 100 candidates selected? Are they different for different questions? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n> ***1. The author defines robustness of the model as its capability to a cluster of questions that share multiple constraints. However, the robustness in general is considered broader than this.***\n\nWe agree that the definition of robustness is considered broader and contains different aspects. They are all important. We choose to focus on robustness to compositional constraints because a reliable compositional QA system should exhibit the ability to manage minor variations in constraints, demonstrating a deep understanding of the text and the relationships between different pieces of information. \nWe believe it would be valuable for future study to broaden the scope to include other dimensions of robustness. We appreciate the reviewer for highlighting this perspective and will include it into our discussion of potential avenues for future research. We will also clarify the scope of this work to compositional constraints.\n\n\n> ***2. The results show that the model is not robust to a cluster of similar questions, it would be interesting to see how the shared constraints (or contradicted constraints) impact the model performance.***\n\nAs shown in Figure 6, we show that very specific questions, with a lot of constraints or limited possible answers, are usually harder for the model to handle compared to questions that have more possible answers or fewer constraints. For instance, typically, given that the answer set is not very large (or considering the closed setting), the question “What are the A that B and C” is more difficult to answer than “What are the A that B”. We do not evaluate for contradicting constraints that result in no answers, but will investigate this in the future.\n\n\n> ***3. In Table 1 under RoMQA, why there are two sentences that are not questions, are they included in the dataset?***\n\nWe use the term “question” to refer to utterances that have information-seeking intent. We will clarify this in the manuscript. Thank you for pointing this out.\n\n> ***4. l. 243, in the closed setting, how are the 100 candidates selected? Are they different for different questions?***\n\nThank you for this question - we will add details on how the 100 candidates are generated in the paper. Given a set A of positive entities (e.g. those that answer the question), we sample at most 50 candidates. Then, we consider all entities that answer one of the constraints in the question (call this B), and sample the remaining entities from the set B-A. In this way, the negative candidates are “hard” because they are very related to the question in the sense that they answer at least one of the constraints.\n\n> ***5. Multi-evidence retrieving is a popular topic in NLP, it would be beneficial if the author adds discussions and comparisons of multi-evidence retrieving in the paper.***\n\nWe thank the reviewer for pointing us to the related works. IIn the final version of our paper, we will include a discussion on multi-evidence retrieval and incorporating the following papers\n\n[1] Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval\n\n[2] Multi-hop Evidence Retrieval for Cross-document Relation Extraction\n\n[3] Multi-Hop Paragraph Retrieval for Open-Domain Question Answering\n\nIf there are any additional related works that we may have overlooked, we would appreciate your guidance in identifying them so that we can include them in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper present a new dataset with high quality, which can be considered as a main contribution.\n2. Creating clusters of similar questions based on shared constraints seems effective.\n3. The paper is well-written and easy to follow. Experiments are thorough and solid. \n", "reasons_to_reject": "1. The author defines robustness of the model as its capability to a cluster of questions that share multiple constraints. However, the robustness in general is considered broader than this. \n2. The results show that the model is not robust to a cluster of similar questions, it would be interesting to see how the shared constraints (or contradicted constraints) impact the model performance.\n", "questions_to_authors": "1. In Table 1 under RoMQA, why there are two sentences that are not questions, are they included in the dataset?\n2. l. 243, in the closed setting, how are the 100 candidates selected? Are they different for different questions? ", "paper_topic_and_main_contributions": "This paper presents a new QA dataset called RoMQA. RoMQA includes clusters of similar questions that share certain implicit constrains, require multi-evidences retrieval, and have multiple answers. The dataset is evaluated with 3 experiment settings: zero-shot, few-shot model, and supervised model, all with closed and open input format. \nResults show that existing QA systems are not robust to variation within a cluster of questions. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "2. Multi-evidence retrieving is a popular topic in NLP, it would be benefit if the author adds discussions and comparisons of multi-evidence retrieving in the paper. ", "Typos_Grammar_Style_and_Presentation_Improvements": "l. 111, such \"as\" RoMQA clusters"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a dataset for QA where questions have many answers. The authors focus the evaluation on robustness where clusters of similar questions are considered.

The reviewers like the robustness focus and the way the dataset is constructed.

On the other hand some reviewers mention that the baselines might be weak, although in my mind using LLMs and other smaller capacity supervised models from the literature makes sense. Reviewers also mention that the paper does not deal with all aspects of robustness, and to that authors replied that the paper focuses on compositionally. Finally, reviewers ask for an error analysis and several specific clarifications that were raised as questions.
Decision: Accept-Findings