QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Year: 2023

Abstract:
Formulating selective information needs results in queries that implicitly specify set operations, such as intersection, union, and difference. For instance, one might search for "shorebirds that are not sandpipers" or "science-fiction films shot in England". To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of entities corresponding to Wikipedia documents. The dataset challenges models to match multiple constraints mentioned in queries with corresponding evidence in documents and correctly perform various set operations. The dataset is constructed semi-automatically using Wikipedia category names. Queries are automatically composed from individual categories, then paraphrased and further validated for naturalness and fluency by crowdworkers. Crowdworkers also assess the relevance of entities based on their documents and highlight attribution of query constraints to spans of document text. We analyze several modern retrieval systems, finding that they often struggle on such queries. Queries involving negation and conjunction are particularly challenging and systems are further challenged with combinations of these operations.1

heading: 1 Introduction
text: People often express their information needs with multiple preferences or constraints. Queries corresponding to such needs typically implicitly express set operations such as intersection, difference, and union. For example, a movie-goer might be looking for a science-fiction film from the 90s which does not feature aliens and a reader might be interested in a historical fiction novel set in France. Similarly,
∗Work done during an internship at Google. 1The dataset is available at https://github.com/
google-research/language/tree/master/language/ quest.
a botanist attempting to identify a species based on their recollection might search for shrubs that are evergreen and found in Panama. Further, if the set of entities that satisfy the constraints is relatively small, a reader may like to see and explore an exhaustive list of these entities. In addition, to verify and trust a system’s recommendations, users benefit from being shown evidence from trusted sources (Lamm et al., 2021).
Addressing such queries has been primarily studied in the context of question answering with structured knowledge bases (KBs), where query constraints are grounded to predefined predicates and symbolically executed. However, KBs can be incomplete and expensive to curate and maintain. Meanwhile, advances in information retrieval may enable developing systems that can address such queries without relying on structured KBs, by
14032
matching query constraints directly to supporting evidence in text documents. However, queries that combine multiple constraints with implicit set operations are not well represented in existing retrieval benchmarks such as MSMarco (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019). Also, such datasets do not focus on retrieving an exhaustive document set, instead limiting annotation to the top few results of a baseline information retrieval system.
To analyze retrieval system performance on such queries, we present QUEST, a dataset with natural language queries from four domains, that are mapped to relatively comprehensive sets of entities corresponding to Wikipedia pages. We use categories and their mapping to entities in Wikipedia as a building block for our dataset construction approach, but do not allow access to this semistructured data source at inference time, to simulate text-based retrieval. Wikipedia categories represent a broad set of natural language descriptions of entity properties and often correspond to selective information need queries that could be plausibly issued by a search engine user. The relationship between property names and document text is often subtle and requires sophisticated reasoning to determine, representing the natural language inference challenge inherent in the task.
Our dataset construction process is outlined in Figure 1. The base queries are semi-automatically generated using Wikipedia category names. To construct complex queries, we sample category names and compose them by using pre-defined templates (for example, A ∩ B \ C). Next, we ask crowdworkers to paraphrase these automatically generated queries, while ensuring that the paraphrased queries are fluent and clearly describe what a user could be looking for. These are then validated for naturalness and fluency by a different set of crowdworkers, and filtered according to those criteria. Finally, for a large subset of the data, we collect scalar relevance labels based on the entity documents and fine-grained textual attributions mapping query constraints to spans of document text. Such annotation could aid the development of systems that can make precise inferences from trusted sources.
Performing well on this dataset requires systems that can match query constraints with corresponding evidence in documents and handle set operations implicitly specified by the query (see
erations are not well represented in existing re-061 trieval benchmarks such as MSMarco (Nguyen062 et al., 2016) and Natural Questions (Kwiatkowski063 et al., 2019). Also, such datasets do not focus on064 retrieving an exhaustive document set, instead lim-065 iting annotation to the top few results of a baseline066 information retrieval system.067
To analyze how well retrieval system handle068 such queries, we present QUEST, a dataset with069 natural language queries from four domains, that070 are mapped to relatively comprehensive sets of en-071 tities corresponding o Wikipedia p ges. We use072 Wikipedia cat gories and their mapping to entities073 in Wikipedia as a building block for our datas t074 const uction appr ach, but do not allow acc ss to075 this semi-structured data source at inference time,076 to simulate text-based retrieval. Wikipedia cate-077 gories represent a broad set of natural language078 descriptions of entity properties and often corre-079 spond to selective information need queries that080 could be plausibly issued by a search engine user081 ([At least 90% of the time based on our filtering?]).082 The correspondence between property names and083 document text is also often subtle and requires so-084 phisticated reasoning to determine relevance, rep-085 resenting the natural language inference challenge086 inherent in the task, while the knowledge of cate-087 gory membership allows us to construct relatively088 comprehensive sets of candidate entities for atomic089 categories and their combinations.090
Our dataset construction process is outlined in091 Figure 1. The base queries in our dataset are092 semi-automatically generated using Wikipedia cat-093 egory names. To construct queries, we sample094 category names and compose them into complex095 queries by using pr -defined templat s (for exam-096 ple, A \ B \ C). Next, we ask crowdwork rs to097 par phrase these a tomatic lly generated queries,098 while nsuring that the p raphras queries are099 fluent and clearly describe what a u r could be100 looking for. These ar then validated for atural-101 ness and fluency by a different set of crowdworkers,102 and filtered according to those criteria. Finally, for103 a large subset of our dataset, we collect scalar rel-104 evance labels based on the entity documents, and105 textual attributions mapping query constraints to106 spans of document text, to aid the development of107 systems that ca ke precise inference based on108 trusted sources.109
Performing well on this dataset requires sys-110 tems that can match query constraints with cor-111
Query: Supernatural horror films set in Nevada Entity: Willy’s Wonderland Document Text: Willy’s Wonderland is a 2021 American action comedy horror film directed by Kevin Lewis from a screenplay by G. O. Parsons. The film stars Nicolas Cage, who also [...] It follows a quiet drifter who is tricked into cleaning up an abandoned family entertainment center haunted by eight murderous animatronic characters. The project was announced in October 2019, [...] When his car catches a flat tire on a remote country road, a quiet drifter ends up stranded outside of Hayesville, Nevada. Mechanic Jed Love picks [...]
Figure 2: An example of a query and relevant entity from QUEST. The attribution for different query constraints can come from different parts of the document.
responding evidence in documents and handle set 112 operations implicitly specified by the query (see 113 Figure 2), while also efficiently scaling to large 114 collections of entities. We evaluate several re- 115 trieval systems by finetuning pretrained models on 116 our dataset. Systems are trained to retrieve multi- 117 answer sets given a query. We find that current dual 118 encoder and cross-attention models up to the size 119 of T5-Large (Raffel et al., 2020) are largely not 120 effective at performing retrieval for queries with 121 set operations. Queries with conjunctions and nega- 122 tions prove to be especially challenging for models 123 and systems are further challenged with combina- 124 tions of set operations. Our error analysis reveals 125 that non-relevant false positive entities are often 126 caused by the model ignoring negated constraints, 127 or ignoring the conjunctive constraints in a query. 128
2 Related Work 129
Previous work in question answering and informa- 130 tion retrieval has focused on QA over knowledge 131 bases as well as open-domain QA and retrieval over 132 a set of entities or documents. We highlight how 133 these relate to our work below. 134
Knowledge Base QA Several datasets have been 135 proposed for question answering over knowledge 136 bases (Berant et al., 2013; Yih et al., 2016; Tal- 137 mor and Berant, 2018; Keysers et al., 2020; Gu 138 et al., 2021, inter alia). These benchmarks re- 139 quire retrieval of a set of entities that exist as nodes 140 or relations in an accompanying knowledge base. 141 Questions are optionally supplemented with logical 142 forms. Lan et al. (2021) provide a comprehensive 143
2
erations are not well represen ed in existing re-061 al be chm rks such as MSMarco (Nguyen062 et al., 2016) and Natural Questions (Kwiatkowski063 et al., 2019). Also, such datasets do not focus on064 retrieving an exhaustive document set, instead lim-065 iting annotation to the top few results of a baseline066 information retrieval system.067
To analyze how well retrieval systems handle068 such queries, we present QUEST, a dataset with069 natural language queries from four domains, that070 are mapped to relatively comprehensive sets of en-071 tities corresponding to Wikipedia pages. We use072 Wikipedia categories and their mapping to entities073 in Wikipedia as a building block for our dataset074 construction approach, but do ot allow access to075 this semi-structured data sourc at inferenc time,076 to simulate text-based retri val. Wikipedia cate-077 gories represent a broad set of natural language078 descriptions of entity properties and often corre-079 spond to selective informati n ne d queries that080 could be pla sibly issued by a search e ine us081 ([At l ast 90% of the time based on our filtering?]).082 Th correspondence between property names and083 document text is also often subtle and requires so-084 phisticated reasoning to determine relevance, rep-085 resenting the natural language inference challenge086 inherent in the task, while the knowledge of cate-087 gory membership allows us to construct relatively088 comprehensive sets of candidate entities for atomic089 categories and their combinations.090
Our dataset construction process is outlined in091 Figure 1. The base queries in our dataset are092 semi-automatically generated using Wikipedia cat-093 egory names. To construct queries, we sample094 category names and compose them into complex095 queries by using pre-defined templates (for exam-096 ple, A \ B \ C). Next, we ask crowdworkers to097 paraphrase these utomatically generated queries,098 while ensuring that the paraphrased queries a e099 fluent and learly describe what a user co ld be100 looking for. These are then validate for natural-101 nes and fluency by a different set of crowdworkers,102 and filtered according to those criteria. Finally, for103 a large subset of our dataset, we collect scalar rel-104 evance labels based on the entity documents, and105 textual attributions mapping query constraints to106 spans of document text, to aid the development of107 syste s that can make precise inferences ba ed on108 trusted sources.109
Performing well on this dataset requires sys-110 tems that can match query constraints with cor-111
Query: Supernatural horror films s t in Nevada Entity: Willy’s Wonderland Document Text: Willy’s Wonderland is a 2021 American action comedy horror film directed by Kevin Lewis from a screenplay by G. O. Parsons. The film stars Nicolas Cage, who also [...] It follows a quiet drifter who is tricked into cleaning up an abandoned family entertainment center haunted by eight murderous animatronic characters. The project was announced in October 2019, [...] When his car catches a flat tire on a remote country road, a quiet drifter ends up stranded outside of Hayesville, Nevada. Mechanic Jed Love picks [...]
Figure 2: An example of a query and relevant entity from QUEST. The attribution for different query constraints can come from different parts of the document.
responding evidence in documents and handle set 112 operations implicitly specified by the query (see 113 Figure 2), while also efficiently scaling to large 114 collections of entities. We evaluate several re- 115 trieval systems by finetuning pretrained models on 116 our dataset. Systems are trained to retrieve multi- 117 answer sets given a query. We find that current dual 118 encoder and cross-attention models up to the size 119 of T5-Large (Raffel et al., 2020) are largely not 120 effective at perfo m ng retrieval f querie with 121 set operations. Queries with conjunctions and nega- 122 tions prove to be especially challenging for models 123 and systems are further challenged with combina- 124 tions of set operations. Our error analysis reveals 125 that non-relevant false positive entities are often 126 caused by the model ignoring negated constraints, 127 or ign ring the co junctive constraints in query. 128
2 Related Work 129
Previous work in question answering and informa- 130 tion retrieval has focused on QA over knowledge 131 bases as well as open-domain QA and retrieval over 132 a set of entities or documents. We highlight how 133 these relate to our work below. 134
Knowledge Base QA Several datasets have been 135 proposed for question answering over knowledge 136 bases (Berant et al., 2013; Yih et al., 2016; Tal- 137 mor and Berant, 2018; Keysers et al., 2020; Gu 138 et al., 2021, inter alia). These benchmarks re- 139 quire retrieval of a set of entities that exist as nodes 140 or relations in an accompanying knowledge base. 141 Questions are optionally supplemented with logical 142 forms. Lan et al. (2021) provide a comprehensive 143
Large Document
Corpus . . .
. . .
. . . . . . . . . . . .
. . . .
. . . .
Retrieved Set
Retrieval System
Figure 2: An example of a query and relevant entity from QUEST. The attribution for different query constraints can come from different parts of the document.
Figure 2), while also efficiently scaling to large collections of entities. We evaluate several retrieval systems by fin tun ng pretr ined models on ur dataset. System are trained to retri ve multidocument sets given a query. We find that current dual encoder and cross-attention models up to the size of T5-Large (Raffel et al., 2020) are largely not effective at performing retrieval for queries with set operations. Queries with conjunctions and negations prove to be especially challenging for models and systems are further challenged with combinations of set operations. Our error analysis reveals that non-relevant false positive entities are often caused by the model ignoring negated constraints, or ignoring the conjunctive constraints in a query.

heading: 2 Related Work
text: Previous work in question answering and information retrieval has focused on QA over kn wledge bases as well as open-domain QA and retrieval over a set of entities or d cuments. We highlight how these relate to our work below.
Knowledge Base QA Several datasets have been pr posed for question answering over knowledge bases (Berant t al., 2013; Yih et al., 2016; Talmor and Berant, 2018; Keysers et al., 2020; Gu et al., 2021, inter alia). These benchmarks require retrieval of a set of entities that exist as nodes
14033
or relations in an accompanying knowledge base. Questions are optionally supplemented with logical forms. Lan et al. (2021) provide a comprehensive survey of complex KBQA datasets.
Previous work has simultaneously noted that large curated KBs are incomplete (Watanabe et al., 2017). Notably, KBQA systems operate over a constrained answer schema, which limits the types of queries they can handle. Further, these schema are expensive to construct and maintain. For this reason, our work focuses on a setting where we do not assume access to a KB. We note that KBQA datasets have also been adapted to settings where a KB is incomplete or unavailable (Watanabe et al., 2017; Sun et al., 2019). This was done by either removing some subset of the data from the KB or ignoring the KB entirely. A key difference from these datasets is also that we do not focus on multihop reasoning over multiple documents. Instead, the relevance of an entity can be determined solely based on its document.
Open-Domain QA and Retrieval Many opendomain QA benchmarks, which consider QA over unstructured text corpora, have been proposed in prior work. Some of these, such as TREC (Craswell et al., 2020), MSMarco (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019) are constructed using "found data", using real user queries on search engines. Thakur et al. (2021) present a benchmark where they consider many such existing datasets. Datasets such as HotpotQA (Yang et al., 2018), and MultiRC (Khashabi et al., 2018) have focused on multi-hop question answering. Other work has explored e-commerce datasets (for example, (Kong et al., 2022)), but these have not been released publicly. Notably, the focus of these datasets differs from ours as we focus on queries that contain implicit set operations over exhaustive answer sets. Such queries are not well represented in existing datasets because they occur in the tail of the query distributions considered.
Multi-Answer Retrieval Related work (Min et al., 2021; Amouyal et al., 2022) also studies the problem of multi-answer retrieval, where systems are required to predict multiple distinct answers for a query. Min et al. (2021) adapt existing datasets (for example, WebQuestionsSP (Yih et al., 2016)) to study this setting and propose a new metric, MRecall@K, to evaluate exhaustive recall of multiple answers. We also consider the problem of multi-answer set retrieval, but consider queries that
implicitly contain set constraints. In concurrent work, RomQA (Zhong et al., 2022) proposes an open-domain QA dataset, focusing on combinations of constraints extracted from Wikidata. RomQA shares our motivation to enable answering queries with multiple constraints, which have possibly large answer sets. To make attribution to evidence feasible without human annotation, RomQA focuses on questions whose component constraints can be verified from single entity-linked sentences from Wikipedia abstracts, annotated with relations automatically through distant supervision, with high precision but possibly low recall (T-Rex corpus). In QUEST, we broaden the scope of queryevidence matching operations by allowing for attribution through more global, document-level inference. To make human annotation for attribution feasible, we limit the answer set size and the evidence for an answer to a single document.

heading: 3 Dataset Generation
text: QUEST consists of 3357 queries paired with up to 20 corresponding entities. Each entity has an associated document derived from its Wikipedia page. The dataset is divided into 1307 queries for training, 323 for validation, and 1727 for testing.
The task for a system is to return the correct set of entities for a given query. Additionally, as the collection contains 325,505 entities, the task requires retrieval systems that can scale efficiently. We do not allow systems to access additional information outside of the text descriptions of entities at inference time. Category labels are omitted from all entity documents.

heading: 3.1 Atomic Queries
text: The base atomic queries (i.e., queries without any introduced set operations) in our dataset are derived from Wikipedia category names2. These are handcurated natural language labels assigned to groups of related documents in Wikipedia3. Category assignments to documents allow us to automatically determine the set of answer entities for queries with high precision and relatively high recall. We compute transitive closures of all relevant categories to determine their answer sets.
However, repurposing these categories for constructing queries poses challenges: 1) lack of evi-
2We use the Wikipedia version from 06/01/2022. 3Note that these category labels can sometimes be conjunc-
tive themselves, potentially increasing complexity.
dence in documents: documents may not contain sufficient evidence for judging their relevance to a category, potentially providing noisy signal for relevance attributable to the document text, 2) low recall: entities may be missing from categories to which they belong. For about half of the dataset, we crowdsource relevance labels and attribution based on document text, and investigate recall through manual error analysis (§5).
We select four domains to represent some diversity in queries: films, books, animals and plants. Focusing on four rather than all possible domains enables higher quality control. The former two model a general search scenario, while the latter two model a scientific search scenario.

heading: 3.2 Introducing set operations
text: To construct queries with set operations, we define templates that represent plausible combinations of atomic queries. Denoting atomic queries as A, B and C, our templates and corresponding examples from different domains are listed in Table 1. Templates were constructed by composing three basic set operations (intersection, union and difference). They were chosen to ensure unambiguous interpretations of resulting queries by omitting those combinations of set operations that are non-associative.
Below we describe the logic behind sampling atomic queries (i.e., A, B, C) for composing complex queries, with different set operations. In all cases, we ensure that answer sets contain between 2-20 entities so that crowdsourcing relevance judgements is feasible. We sample 200 queries per template and domain, for a total of 4200 initial queries. The dataset is split into train + validation (80-20 split) and testing equally. In each of these sets, we sampled an equal number of queries per template. Intersection. The intersection operation for a template A∩B is particularly interesting and potentially challenging when both A and B have large answer sets but their intersection is small. We require the minimum answer set sizes of each A and B to be fairly large (>50 entities), while their intersection to be small (2-20 entities). Difference. Similar to intersection, we require the answer sets for both A and B to be substantial (>50 entities), but also place maximum size constraints on both A (<200 entities) and B (<10000 entities) as very large categories tend to suffer from recall issues in Wikipedia. We also limit the intersection of A and B (see reasoning in Appendix B). Union. For the union operation, we require both A and B to be well-represented through the entities in the answer set for their union A ∪ B. Hence, we require both A and B to have at least 3 entities. Further, we require their intersection to be non-zero but less than 1/3rd of their union. This is so that A and B are somewhat related queries.
For all other templates that contain compositions of the above set operations, we apply the same constraints recursively. For example, for A∩B\C, we sample atomic queries A and B for the intersection operation, then sample C based on the relationship between A ∩B and C.

heading: 3.3 Annotation Tasks
text: Automatically generating queries based on templates results in queries that are not always fluent and coherent. Further, entities mapped to a query may not actually be relevant and don’t always have attributable evidence for judging their relevance. We conduct crowdsourcing to tackle these issues. The annotation tasks aim at ensuring that 1) queries are fluent, unambiguous and contain diverse natural language logical connectives, (2) entities are verified as being relevant or non-relevant and (3) relevance judgements are attributed to document text for each relevant entity. Crowdsourcing is performed in three stages, described below. More annotation details and the annotation interfaces can be found in Appendix C.

heading: 3.3.1 Paraphrasing
text: Crowdworkers were asked to paraphrase a templatically generated query so that the paraphrased query is fluent, expresses all constraints in the original query, and clearly describes what a user could be looking for. This annotation was done by one worker per query.

heading: 3.3.2 Validation
text: This stage is aimed at validating the queries we obtain from the paraphrasing stage. Crowdworkers were given queries from the first stage and asked to label whether the query is 1) fluent, 2) equivalent to the original templatic query in meaning, and 3) rate its naturalness (how likely it is to be issued by a real user). This annotation was done by 3 workers per query. We excluded those queries which were rated as not fluent, unnatural or having a different meaning than the original query, based on a ma-
jority vote. Based on the validation, we removed around around 11% of the queries from stage 1.

heading: 3.3.3 Relevance Labeling
text: Next, crowdworkers were asked to provide relevance judgements for the automatically determined answer sets of queries. Specifically, they were given a query and associated entities/documents, and asked to label their relevance on a scale of 0-3 (definitely not relevant, likely not relevant, likely relevant, definitely relevant). They were asked to ensure that relevance should mostly be inferred from the document, but they could use some background knowledge and do minimal research.
We also asked them to provide attributions for document relevance. Specifically, we ask them to first label whether the document provides sufficient evidence for the relevance of the entity (complete/partial/no). Then, for different phrases in the query (determined by the annotator), we ask them to mark sentence(s) in the document that indicate its relevance. The attribution annotation is broadly inspired by Rashkin et al. (2021). For negated constraints, we ask annotators to mark attributable sentences if they provide counter-evidence. Since this annotation was time-intensive, we collected these annotations for two domains (films and books). We found that relevance labeling was especially difficult for the plants and animals domains, as they required more specialized scientific knowledge. In our pilot study prior to larger scale data collection, we collected 3 relevance ratings from different annotators for 905 query and document pairs from the films domain. In 61.4% of cases, all 3 raters judged the document to be “Definitely relevant” or “Likely relevant” or all 3 raters judged the document to be “Definitely not relevant” or “Likely not relevant”. The Fleiss’ kappa metric on this data was found to be K=0.43. We excluded all entities which were marked as likely or definitely not relevant to a query based on the document text from its answer set. Around 23.7% of query-document pairs from stage 2 were excluded.

heading: 3.4 Dataset Statistics
text: Basic dataset statistics are reported in Table 2. The dataset contains more entities from the films domain, because this domain is more populated in Wikipedia. The average length of queries is 8.6 words and the average document length is 452 words. Documents from the films and books domains are longer on average, as they often contain plots and storylines. Around ∼69% of entities have complete evidence and ∼30% have partial evidence. Evidence was labeled as partial when not all phrases in the query had explicit evidence in the document (i.e., they may require background knowledge or reasoning). There are on average 33.2 words attributed for each entity with the maximum attribution text span ranging up to length 1837 words. Finally, the average answer set size is 10.5 entities.

heading: 3.5 Additional Training Examples
text: Beyond the annotated data, we generated additional synthetic examples for training. We found including such examples improved model performance, and we include these examples for the experiments in §4. To generate these examples, we sample 5000 atomic queries from all domains, ensuring that they do not already appear as sub-queries in any of the queries in QUEST and use their corresponding entities in Wikipedia as their relevant entity set.

heading: 4 Experimental Setup
text: We evaluate modern retrieval systems to establish baseline performances. We also perform extensive error analysis to understand patterns of model errors and the quality of the labels in QUEST.

heading: 4.1 Task Definition
text: We consider a corpus, E , that contains entities across all domains in the dataset. Each entity is accompanied with a document based on its Wikipedia page. An example in our dataset consists of a query,
x, and an annotated set of relevant entities, y ⊂ E . As described in §3, for all examples |y| < 20. Our task is to develop a system that, given E and a query x, predicts a set of relevant entities, ŷ ⊂ E .

heading: 4.2 Evaluation
text: Our primary evaluation metric is average F1, which averages per-example F1 scores. We compute F1 for each example by comparing the predicted set of entities, ŷ, with the annotated set, y.

heading: 4.3 Baseline Systems
text: We evaluated several combinations of retrievers and classifiers, as shown in Figure 3. For the retriever component, we consider a sparse BM25 retriever (Robertson et al., 2009) and a dense dual encoder retriever (denoted DE). Following Ni et al. (2022), we initialize our dual encoder from a T5 (Raffel et al., 2020) encoder and train with an in-batch sampled softmax loss (Henderson et al., 2017). Once we have a candidate set, we need to determine a set of relevant entities. To classify relevance of each candidate document for the given query, we consider a cross-attention model which consists of a T5 encoder and decoder.4 We train the cross-attention classifier using a binary cross-entropy loss with negative examples based on non-relevant documents in top 1,000 documents retrieved by BM25 and random non-relevant documents (similarly to Nogueira and Cho (2019)). As cross-attention classification for a large number of candidates is computationally expensive, we restrict BM25 and the dual encoder to retrieve 100 candidates which are then considered by the crossattention classifier. As our T5-based dual encoder can only efficiently accommodate up to 512 tokens,
4Scores from BM25 and dual encoders trained with a softmax loss are not normalized to provide relevance probabilities for documents. We found that naively applying a global threshold to these scores to produce answer sets did not perform as well as using a classifier trained with a binary cross-entropy loss to predict document relevance.
we truncate document text. We discuss the impact of this and alternatives in §5. Further, since T5 was pre-trained on Wikipedia, we investigate the impact of memorization in Appendix D. Additional details and hyperparameter settings are in Appendix A.

heading: 4.4 Manual Error Annotation
text: For the best overall system, we sampled errors and manually annotated 1145 query-document pairs from the validation set. For the retriever, we sampled relevant documents not included in the top-100 candidate set and non-relevant documents ranked higher than relevant ones. For the classifier, we sampled false positive and false negative errors made in the top-100 candidate set. This annotation process included judgements of document relevance (to assess agreement with the annotations in the dataset) and whether the document (and the truncated version considered by the dual encoder or classifier) contained sufficient evidence to reasonably determine relevance. We also annotated relevance for each constraint within a query. We discuss these results in §5.

heading: 5 Results and Analysis
text: We report the performance of our baseline systems on the test set in Table 3. In this section, we summarize the key findings from our analysis of these results and the error annotation described in §4.4.
Dual encoders outperform BM25. As shown in Table 3, the best overall system uses a T5Large Dual Encoder instead of BM25 for retrieval. The performance difference is even more significant when comparing recall of Dual Encoders and BM25 directly. We report average recall (average
per-example recall of the full set of relevant documents) and MRecall (Min et al., 2021) (the percentage of examples where the candidate set contains all relevant documents), over various candidate set sizes in Table 4. Retrieval and classification are both challenging. As we consider only the top-100 candidates from the retriever, the retriever’s recall@100 sets an upper bound on the recall of the overall system. Recall@100 is only 0.476 for the T5-Large Dual Encoder, and the overall recall is further reduced by the T5-Large classifier to 0.368, despite achieving only 0.165 precision. This suggests that there is room for improvement from both stages to improve overall scores. As performance improves for larger T5 sizes for both retrieval and classification, further model scaling could be beneficial. Models struggle with intersection and difference. We also analyzed results across different templates and domains, as shown in Table 5. Different constraints lead to varying distributions over answer set sizes and the atomic categories used. Therefore, it can be difficult to interpret differences in F1 scores across templates. Nevertheless, we found the queries with set union have the highest average F1 scores. Queries with set intersection have the lowest average F1 scores, and queries with set difference also appear to be challenging.
To analyze why queries with conjunction and negation are challenging, we labeled the relevance of individual query constraints (§4.4), where a system incorrectly judges relevance of a non-relevant document. The results are summarized in Table 6. For a majority of false positive errors involving intersection, at least one constraint is satisfied. This could be interpreted as models incorrectly treating
intersection as union when determining relevance. Similarly, for a majority of examples with set difference, the negated constraint is not satisfied. This suggests that the systems are not sufficiently sensitive to negations.

heading: A 0.231 0.436 0.209 0.214 0.274

heading: A ∪B 0.264 0.366 0.229 0.271 0.282

heading: A ∩B 0.115 0.138 0.049 0.063 0.092

heading: A \B 0.177 0.188 0.216 0.204 0.193

heading: A ∪B ∪ C 0.200 0.348 0.306 0.294 0.287

heading: A ∩B ∩ C 0.086 0.121 0.07 0.065 0.086

heading: A ∩B \ C 0.119 0.112 0.121 0.136 0.122
text: There is significant headroom to improve both precision and recall. As part of our manual error analysis (§4.4), we made our own judgements of relevance and measured agreement with the relevance annotations in QUEST. As this analysis focused on cases where our best system disagreed with the relevance labels in the dataset, we would expect agreement on these cases to be significantly lower than on randomly selected query-document pairs in the dataset. Therefore, it provides a focused way to judge the headroom and annotation quality of the dataset.
For false negative errors, we judged 91.1% of the entities to be relevant for the films and books domains, and 81.4% for plants and animals. Notably, we collected relevance labels for the films and books domains and removed some entities based on these labels, as described in §3, which likely explains the higher agreement for false negatives from these domains. This indicates significant headroom for improving recall as defined by QUEST, especially for the domains where we collected relevance labels.
For false positive errors, we judged 28.8% of the entities to be relevant, showing a larger disagreement with the relevance labels in the dataset. This is primarily due to entities not included in the entity sets derived from the Wikipedia category taxonomy (97.7%), rather than entities removed due to relevance labeling. This is a difficult issue to fully resolve, as it is not feasible to exhaustively label relevance for all entities to correct for recall issues in the Wikipedia category taxonomy. Future work can use pooling to continually grow the set
of relevant documents (Sparck Jones and Van Rijsbergen, 1975). Despite this, our analysis suggests there is significant headroom for improving precision, as we judged a large majority of the false positive predictions to be non-relevant.
Truncating document text usually provides sufficient context. In our experiments, we truncate document text to 512 tokens for the dual encoder, and 384 tokens for the classifier to allow for the document and query to be concatenated. Based on our error analysis (§4.4), out of the documents with sufficient evidence to judge relevance, evidence occurred in this truncated context 93.2% of the time for the dual encoder, and 96.1% of the time for the classifier. This may explain the relative success of this simple baseline for handling long documents. We also evaluated alternative strategies but these performed worse in preliminary experiments5. Future work can evaluate efficient transformer variants (Guo et al., 2022; Beltagy et al., 2020).

heading: 6 Conclusion
text: We present QUEST, a new benchmark of queries which contain implicit set operations with corresponding sets of relevant entity documents. Our experiments indicate that such queries present a
5For the dual encoder, we split documents into overlapping chunks of 512 tokens, and aggregated scores at inference (Dai and Callan, 2019). For the cross-attention model, we evaluated using BM25 to select the top-3 passages of length 128.
challenge for modern retrieval systems. Future work could consider approaches that have better inductive biases for handling set operations in natural language expressions (for example, Vilnis et al. (2018)). The attributions in QUEST can be leveraged for building systems that can provide finegrained attributions at inference time. The potential of pretrained generative LMs and multi-evidence aggregation methods to answer set-seeking selective queries, while providing attribution to sources, can also be investigated.

heading: 7 Limitations
text: Naturalness. Since our dataset relies on the Wikipedia category names and semi-automatically generated compositions, it does not represent an unbiased sample from a natural distribution of real search queries that contain implicit set operations. Further, we limit attention to non-ambiguous queries and do not address the additional challenges that arise due to ambiguity in real search scenarios. However, the queries in our dataset were judged to plausibly correspond to real user search needs and system improvements measured on QUEST should correlate with improvements on at least a fraction of natural search engine queries with set operations.
Recall. We also note that because Wikipedia categories have imperfect recall of all relevant entities (that contain sufficient evidence in their documents), systems may be incorrectly penalised for predicted relevant entities assessed as false positive. We quantify this in section 5. We have also limited the trusted source for an entity to its Wikipedia document but entities with insufficient textual evidence in their documents may still be relevant. Ideally, multiple trusted sources could be taken into account and evidence could be aggregated to make relevance decisions. RomQA (Zhong et al., 2022) takes a step in this latter direction although the evidence attribution is not manually verified.
Answer Set Sizes. To ensure that relevance labels are correct and verifiable, we seek the help of crowdworkers. However, this meant that we needed to restrict the answer set sizes to 20 for the queries in our dataset, to make annotation feasible. On one hand, this is realistic for a search scenario because users may only be interested in a limited set of results. On the other hand, our dataset does not model a scenario where the answer set sizes are much larger.

heading: Acknowledgements
text: We would like to thank Isabel Kraus-Liang, Mahesh Maddinala, Andrew Smith, Daphne Domansi, and all the annotators for their work. We would also like to thank Mark Yatskar, Dan Roth, Zhuyun Dai, Jianmo Ni, William Cohen, Andrew McCallum, Shib Sankar Dasgupta and Nicholas Fitzgerald for useful discussions.

heading: A Experiment Details and Hyperparameters
text: All models were fine-tuned starting from T5 1.1 checkpoints 6. We fine-tune T5 models on 32 Cloud TPU v3 cores7. Fine-tuning takes less than 8 hours for all models.
Dual Encoder. We used the t5x_retrieval library 8 for implementing dual encoder models. We tuned some parameters based on results on the validation set. Relevant hyperparameters for training the dual encoder are:
• Learning Rate: 1e-3
• Warmup Steps: 1500
• Finetuning Steps: 15000
• Batch Size: 512
• Max Query Length: 64
• Max Candidate Length: 512
Classifier. For negative examples, we sampled 250 random non-relevant documents and sampled 250 non-relevant documents from the top-1000 documents retrieved by BM25. We also replicated each positive example 50 times. We found an approximately even number of positive and negative examples lead to better performance than training with a large class imbalance. We found a combination of random negatives and negatives from BM25 performed better than using only either individual type of negative examples. Additionally, selecting negative examples from BM25 performed better than selecting negative examples from the T5-Large dual encoder.
For the T5 input we concatenated the query and truncated document text. The T5 output is the string “relevant” or “not relevant”. To classify document relevance at inference time, we applied a threshold to the probability assigned to the “relevant” label, which we tuned on the validation set. When classifying BM25 candidates we used a threshold of 0.9 and when classifying the dual encoder candidates we used a threshold of 0.95.
Other relevant hyperparameters for training the classifier are:
6https://github.com/googleresearch/t5x/blob/main/docs/models.md
7https://cloud.google.com/tpu/ 8https://github.com/google-research/t5x_retrieval
• Learning Rate: 1e-3
• Warmup Steps: 1000
• Finetuning Steps: 10000
• Batch Size: 1024
• Max Source Length: 512
• Max Target Length: 16

heading: B Set Difference and Recall
text: Notation and Assumptions Let us assume we have two sets derived from the Wikipedia category graph, Â and B̂. The Wikipedia category graph can be missing some relevant entities, such that Â ⊂ A and B̂ ⊂ B, where A and B are interpreted as the hypothetical sets containing all relevant entities. We quantify the degree of missing entities by denoting recall as rA and rB , such that |Â| = rA ∗ |A| and |B̂| = rB ∗ |B|. We quantify the fraction of elements in A that are also in B as r∩, such that |A ∩ B| = r∩ ∗ |A|. For simplicity, we also assume that the overlap between Â and B̂ is such that |Â ∩B| = rA ∗ |A ∩B| and |Â ∩ B̂| = rA ∗ rB ∗ |A ∩B|. Derivation What is the recall (r) and precision (p) of Â \ B̂ relative to A \B as a function of rA, rB , and r∩?
First, we derive this function for recall:9
r = |(A \B) ∩ (Â \ B̂)|
|(A \B)|
r = |(Â \B)| |(A \B)|
r = |Â| − |Â ∩B| |A| − |A ∩B|
r = rA ∗ |A| − rA ∗ r∩ ∗ |A| |A| − (r∩ ∗ |A|)
r = rA ∗ (1− r∩) ∗ |A|
(1− r∩) ∗ |A| r = rA
And for precision:
p = |(A \B) ∩ (Â \ B̂)|
|(Â \ B̂)| 9We note some useful properties of pairs of sets X and Y : X \ Y = X ∩ Y c, |X \ Y | = |X| − |X ∩ Y |, if X ⊂ Y then X ∩ Y = X , and if X ⊂ Y then Y c ⊂ Xc.
p = |(Â \B)| |(Â \ B̂)|
p = |Â| − |Â ∩B| |Â| − |Â ∩ B̂|
p = rA ∗ |A| − rA ∗ r∩ ∗ |A|
rA ∗ |A| − rA ∗ rB ∗ r∩ ∗ |A|
p = rA ∗ (1− r∩) ∗ |A|
rA ∗ (1− rB ∗ r∩) ∗ |A|
p = (1− r∩)
(1− rB ∗ r∩) Discussion While recall is simply equal to rA, precision is a more complicated function of rB and r∩, and can be very low for large values of r∩. Intuitively, if subtracting B̂ from Â removes most of Â, then the precision of the resulting set will be dominated by the relevant entities missing from B̂. This motivates limiting the intersection of the two sets used to construct queries involving set intersection. For example, if rB = 0.95, then with r∩ < 0.8, we can ensure p > 0.83.

heading: C Annotation Details
text: The annotation tasks in QUEST were carried out by participants who were paid contractors. They are based in Austin, TX and either have a bachelor’s degree (55%) or equivalent work experience (45%). They were paid by the hour for their work and were recruited from a vendor who screened them for knowledge of US English. They were informed of how their work would be used and could opt out. They received a standard contracted wage, which complies with living wage laws in their country of employment. The annotation interfaces presented to the annotators are shown in Figures 4, 5 and 6.
D Impact of Memorization of Pre-training Data
Since the T5 checkpoints we use to initialize our models were pre-trained on the C4 corpus (which includes Wikipedia), we investigate whether these models have memorized aspects of the Wikipedia category graph. We compare recall of the T5-based dual encoder model for Wikipedia documents that were created prior to the pre-training date of the T5 checkpoint compared with documents that were added after pre-training. We report these in Table 7, along with the recalls for the same sets of documents with a BM25 retriever, for a baseline
comparison. We note that the ratio of scores between the documents added before pre-training to documents added after pre-training is similar for both systems, which suggests factors other than memorization may explain the difference. For example, the documents created before vs. after the pre-training date have average lengths of 759.7 vs. 441.2 words, respectively.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. We did not identify any risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We will release the code and our dataset publicly.
3 B1. Did you cite the creators of artifacts you used? Appendix A
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Yes, we will use an MIT license to release our dataset.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix A
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Appendix C
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix C
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix C
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix C
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix C

heading: Future Work
text: Future work could consider approaches that have better inductive biases for handling set operations in natural language expressions (for example, Vilnis et al. (2018)). The attributions in QUEST can be leveraged for building systems that can provide finegrained attributions at inference time. The potential of pretrained generative LMs and multi-evidence aggregation methods to answer set-seeking selective queries, while providing attribution to sources, can also be investigated