SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DORIS-MAE:
Scientific Document Retrieval using Multi-level
Aspect-based Queries
Jianyou Wang * Kaicheng Wang * Xiaoyue Wang * Prudhviraj Naidu
Leon Bergen
Ramamohan Paturi†
Laboratory for Emerging Intelligence
La Jolla, CA 92093
{jiw101, kaw036, xiw027, prnaidu, lbergen, rpaturi} @ ucsd.edu
In scientific research, the ability to effectively retrieve relevant documents based on complex, multifaceted queries is critical. Existing evaluation datasets for this task are limited, primarily due to the high cost and effort required to annotate resources that effectively represent complex queries. To address this, we propose a novel task, Scientific DOcument Retrieval using Multi-level Aspect-based quEries (DORIS-MAE), which is designed to handle the complex nature of user queries in scientific research. We developed a benchmark dataset within the field of computer science, consisting of 100 human-authored complex query cases. For each complex query, we assembled a collection of 100 relevant documents and produced annotated relevance scores for ranking them. Recognizing the significant labor of expert annotation, we also introduce Anno-GPT, a scalable framework for validating the performance of Large Language Models (LLMs) on expert-level dataset annotation tasks. LLM annotation of the DORIS-MAE dataset resulted in a  $500\mathrm{x}$  reduction in cost, without compromising quality. Furthermore, due to the multi-tiered structure of these complex queries, the DORIS-MAE dataset can be extended to over 4,000 sub-query test cases without requiring additional annotation. We evaluated 17 recent retrieval methods on DORIS-MAE, observing notable performance drops compared to traditional datasets. This highlights the need for better approaches to handle complex, multifaceted queries in scientific research. Our dataset and codebase are available at https://github.com/Real-Doris-Mae/Doris-Mae-Dataset.
Scientists often have complex questions that require thorough exploration within various parts of their field (Figure 1). Finding relevant scientific literature, one of many challenges in this process, can be especially difficult when dealing with multi-faceted queries. These queries typically encompass numerous interconnected topics and require an information retrieval (IR) system capable of recognizing and responding to this level of complexity.
Figure 1: Example from the DORIS-MAE dataset. Each query is broken down into aspects and sub-aspects. Aspects are semantically distinct components of the query, and sub-aspects are minimal requirements that can be extracted from the aspects.
Information retrieval, especially query-based document retrieval [78, 3, 34], is integral to many applications, from search engines [9, 13, 29] and content recommendations [58, 31, 41], to open-domain question answering (QA) [82, 33, 15]. A persistent challenge, however, is the low accuracy in processing complex and multi-intent user queries. Despite advanced search engines using semantic understanding and user behavior data in addition to keyword matching [13], these systems still fall short when dealing with complex, multi-intent queries. Neural Information Retrieval (NIR) models [63, 85, 75, 23, 66, 85, 47] are primarily trained on relatively simple queries [57, 17, 64, 61, 87]. Moreover, MacAvaney et al. [49] creates a variety of diagnostic probes, revealing NIR models' instability when processing out-of-distribution textual inputs, potentially because they do not comprehend the deeper semantics of text. These limitations can lead to inadequate performance with more complex queries. While open-domain QA models like ChatGPT [11] have shown significant capability, they frequently produce incorrect or even fabricated responses [4, 46, 50, 39, 89], and are prohibitively expensive if directly applied to an entire corpus of scientific documents.
To address these challenges, we introduce a novel task, Scientific Document Retrieval for Multi-level Aspect-based quEries (DORIS-MAE). DORIS-MAE extends query-based and example-based IR paradigms [20,44], aiming to give users more control in formulating queries using natural language. In order to advance research in this area, we present the DORIS-MAE dataset, comprising 100 unique complex queries in the computer science domain, paired with ranked pools of relevant CS article abstracts. Each query is organized into a hierarchical structure of aspects/sub-aspects, which aids annotation.
A distinguishing feature of the DORIS-MAE dataset is its aspect-based hierarchical structure shown in Figure 1. This feature aids automation of the annotation process, expands the test case volume without additional annotation, and opens up exploration into the usefulness of aspect information for retrieval methods.
Furthermore, we introduce Anno-GPT, a pipeline for validating Large Language Model (LLM) annotations in a statistically sound manner. Our tests reveal that LLM annotations achieve quality comparable to those produced by human annotators, but with considerable savings in both time and cost. Additionally, the design of our pipeline lends itself to easy adaptation for different domains.
In our experiments, we evaluated 17 IR/NIR (Information Retrieval/Neural Information Retrieval) methods using the DORIS-MAE dataset. The methods have worse performance on DORIS-MAE compared to traditional document retrieval datasets, highlighting the complexity of DORIS-MAE, and the need for more sophisticated retrieval methods.
Our main contributions are three-fold. First, by formulating the DORIS-MAE task we are shedding a new light on tackling complex, multi-faceted queries during scientific research. Second, we propose Anno-GPT, a procedure for rigorously evaluating the ability of LLMs to replace human experts for challenging annotation tasks. Third, we demonstrate the value of breaking complex queries down
to an Multi-level Aspect-based hierarchical structure, for both annotation accuracy and potential improvements in retrieval methods.
A range of methods have been developed for document retrieval and re-ranking. Classic retrieval methods like TF-IDF [67] and BM25 [72] utilize keyword matching between queries and documents, and fall short when the necessary key phrases are not known to users. To address this, researchers have applied deep learning techniques to develop NIR models, including RocketQA-v2 [63], ColBERT-v2 [66], SimLM [75], and SPLADE-v2 [23]. These models generate latent vector representations for queries and documents, which are effective for many retrieval tasks, but may not be able to handle textual inputs that are outside of their training distributions [49].
Other models such as SPECTER [18] and aspect-based ASPIRE [54] focus on calculating document-level similarity. These models are designed to retrieve given an existing paper as input, which makes them less suited for open-ended queries. Other retrieval models, such as text-embedding models (ada-002 by OpenAI [28] and E5-large-v2 [74]) and sentence-embedding model Sentence-BERT [62], all face similar limitations as NIR methods in that they struggle to simultaneously represent multiple aspects of a query. Despite these challenges, they tend to perform better on our dataset than specialized NIR models for efficient dense passage retrieval, indicating their potential for complex tasks like DORIS-MAE.
Large-scale traditional IR datasets [57, 86, 40, 66, 69] primarily contain simple, web-based queries. Models performing well on these datasets often struggle on DORIS-MAE. In contrast, more specialized IR datasets [56, 14, 53] require human expert annotation, and consequently are more limited in their scale. Mysore et al. [53] and Chan et al. [14] introduce the concept of "aspect" in document retrieval tasks. These datasets provide pre-defined categories for aspects such as "background" or "method". DORIS-MAE extends this work by allowing for open-ended aspects based on the context of the queries. In Narrative-Driven Recommendation (NDR) research [6, 2, 55], user queries are descriptions that capture a range of users' needs. These queries are related to everyday tasks, such as finding restaurants and entertainment.
Progress in LLMs [11] and prompt-engineering [24, 45, 59, 79, 65] has made it feasible to leverage LLMs such as ChatGPT for annotating NLP tasks [90, 21], even outperforming crowd-workers in some cases [60, 70, 27]. However, these tasks do not necessitate domain-specific knowledge, and the annotations produced may not measure up to expert annotations. Faggioli et al. [22], MacAvaney and Soldaini [48] explored the notion of using LLMs to assist human in relevance judgements, and evaluated on TREC-DL datasets [19], which have single-faceted queries. Our work seeks to extend these efforts to new annotation tasks requiring domain expertise, while introducing a separate pipeline stage and a hypothesis testing stage.
The DORIS-MAE task uses a dataset of 100 complex, human-written queries, each containing between 95 to 226 words. Each complex query is broken down into a hierarchy of aspects and sub-aspects, with aspects representing significant semantic components of the original query, typically a sentence or a few sentences. Sub-aspects further decompose an aspect into simpler, verifiable requirements. Both aspects and sub-aspects are generally one sentence long, though their semantic complexity varies. A complex query can have up to 9 aspects, and each aspect can further contain up to 6 sub-aspects. Figure  $\boxed{I}$  shows an example.
For each complex query  $Q$ , we created a pool  $\mathcal{P}$  of approximately 100 potentially relevant scientific abstracts to evaluate the re-ranking performance of various retrieval/eranking methods. Within the context of  $Q$  and its  $\mathcal{P}$ , the collection of aspects and sub-aspects is together denoted as (with slight abuse of notation)  $Q := \{a_i\}$ . For any aspect or sub-aspect  $a_i \in Q$ , and any paper abstract  $p_j \in \mathcal{P}$ , they form a question pair  $(a_i, p_j)$ . We generated a total of 165,144 question pairs from the 100 queries, together denoted as  $\mathcal{D} := \{(a_i, p_j)\}$ .
To compute the relevance of a paper  $p_j$  for a query  $Q$ , we first break the query down into its aspects. We then compute the relevance score  $S(a_i, p_j)$ , which measures the relevance of paper  $p_j$  for aspect
$a_{i}$ . The relevance score  $S(p_{j}|Q)$  for the query  $Q$  is the sum of its aspect relevance scores. An average score  $\frac{S(p_j|Q)}{|Q|} \geq 1$  indicates  $p_j$  is a relevant abstract for  $Q$ .
$$
S \left(p _ {j} \mid Q\right) := \sum_ {a _ {i} \in Q} S \left(a _ {i}, p _ {j}\right) \tag {1}
$$
Thus, for each complex query and candidate pool, we provide a complete ranking for the list of abstracts. Additionally, we can choose a combination of aspects within a complex query  $Q$  and concatenate the corresponding sentences to form a sub-query  $q \subset Q$ , and can calculate the relevance score between a paper abstract  $p_j$  and a sub-query  $q$ .
$$
S \left(p _ {j} \mid q\right) := \sum_ {a _ {i} \in q} S \left(a _ {i}, p _ {j}\right) \tag {2}
$$
Hence, for any sub-query  $q \subset Q$  and the candidate pool  $\mathcal{P}$  of  $Q$ , we can also provide a complete ground-truth ranking order for  $\mathcal{P}$ . This allows the dataset to extend to over 4000 sub-query test cases at no additional annotation cost.
Candidate pool abstracts are taken from a database of 360,000 computer science papers between 2011-2021 sourced from arXiv. We complemented each arXiv paper with its corresponding citation information by cross-matching it on Semantic Scholar [37].
3.1 Query Formation
The DORIS-MAE task aims to mirror real-world scenarios where a researcher has an incomplete concept for a research project and needs to explore the breadth of existing research to establish a solid starting point. The 100 complex queries in our dataset simulate this scenario.
Each query is based on one or more existing research papers. We randomly selected 140 papers from AI, NLP, ML, and CV categories on arXiv. We examined each selected paper's motivation, background, related work, methodology, and experimental results. Using this information, we reverse-engineered a complex query designed to reflect the early thought process of the paper's authors. Overall, DORIS-MAE contains 80 queries derived from single paper abstracts, with 20 queries from each of AI, NLP, ML and CV. We created an additional 20 composite queries, each integrating ideas from 2-3 abstracts. These composite queries are designed to simulate more interdisciplinary and unexplored research ideas than the other 80 queries.
3.2 Decomposing Queries to Aspects
In this section, we discuss our process of decomposing queries into a list of aspects and further breaking down aspects into sub-aspects. The guiding principles for determining aspects are as follows:
(i) Each aspect must correspond to a prominent and semantically meaningful component in the query (refer to Figure 1 for this correspondence).
(ii) Each aspect should contain sufficient context to make sense independently, eliminating potential ambiguities.
(iii) Each aspect must be semantically distinct and unrelated to others, ensuring their contents are disjoint and do not overlap.
Some aspects may fall into broader facets such as background, method, or result as defined in [53]. Regarding decomposing aspects to sub-aspects, our criteria are:
(i) Each sub-aspect should not contain more information than its parent aspect.
(ii) Each sub-aspect should represent a semantic segment of its parent aspect
(iii) Different sub-aspects may overlap semantically but each should pose a distinct question.
(iv) Each sub-aspect should be as simple as possible.
<sup>3</sup>https://arxiv.org/
Given the inherent difficulty in recalling the full details of a query while ranking a large candidate pool, the aspect-based hierarchical structure of our complex queries provides a systematic, efficient, and interpretable approach for annotation. This leads to a more precise ranking of the candidate pool. As each aspect corresponds to a specific part of the original query  $Q$ , combining several aspects is equivalent to concatenating their corresponding parts in the query, forming semantically coherent sub-queries  $q \subset Q$ .
3.3 Candidate Pooling
To create a candidate pool  $\mathcal{P}$  of approximately 100 paper abstracts for each query  $Q$ , we used a variety of IR and NIR methods, similar to the approach in [53]. More specifically, we utilized popular IR search algorithms TF-IDF [67] and BM25 [72] at different granularities (i.e., sentence and paragraph-level for each query) to retrieve around 80 paper abstracts. We then employed OpenAI's text embedding model, ada-002 [28], to extract 20 more paper abstracts. We also use citation signals from semantic scholar [37] to included any papers that directly cite or are cited by any of the original papers used to create that specific query. Lastly, to prevent bias towards lexicon-matching retrieval methods, we excluded from  $\mathcal{P}$  any papers that authors referenced during query  $Q$ 's formulation. See Appendix C.6 for a sensitivity analysis of our candidate pool construction procedure.
4 Anno-GPT Framework
We propose Anno-GPT, a framework for developing a statistically sound annotation pipeline. We use chatgpt-3.5-turbo-0301 to annotate all 165,144 question pairs  $\mathcal{D} \coloneqq \{(a_i, p_j)\}$  where  $a_i$  is an aspect/sub-aspect of a query, and  $p_j$  is an abstract in the query's candidate pool. This strategy minimizes human annotation efforts. The key to this approach lies in breaking down complex queries into simpler aspects and sub-aspects, ensuring an objective and manageable annotation task. Without this structured approach, we found that evaluating an abstract's relevance to a complex query was challenging, due to variability in how partial relevance was assessed. However, the scenario changes significantly when dealing with question pairs  $(a_i, p_j)$ , which only look at a single aspect or sub-aspect of the query. In such cases, assigning a coarse-scale relevance score between 0-2 becomes feasible, maintaining a reasonable degree of objectivity.
The performance of the annotation pipeline may be influenced by several factors: the procedure for breaking down queries into aspects, the criteria used for scoring query relevance, and the LLM prompt selection [24, 45, 59, 79, 65]. In order to avoid overfitting in this pipeline, our methodology comprises two distinct development and testing stages. The development stage involves optimizing all stages of the pipeline, and using feedback and observed outcomes to iteratively refine this strategy. The testing stage uses a prespecified hypothesis test. The null hypothesis is that there is no significant difference between the agreement levels of ChatGPT and humans, and those among humans themselves. After the pipeline has been optimized, we evaluate it on a test set. We compute the difference between ChatGPT-human agreement and human-human agreement. If this difference is sufficiently close to 0, with a small confidence interval, we can use the optimized prompt  $\pi$  to annotate the full dataset.
Below we give a summary of the Anno-GPT framework:
i Construct aspect-document question pairs,  $\mathcal{D} \coloneqq \{(a_i, p_j)\}$ .
ii Select development set  $S_{\mathrm{dev}} \subset \mathcal{D}$ , and use human annotators to score  $S_{\mathrm{dev}}$ .
iii Select test set  $S_{\mathrm{test}} \subset \mathcal{D} \setminus S_{\mathrm{dev}}$  based on desired power, and use human annotators to score  $S_{\mathrm{test}}$ .
iv Optimize prompting strategy  $\pi$  and other hyperparameters on  $S_{\mathrm{dev}}$ . Fix  $\pi$ .
v Evaluate fixed  $\pi$  on  $S_{\mathrm{test}}$
vi If satisfactory performance is achieved on  $S_{\mathrm{test}}$ , proceed to use  $\pi$  to annotate the entire dataset  $\mathcal{D}$ .
vii Otherwise, repeat steps iii, iv, v for  $S_{\mathrm{dev}} \gets S_{\mathrm{dev}} \cup S_{\mathrm{test}}$  and new  $S_{\mathrm{test}}$ .
It is important to note that automated annotation for DORIS-MAE only used a single development/testing cycle, and therefore did not go into step vii. Therefore, there was no risk of inflated estimates of annotation accuracy due to multiple comparisons [30, 83, 1]. If multiple cycles are necessary, then the new  $S_{\mathrm{test}}$  must be sufficiently large to avoid these problems.
4.1 Annotation Guidelines
Our team of annotators consists of three graduate students in computer science, all with at least two years of research experience in NLP, CV, ML, and AI. Annotators, both human and ChatGPT, are asked to score each question pair using a 3-point grading scale (0-2):
- Level 0: The abstract is unrelated or does not provide any help to the key components of the aspect or sub-aspect.
- Level 1: The abstract answers some (or all) key components (either explicitly or within one natural inferential step), but at least one key component is not answered explicitly.
- Level 2: The abstract directly answers all the key components of the aspect or sub-aspect.
We decided to include both direct and indirect coverage under Level 1, acknowledging that distinguishing between these cases could be challenging and potentially subjective. The detailed guidelines for human annotation can be found in Appendix B.
4.2 Optimization of Annotation Pipeline
For the development stage, two annotators independently annotated a randomly selected set  $S_{\mathrm{dev}}$  of 90 question pairs from the complete set  $\mathcal{D}$  of 165,144 pairs. This annotated development set then served as the basis for refining the prompting strategies for ChatGPT.
The quality of annotations was evaluated using three metrics: macro F1 score, exact accuracy (agreement), and Spearman's rank correlation coefficient (Spearman's  $\rho$ ). These metrics measure the agreement level between annotators, and have been used successfully in similar tasks [53]. After satisfactory agreement levels were achieved between ChatGPT and human annotations on the development set, we transitioned to the hypothesis testing stage, with all three annotators involved. In this stage, we employed the fixed finalized prompting strategy, ensuring no overfitting or leakage from the test set.
Our prompting strategy development involved experimenting with recent methodologies such as few-shot in-context learning (ICL) [16, 11, 80, 51, 91] and chain of thought (CoT) [38, 81, 77, 32]. We found that the CoT approach offered the most robust and optimal results for the task of annotating question pairs  $(a_{i},d_{j})\in \mathcal{D}$ . A comprehensive description of the prompt engineering process can be found in Appendix B
4.3 Annotation Evaluations
Hypothesis testing was conducted using a sample of 250 question pairs  $S_{\mathrm{test}}$  from  $\mathcal{D}$ , distinct from the development set. The selected pairs were independently annotated by three human annotators. We used bootstrapping to estimate the  $95\%$  confidence intervals for the macro F1 score, accuracy (agreement), and Spearman's  $\rho$ . Though the sampling temperature is fixed at zero, recognizing small randomness introduced by GPU non-determinism [52], the ChatGPT annotations are run twice and the pairwise comparisons with human annotators are averaged across these runs.
In addition to these prespecified analyses, we conducted post-hoc analyses using an adjudication procedure to create a more stable set of human annotations [53]. We use majority voting [8] between the three human annotators to decide the adjudicated annotation.
The results presented in Table  $\boxed{1}$  show that the rate of ChatGPT-human agreement is within range of human-human agreement. Specifically, ChatGPT's performance is comparable to that of human annotators as measured by F1 and exact agreement (accuracy). The average agreement level for ChatGPT is numerically lower than average human agreement level as measured by Spearman's  $\rho$ . We note that the lowest Spearman's  $\rho$  among two humans is  $46.51\%$ , which is comparable to the average ChatGPT/human's  $\rho = 46.61\%$ , suggesting ChatGPT's performance is still within the range of human-level agreement. All  $p$ -values are larger than the  $\alpha = 0.05$  criterion.
In a post-hoc analysis, we found that comparing ChatGPT to adjudicated human annotations numerically increased the rate of agreement. This provides qualitative evidence for ChatGPT's performance relative to that of human experts. We further analyzed instances where ChatGPT's annotations diverged from those of humans. Interestingly, the nature of these discrepancies was similar to those found between humans, with differences largely revolving around the interpretation of key components in aspect/sub-aspects. For example, ChatGPT occasionally differed from human annotators in
determining the importance of a given component. Detailed examples of ChatGPT's reasoning and a comprehensive error analysis can be found in Appendix B
Table 1: Annotation agreement between humans and ChatGPT. H is human, G is ChatGPT, A is Adjudication, CI is  $95\%$  confidence interval.  $p$ -values correspond to the null hypothesis that there is no difference between avg. H&H and avg. H&G. Higher  $p$ -values indicate less evidence of a difference between ChatGPT and humans.
Metrics	G&A	avg. H&H	avg. G&H	H&H CI	G&H CI	p-value
F1-score (macro)	64.17	58.33	57.46	(52.33, 63.46)	(50.93, 62.79)	0.74
Accuracy	67.40	64.13	62.07	(59.73, 68.80)	(57.67, 66.13)	0.41
Spearman's ρ	52.63	54.31	46.61	(46.87, 61.56)	(38.67, 54.41)	0.07
4.4 Scalability of Annotations