Scientific Document Retrieval using Multi-level Aspect-based Queries
Authors: Jianyou Wang
Kaicheng Wang
Xiaoyue Wang
Prudhviraj Naidu
Leon Bergen
Ramamohan Paturi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: natural language processing
large language model
retrieval
information retrieval
document retrieval
query-based document retrieval
aspect-based document similarity
benchmark dataset
TL;DR: A new and challenging complex query-based Information Retrieval Task Benchmark created with the help of LLM.

Abstract: In scientific research, the ability to effectively retrieve relevant documents based on complex, multifaceted queries is critical. Existing evaluation datasets for this task are limited, primarily due to the high costs and effort required to annotate resources that effectively represent complex queries. To address this, we propose a novel task,  $\textbf{S}$cientific $\textbf{Do}$cument $\textbf{R}$etrieval using $\textbf{M}$ulti-level $\textbf{A}$spect-based qu$\textbf{E}$ries (DORIS-MAE), which is designed to handle the complex nature of user queries in scientific research. We developed a benchmark dataset within the field of computer science, consisting of 100 human-authored complex query cases. For each complex query, we assembled a collection of 100 relevant documents and produced annotated relevance scores for ranking them. Recognizing the significant labor of expert annotation, we also introduce Anno-GPT, a scalable framework for evaluating the viability of Large Language Models (LLMs) such as ChatGPT-3.5 for expert-level dataset annotation tasks. The application of Anno-GPT to annotate the DORIS-MAE dataset resulted in a 500x reduction in cost, without compromising quality. Furthermore, due to the multi-tiered structure of these complex queries, our DORIS-MAE dataset can be extended to over 4,000 sub-query test cases without requiring additional annotation. We evaluated 17 recent retrieval methods on DORIS-MAE, observing notable performance drops compared to traditional datasets. This highlights DORIS-MAE's challenges and the need for better approaches to handle complex, multifaceted queries in scientific research. Our dataset and codebase are available at https://github.com/Real-Doris-Mae/Doris-Mae-Dataset .

DORIS-MAE:
Scientific Document Retrieval using Multi-level Aspect-based Queries
Jianyou Wang
Kaicheng Wang
Xiaoyue Wang
Prudhviraj Naidu
Leon Bergen f
Ramamohan Paturi
Laboratory for Emerging Intelligence University of California, San Diego
La Jolla, CA 92093
[jiw101, kaw036, xiw027, prnaidu, lbergen, rpaturi] @ ucsd.edu
Abstract
In scientific research, the ability to effectively retrieve relevant documents based on complex, multifaceted queries critical. Existing evaluation datasets for this task are limited, primarily due to the high cost and effort required to annotate resources that effectively represent complex queries. To address this, we propose a novel task, Scientific DOcument Retrieval using Multi-level Aspect-based quEries (DORIS- MAE), which is designed to handle the complex nature of user queries in scientific research. We developed benchmark dataset within the field of computer science, consisting of 100 human-authored complex query cases. For each complex query, assembled collection of 100 relevant documents and produced annotated relevance scores for ranking them. Recognizing the significant labor of expert annotation, we also introduce Anno-GPT, scalable framework for validating the performance of Large Language Models (LLMs) on expert-level dataset annotation tasks. LLM annotation of the DORIS-MAS dataset resulted in a 500x reduction cost, without compromising quality. Furthermore, due to the multi-tieres structure of these complex queries, the DORIS-ME MAEE dataset can be extended to over 4,000 sub-query test cases without requiring additional annotation. We evaluated 17 recent retrieval methods on DORIS-MA -MEE, observing notable performance drops compared to traditional datasets. This highlights the need for better approaches handle complex, multifaceted queries scientific research. Our dataset and codebase are available at https//gsthhbbbooom////lDDD/////
Introduction
Scientists often have complex questions that require thorough exploration within various parts of their field (Figure M Finding relevant scientific literature, one of many challenges in this process, can be especially difficult when dealing with multi multi-faceted queries. These queries typically encompass numerous interconnected topics and require an information retrieval (IR) system capable of recognizing and responding to this level of complexity.
"Equal Contribution
1 Equul Contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Query
Aspects
Sub-aspects
The document should describe model tthtt can instruct action autonomous vehicle
am working autonomous driving research
The cocument should include a
need to minimize the risk of autonomous vehicles to
model deployed in real-lime that
the public, especially pedestrians. Therefore, am
>>an inform the autonomous vehicles to avoid crowded streets
trying to build
model that will inform my autonomous vehicles to avoid crowded streets where
The document should introduce model that can count the number of objects from an image or video
where lots of people are walking)
lots of people are walking This model has to be
deployed in real-time. My current idea is to use a computer vision model that can extract and count the
The cocument should include
pedestrian defection model
The document should introduce model that is deployed in real-lme
number of pedestrians from surveilancee footage
that extract count
of pedastrians surveillance foolage videos
videos, It needs to
fast pedestrian detection
model. Foo should go about doing Should my
The document should introduce a model thht can detect pedestrians from image video
model be able
use videos, videos simply just
The document should Include comparison of pedestrian detection models taking images input and other pedestrian detection models
images ?
computational load
would be higher but I could get more accurate
estimate of the walking speed of these pedestrians
The document should introduce model compares performance of computer vision models that take images or videos as input
during a time interval.
taking video input
Figure 1: Example from the DORIS-MAE dataset. Each query is broken down into aspects and sub-aspects. Aspects are semantically distinet components of the query, and sub-aspects are minimal requirements that can be extracted from the aspects.
is integral to many to open- A persistent challenge, however, is the low accuracy in processing complex and multi-intent intent user queries. Despite advanced search engines using semantic understanding and user behavior data in addition to keyword matching [13], these systems still fall short when dealing with complex, multi-intent queries. Neural Information Retrieval (NIR)
Information retrieval, especially query- based document retrieval ig 3
applications, from search engines 9
29
and content recommendations 58
domain question answering (QA) 82
33 15
models 633
85 75
66
77 are primarily trained relatively simple queries (57 17 64
61
Moreover, MacAvaney et al. 49 creates a variety of diagnostic probes, revealing NIR models' instability when processing out-of-distribution textual inputs, potentially because they do not comprehend the deeper semantics text. These limitations can lead to inadequate performance with more complex queries. While open-domain QA models like ChatGPT [[]] have shown significant
87
capability, they frequently produce incorrect or even fabricated responses 46 50 39
39]
and are
prohibitively expensive if directly applied to an entire corpus of scientific documents.
To address these challenges, we introduce a novel task, Scientific Document Retrieval for Multi-level Aspect-based quEries (DORIS-MO -MAE.. DORIS MAA extends query- based and example-base based IR paradigms 200 1 aiming to give users more control in formulating queries using natural language. In order to advance research in this area, we present the DORIS-MAE dataset, comprising 100 unique complex queries in the computer science domain, paired with ranked pools of relevant CS article abstracts. Each query is organized into a hierarchical structure of aspects/sub-aspects which aids annotation.
A distinguishing feature of the DORIS-MAE dataset is its aspect-based hierarchical structure shown in Figure [ This feature aids automation of the annotation process, expands the test case volume without additional annotation, and opens up exploration into the usefulness of aspect information for retrieval methods.
Furthermore, we introduce Anno-GPT, a pipeline for validating Large Language Model (LLM) annotations in statistically sound manner. Our tests reveal that LLM annotations achieve quality comparable to those produced by human annotators, but with considerable savings in both time and cost. Additionally, the design of our pipeline lends itself to easy adaptation for different domains.
In our experiments, we evaluated 17 IR/NIR (Information Retrieval/Neural Information Retrieval) methods using the DORIS-MAN MAE dataset. The methods have worse performance on PORIS-MAL compared to traditional document retrieval datasets, highlighting the complexity of PORIS-MAE. and the need for more sophisticated retrieval methods.
Our main contributions are three-fold. First, by formulating the DORIS-MAE task we are shedding a new light on tackling complex, multi-faceted queries during scientific research. Second, we propose Anno-GPT, a procedure for rigorously evaluating the ability of L.LMs to replace human experts for challenging annotation tasks. Third, we demonstrate the value of breaking complex queries down

---

to an Multi-level Aspect-basee based hierarchical structure, for both annotation accuracy and potential improvements in retrieval methods.
2 Related Work
A range of methods have been developed for document retrieval and re-ranking. Classic retrieval methods like TF-IDF 67 and BM25 TP]
utilize keyword matching between queries and documents, and fall short when the necessary key phrases are not known users. To address this, researchers have applied deep learning techniques to develop NIR models, including RocketQA-v2 G ColBERT-v2 1661, SimLM 775 and SPLADE-v2 23 These models generate latent vector representations for queries and documents, which are effective for many retrieval tasks, but may not be able to handle textual inputs that are outside of their training distributions
49
Other models such as SPECTER [8] and aspect-based ASPIRE focus on calculating document- level similarity. These models are designed to retrieve given an existing paper input, which makes them less suited for open-ended queries. Other retrieval models, such as text-embedding models (ada-002 by OpenAI [28] and E5-large-v2 (74)) and sentence-embedding model Sentence-BERT [62], all face similar limitations as NIR methods in that they struggle to simultaneously represent multiple aspects of query. Despite these challenges, they tend to perform better on our dataset than specialized NIR models for efficient dense passage retrieval, indicating their potential for complex tasks like DORIS-MAE.
primarily contain simple, web-based queries. Models performing well on these datasets often struggle on DORIS-MAE. In contrast, more special- require human expert annotation, and consequently are more limited in and Chan et al. [4] introduce the concept of "aspect" in document retrieval tasks. These datasets provide pre-defined categories for aspects such as "background" or "method". DORIS-MAS extends this work by allowing for open-ended aspects based on the context of the queries. In Narrative-Drive Driven Recommendation (NDR) research 6 2 55
Large-scale traditional IR datasets 57
69
ized IR datasets [56 14
5%
their scale. Mysore et al.
[
user queries are descriptions that capture a range of users' needs. These queries related to everyday tasks, such as finding restaurants and entertainment.
has made it feasible to leverage 90 even outperforming wrowd-workers in some cases 660 70 27 However, these tasks do not necessitate domain-specifer knowledge, and the annotations produced may not measure up to expert annotations. Faggioli al. (22). MacAvaney and Soldaini [88] explored the notion of using LLMs to assist human in relevance judgements, and evaluated on TREC-DL datasets [19, which have single-faceted queries. Our work seeks to extend these efforts to new annotation tasks requiring domain expertise, while introducing separate pipeline stage and a hypothesis testing stage.
Progress in LLMs [ and prompt-engineering
24
45
59
79
15
LLMs such as ChatGPT for annotating NLP tasks
3 Dataset Description
The DORIS-MAE task uses a dataset of 100 complex, human wrrtten queries, each containing between 95 to 226 words. Each complex query is broken down into a hierarchy of aspects and sub- aspects, with aspects representing significant semantic components of the original query, typically a sentence or a few sentences. Sub-aspects further decompose an aspect into simpler, verifiable requirements. Both aspects and sub-aspects are generally one sentence long, though their semantic complexity varies. A complex query can have up to aspects, and each aspect can further contain up sub-aspects. Figure shows an example.
For each complex query Q, we created a pool P of approximately 100 potentially relevant scientific abstracts to evaluate the re-ranking performance of various retrieval/reranking methods. Within the context of Q and its P, the collection of aspects and sub-aspects is together denoted as (with slight abuse of notation) Q == {a,)- )- For any aspect or sub-aspect a; Q, and any paper abstract pj P. they form a question pair (a,, Pj). We generated a total of 165,144 question pairs from the 100 queries, together denoted as DD ((a..P)))-
To compute the relevance of a paper Pj for query Q, we first break the query down into aspects. We then compute the relevance score S(a,, P2), which measures the relevance of paper p5 for aspect
3

---

a, The relevance score S(p, Q) for the query Q is the sum of its aspect relevance scores. An average score S(p, Q) indicates Pj relevant abstract for Q.
Q21
{(, = Σ Ps)
(I)
acEq
Thus, for each complex query and candidate pool, we provide a complete ranking for the list of abstracts. Additionally, we can choose a combination of aspects within complex query Q and concatenate the corresponding sentences to form a sub-query C Q, and can calculate the relevance score between paper abstract Pj and sub-query q.
S(p,lg) = Σ S(a.. P2)
ag
Hence, for any sub-query q C Q and the candidate pool P of Q, also provide complete ground-truth ranking order for P. This allows the dataset to extend to over 4000 sub-query test cases no additional annotation cost.
Candidate pool abstracts are taken from a database of 360,000 computer science papers between 2011 1021-2021 sourced from a We complemented each arXiv paper with its corresponding citation information by cross-matching it on Semantic Scholar 37
3.1 Query Formation
The DORIS-MAE task aims to mirror real-world scenarios where researcher has an incomplete concept for a research project and needs to explore the breadth of existing research to establish solid starting point. The 100 complex queries in our dataset simulate this scenario.
Each query is based on one or more existing research papers. We randomly selected 140 papers from AI, NLP, ML, and CV categories on arXiv. We examined each selected paper's motivation, background, related work, methodology, and experimental results. Using this information, we reverse- engineered complex query designed to reflect the early thought process of the paper's authors. Overall, DORIS- MASE contains 80 queries derived from single paper abstracts, with 20 queries from each of AL NLP, ML and CV. We created an additional 20 composite queries. each integrating ideas from 2-3 abstracts. These composite queries are designed to simulate more interdisciplinary and unexplored research ideas than the other 80 queries.
3.2 Decomposing Queries to Aspects
In this section, discuss our process of decomposing queries into list of aspects and further breaking down aspects into sub-aspects. The guiding principles for determining aspects are as follows:
(i) Each aspect must correspond to prominent and semantically meaningful component in the query (refer to Figure I for this correspondence).
(ii) Each aspect should contain sufficient context to make sense independently, eliminating potential ambiguities.
(iii) Each aspect must be semantically distinct and unrelated to others, ensuring their contents are disjoint and do not overlap.
Some aspects may fall into broader facets such as background, method, or result as defined in Regarding decomposing aspects to sub-aspects, criteria are:
(i) Each sub-aspect should not contain more information than its parent aspect.
(ii) Each sub-aspect should represent semantic segment of its parent aspect
(iii) Different sub-aspects may overlap semantically but each should pose distinct question.
(iv) Each sub-aspect should be simple as possible.
'https///axiv.orr//

---

Given the inherent difficulty in recalling the full details of query while ranking a large candidate pool, the aspect-based hierarchical structure of our complex queries provides a systematic, efficient, and interpretable approach for annotation. This leads to more precise ranking of the candidate pool. As each aspect corresponds a specific part of the original query Q, combining several aspects equivalent to concatenating their corresponding parts in the query, forming semantically coherent sub-queries q C Q.
3.3 Candidate Pooling
To create candidate pool P of approximately 100 paper abstracts for each query Q, we used variety of IR and NIR methods, similar to the approach in [53]. More specifically, we utilized popular IR search algorithms TF-IDF [67 and BM25 7 at different granularities (i.e., sentence and paragraph-Ievel for each query) to retrieve around 80 paper abstracts. We then employed OpenAl's text embedding model, ada-002
2X to extract 20 more paper abstracts. We also use citation signals
from semantic scholar 37 to included any papers that directly cite or are cited by any of the original papers used to create that specific query. Lastly, prevent bias towards saaicon-mtchhing retrieval methods, we excluded from P any papers that authors referenced during query Q's formulation. See Appendix C.6 for sensitivity analysis of our candidate pool construction procedure.
4 Anno-GPT Framework
We propose Anno-GPT, a framework for developing a statistically sound annotation pipeline. We use 2attgpt-3.5-turb--0533
annotate all 165.144 question pairs D == {(aa.,;)) where a, is an aspect/sub-aspedt of a query, and Py is an abstract in the query's ss candidate pool. This strategy minimizes human annotation efforts. The key to this approach lies in breaking down complex queries into simpler aspects and sub-aspects, ensuring an objective and manageable annotation task. Without this structured approach, we found that evaluating an abstract's relevance challenging, due to variability in how partial relevance was assessed. However, the scenario changes significantly when dealing with question pairs (a,, (a.,P)). which only look at a single aspect or sub- aspect of the query. In such cases, assigning coarse-scale relevance score between 0-2 becomes feasible, maintaining a reasonable degree of objectivity.
complex query was
The performance of the annotation pipeline may be influenced by several factors: the procedure for breaking down queries into aspects, the criteria used for scoring query relevance, and the LLM prompt selection 24 45 59 79 [5]. In order to avoid overfitting in this pipeline, our methodology comprises two distinct development and testing stages. The development stage involves optimizing all stages of the pipeline, and using feedback and observed outcomes to iteratively refine this strategy. The testing stage uses a prespecified hypothesis test. The null hypothesis is that there no significant difference between the agreement levels of ChatGPT and humans, and those among humans themselves. After the pipeline has been optimized, we evaluate on a test set. We compute the difference between ChatGPT- agreement and human-human agreement. If this difference is sufficiently close to O, with small confidence interval, we can use the optimized prompt π to annotate the full dataset.
Below we give a summary of the Anno-GPT framework:
i Construct aspect-document question pairs, D=
= [(),,;
ii Select development set Saev D, and use human annotators to score Saev-
iii Select test set Siest C D S dev based on desired power, and use human annotators to score Stest
iv Optimize prompting strategy स and other hyperparameters on Saev. Fix π.
Evaluate fixed π Stest
vi If satisfactory performance is achieved on Stest proceed to use æ to annotate the entire dataset DD
vii Otherwise, repeat steps iii, iv, v for Sdey + Saev Steat and new Stest
is important to note that automated annotation for DORIS-MAE only used a single develop- ment/testing cycle, and therefore did not go into step vii. Therefore, there was no risk of inflated estimates of annotation accuracy due to multiple comparisons 30 X []. If multiple cycles are necessary, then the new Sest must be sufficiently large to avoid these problems.
5

---

4.1 Annotation Guidelines
Our team of annotators consists of three graduate students in computer science, all with at least two years of research experience in NLP, CV, ML, and Al. Annotators, both human and ChatGPT, are asked to score each question pair using a 3-point grading scale (0-2):
Level 0: The abstract is unrelated or does not provide any help to the key components of the aspect or sub-aspect.
Level I: The abstract answers some (or all) key components (either explicitly or within one natural inferential step), but at least one key component is not answered explicitly.
Level 2: The abstract directly answers all the key components of the aspect sub-aspect
We decided to include both direct and indirect coverage under Level 1, acknowledging that distin- guishing between these cases could be challenging and potentially subjective. The detailed guidelines for human annotation can be found in Appendix B
4.2 Optimization of Annotation Pipeline
For the development stage, two annotators independently annotated a randomly selected set Seev of 90 question pairs from the complete set DD of 165,144 pairs. This annotated development set then served as the basis for refining the prompting strategies for ChatGPT.
The quality of annotations was evaluated using three metrics: macro Fl score, exact accuracy (agreement), and Spearman's 's rank correlation coefficient (Spcarman's 's p). These metrics measure the agreement level between annotators, and have been used successfully in similar tasks [53]. After satisfactory agreement levels were achieved between ChatGPT and human annotations on the development set, transitioned to the hypothesis testing stage, with all three annotators involved. In this stage, we employed the fixed finalized prompting strategy, ensuring no overfitting or leakage from the test set.
Our prompting strategy development involved experimenting with recent methodologies such as few-shot in-context learning (ICL.) [16.
11 30 51 91] and chain of thought (CoT) [38 81 77 32
We found that the CoT approach offered the most robust and optimal results for the task of annotating question pairs (a., d,) E D. A comprehensive description of the prompt engineering process can be found in Appendix
4.3 Annotation Evaluations
Hypothesis testing was conducted using a sample of 250 question pairs Stest from D, distinct from the development set. The selected pairs were independently annotated by three human annotators. We used bootstrapping to estimate the 95% confidence intervals for the macro Fl score, accuracy (agreement), and Spearman's p. Though the sampling temperature is fixed zero, recognizing small randomness introduced by GPU non-determinism [52] the ChatGPT annotations are run twice and the pairwise comparisons with human annotators are averaged across these runs.
In addition to these prespecified analyses, we conducted post-hoc analyses using an adjudication procedure to create more stable set of human annotations [53] We use majority voting [8] between the three human annotators to decide the adjudicated annotation.
The results presented in Table [ show that the rate of ChatGPT-human agreement is within range of human-human agreement. Specifically, ChatGPT's performance is comparable to that human annotators measured by F1 and exact agreement (accuracy). The average agreement level for ChatGPT is numerically lower than average human agreement level as measured by Spearman's 's p. We note that the lowest Spearman's 's among two humans is 46.51%, which is comparable to the average ChatGPT/human's 's = 46.61% 61%, suggesting ChatGPT's performance is still within the range of human-level agreement. All p-values are larger than the 0.05 criterion.
In post-hoc analysis, we found that comparing ChatGPT adjudicated human annotations numeri- cally increased the rate of agreement. This provides qualitative evidence for ChatGPT's performance relative to that human experts. We further analyzed instances where ChatGPT's annotations diverged from those of humans. Interestingly, the nature of these discrepancies was similar to those found between humans, with differences largely revolving around the interpretation of key compo- nents in aspect/sub-aspects For example, ChatGPT occasionally differed from human annotators in

---

determining the importance of given component. Detailed examples of ChatGPT's reasoning and a comprehensive error analysis can be found in Appendix B
Table 1: Annotation agreement between humans and ChatGPT. H is human, G is ChatGPT, A is Adjudication. is is 95% confidence interval. p-values correspond to the null hypothesis that there is no difference between avg. H&H and avg. H&G. Higher p-values indicate less evidence of a difference between ChatGPT and humans.
Metrics
G&A
avg. H&H avg. G&H
H&H CI
G&H CI
p-value
Fl-score (macro)
64.17
58.33
57.46
(52.33, 63.46) (50.93, 62.79)
0.74
Accuracy
67.40
64.13
62.07
(59.73, 68.80)
(57.67, 66.13)
0.41
Spearman's p
52.63
54.31
46.61
(46.87, 61.56)
(38.67, 54.41)
0.07
4.4 Scalability of Annotations
The hypothesis testing results support the use of ChatGPT for annotation. At deployment, the pipeline annotated all 165,144 aspect-paper pairs within a span 24 hours, at a cost under $150. By contrast, human experts typically require approximately 4 minutes per question pair. resulting in an estimated 11,146 hours annotate the entire dataset. The deployment resulted in a time reduction by a factor of 500, and cost reduction by a factor of 1,000, without sacrificing annotation quality. Upon completion of the annotation process, we utilized Equations T and - to compile the results and compute the final rankings for both full-query and sub-query test cases. Anno-GPT could potentially utilize any high-performance LLM to replace ChatGPT and can adapted for other expert-level tasks, given the availability of small set of domain expert annotations for validation.
5 Retrieval Results
This section presents the results of testing 17 models discussed in Section on on the DORIS-MAE dataset. When available, we trained the model our CS corpus and denoted the best version as trained in domain (ID), see full training details in Appendix [.I] To contextualize their performance on DORIS- we compare the results with these models' previously reported performances on and Wiki-QA [86].
various IR datasets, including MS MARCO 577. LoTTE
66
NQ 40
5.1 DORIS-MAE Benchmarking Results
complex queries as inputs to these models. We employ variety of metrics common in the IR/NIR literature for the evaluation, including R 5, R @ 20, R-Precision (RP), NDCG approach for models like RocketQA-v2 [63 and ColBERT-v2 (66] that were not designed to handle long queries. For these cases, we allow the models to process the input as either single text string or sentence-by-sentencee and report the maximum performance achieved. Uniquely among the models that we consider, the ASPIRE models (TSApire/OTAApirr) are designed to handle multi- aspect queries. For brevity. we only report the higher number among these two options for models in Table For more detailed results, refer to Appendix C
In our benchmarking process for DORIS-MAE,
10% MRR @ 10, and MAP. For faimesss,
adopted alternative
To better interpret the results, we compare against a random ranking baseline. In general, the models show consistent behavior, with larger and more general-purpose models (like L5-Large- V2, RocketQA-v2, ada-002, Specter-v2) faring better than the smaller and more specialized ones (like SciBERT [ ColBERT-v2, BM25, TF-IDF). Though the Aspire models were designed for multi-aspect queries, they do not have strong performance on the complex queries in DORIS-MAE.
When we compare the DORIS-MAR MAE performance of these models with their reported results on traditional retrieval datasets MS MARCO (in Table 3) and NQ (in Table D. we observe significant reduction in their performances is available in previously published work. The results highlight the challenges posed by DORIS-MAE and suggest gaps in the ability of existing methods generalize well to complex query retrieval.
DORIS-MAE We choose metrics for comparison based on what
Finally, in Table 5 we make a comparison with the model performances on specialized retrieval datasets such as CSFCube [53], RELISH [00, and TRECCOVID TA] The comparison reveals
7

---

Table 2: Query level performance full DORIS-MAE. Standard estimated by bootstrapping. ID means model trained in domain.
Method
R@5
R@ 20
RP
NDCO 10%
MRR @ 10
MAP
Random
4.41
18.48
16.29
7.31
3.59
19.63
F5-I.-v2
16.51 2.05
43.77
2.14
37.46
+: 2.44
25.90
2.15
14.85
2.73
40.49
4: 2.32
RocketQA-v
63
15.63
45.41
2.33
34.36
+: 2.32
30.30
2.26
20.87
40.18
2.23
ada-002 23
15.38
1.95
42.84
2.53
35.81
2.66
27.46
2:48
3:21
40.37
2.55
SimCSE 25
14.90 +: 1.89 42.62
2:2.0
35.27
2.234
26.88
2.36
21.19
3.477
39.02
2.35
SPLADE-v2 25
1.89 1.89
2.33
31.65
42.38
26.08
2:00
17.82
2.99
37.23 4: 2.26
SPECTER-v2
24.50
43.56
33.41
25.65
2223
17.19
37.12 2.10
SPECTER
1.76
42.52
31.55
2
2..27
14,48:
2.78
36.02 + 2.19
TSAspire [44
1.82
41.25
33.81
26.63
2.0
2.05
15.59
2,59
37.00 + 2.29
SemtBERT (62
14.09 1: 1.88 44.69 1:447
33.79
1:211
21.88
2.07
13.23
2.69
37.75 1 2.28
OTAspire 54
13.34 1.56 42.25 2:53
33.63
12.38
25.52
2.29
2.6 14.18
36.70 1 2.22
ANCERRP 33
13.21 1:
34.54
30.51
20.30
13.87
34,53
SPLADE-v2
11.80
2212
29.90
2:20
21.35
2.12
14.30
2.77
33,98 +
LLAMA
12.74 188
34.51
28.33 #214
16.65
= 1.68
11.78
2:45
31.29 + 1.99
75
12.68 1.77
35.67
#249
28.90
18.91
1.86
11.29
2:44
33.06 + 2.34
BM25 72
8.80
30.50
2.38
21.94
13.23
= 1.77
9.19
2.46
25.99 + 1.68
CoIBERT-v2 66
8.45
27.86
2.29
22.33
2.01
1
6.69
25.80 +
TF-IDF 67
10.71 1.48
29.22
2.25
24.79
# 2.66
18.25
201
12
255
28.77
- 1.81
ERNIE 47
6.49 + 0.94
22.58
1.72
20.18
+ 1.82
9.66
= L18
3.77
= 1.6
22.71
1 1.65
SciBERT
5.13 + 1.25
17.99
1.69
17.13
+ 1.88
7.50
1.34
3
157
20.34 + 1.64
consistent level of difficulty between DORIS- MAE and these completely human- annanttn datasets, indicating that DORIS-MAE presents similarly challenging retrieval task.
Table 3: Comparison with MS MARCO. Stats col-
Table 4: Comparison with NQ. Stats collected from M
lected from [3
2
MS MARCO
DORIS MRR@ @ 10
NQ R@20
DORIS R 20
Ranking Method
MRR@ 10
Ranking Method
ANCE
33.0
13.87
ANCEFMeP
81.9
34.54
ColBERT-v2
39.7
6.69
BM25
59.1
30.50
RocketQA-v2
41.9
20.87
RocketQA-v2
83.7
45,41
SimLM
41.1
11.29
SimLM
85.2
35.67
SPLADE-v2
36.8
14.30
Table 5: Comparison with CSFCube, TRECCOVID, and RELISH. Results are from
13
CSFCube MAP
TRECCOVID MAP
RELISH MAP
DORIS-MAE
Ranking Method
R@200
MAP
R@20
TSAspire
40.26
26.24
61.29
33.81
37.00
41.25
OTAspire
40.79
30.92
62.57
33.63
36.70
42.25
Specter-v2
18.32
52.12
28.24
60.62
33.41
37.12
43.36
TF-IDF
14.59
39.69
24.79
28.77
29.22
BM25
13.50
42.73
21.94
25.99
30.50
5.2
Additional Experiments
Up until now, the hierarchical aspect- based structure that Anno-GPT utilizes has been hidden from all the evaluated models because of inability existing methods to break down a complex query automatically. Even though these structures are not readily available for real-life retrieval methods, they may still hold value once this query decomposition process can be fully automated. To explore this potential, we conduct experiment where instead of using the original query, each retrieval

---

method had access to a concatenated string of all aspects within query. excluding sub-aspects. The results, as illustrated in Table D show that Sentence-BERT performs best on four metrics: @ RP, and MAP. These findings suggest that the use of aspect information could be potentially beneficial to guide retrieval methods.
The hierarchical structure of our dataset can be used to create additional, less complex tasks involving only parts of the query. For instance, by pulling out parts of the query corresponding to 2 Aspects, we are able to generate over 1000 test cases. For this task, we found a significant increase in the number of relevant abstracts. After re-evaluating all models for the sub-query DORIS-MAE, we observed model performance consistent with those seen in previous benchmarks, as indicated in Table 4 Comparing Table 7 with Table 2 we observed noticeably higher numbers for metrics such as RP, NDCG 10% MRR and MAP, which are indicators of better model performances on this sub-query task. Since models remain unchanged, this suggest the sub-query task is less challenging than full-query task. This is intuitive since sub-queries are less complex and contain fewer aspects. Overall, the creation of these sub-query tasks underscores the adaptability of our dataset, which could accommodate a range of task complexities under the setting of DORIS-MAE.
Table 6: Ranking performance given model access to aspects. Full table is in Appendix C
Method
R@5
R@200
RP
NDCG 10%
MRR @ 10
MAP
ada-002
14.09
42.23
33.56
26.54
20.20
37.62
SeatBERT
17.73
45.34
35.67
25.00
15.52
39.87
RacketQA-v2
13.83
43.81
32.59
27.45
16.08
37.90
Table 7: Ranking performance on sub-query (2 Aspects) task. Full table is in Appendix c
Method
R@5
@@
RP
NDCG 10%
MRR @ 10
MAP
ada-002
13.49
40.24
47.35
39.00
24.33
51.67
SentBERT
12.15
36.71
45.08
34.78
20.71
48.96
RocketQA-v2
12.79
39.19
46.47
38.78
23.72
50.81
5.3 Supervised Learning on DORIS-MAE
To assess the utility of our dataset for training IR models, we conducted an experiment where we allocated 40 queries for training and the remaining 60 for testing. Using supervised contrastive learning (SCL., we optimized a margin-based triplet loss as presented in Equation 3] Each triplet, represented as (A, P. N), consisted of a query and two abstracts. The higher-ranked abstract in the training data served as the positive instance (i.e., P), with the other functioning as the negative instance (i.e., NN. Given each query's 's candidate pool size of 2 100, we derived multiple triplets. This process yielded 3,000 triplets from the designated 40 training queries. Subsequently, we fine-tuned an E5-L-v2 model, a RoBERTa-based text embedding variant with 355 million parameters, over single epoch.
<A.N> <A,P> + mm,0) A,P,N m [AIIIIIT [|| [|P||
L(A,P,N) = max
(3)
When evaluating the model on the 60 test queries, we noted marked improvement across all metrics with the SCL-trained model in contrast to the pre-trained baseline. This comparison can be found in Table 8 These positive outcomes, achieved using supervised contrastive learning on DORIS-MA MAR,, underscore the utility of our train/test split for model fine-tuning.
Note that Table 2 reports models performance on the full DORIS MAAE dataset. We also report models performance on our proposed test set of 60 queries in Appendix C.4
6 Conclusion and Future Work
This paper introduces a novel task, Scientific DOcument Retrieval using Multi-level Aspect-based quEries (DARIS-MAE), aimed at modeling the process of information retrieval in the context of

---

Table 8: Comparison of SCL pretrained. Standard errors estimated bootstrapping.
Method
R R
R@2 20
RP
NDCG
10%
MRR @ 10
MAP
SCL-trained E5-v2
19.57 + 2.33
52.45 3.17
44.47
3311
34.67
3.17
23.16 4.28
49.15 3.14
pretrained E5-v2 14.70 + 1.72 42.38 + 2.59
38.24 2.94
26.31
2.94
14.53
40.62 2.8
scientific research. We also present dataset for DORIS-MAE generated using the Anno-GPT framework.
The results show room for improvement in the performance of current retrieval methods when dealing with DORIS-MAE, Future studies may explore modifications to model architectures and training procedures better address complex, multifaceted queries.
An understanding of how noise in aspect annotation affects the overall task an interesting point for future investigation, it can shed light on how errors in the automated annotations may affect the final candidate pool ranking in DORIS-MAE.
The hierarchical structure of complex queries, as exemplified in DORIS-MAE, is an area that warrants further attention. Future work might include the development of more sophisticated automated query breakdown methods, potentially drawing from advances in question decomposition [36 92).
sequencc-to-ssqqeene modelling 42 43
and semantic parsing 76
Limitations: DORIS-MAE currently contains queries and abstracts from the computer science domain. Consequently, models trained on this dataset may not generalize well other disciplines. An extended, multi-domain version of DORIS-MAL is logical direction for future work. The task of determining aspect relevance is challenging due to complexity the abstracts. Improved annotation guidelines and training (for both humans and models) may address this challenge. Finally, while we harnessed L.LMs to streamline the annotation, the generation of queries and their aspect decomposition remains manual. We found the development of reliable, automated query generation pipeline difficult, but anticipate that advances in LLMs might bridge this gap in the near future.
10

---

References
[1] H. Abdi et al. Bonferroni and Šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics, 3(01): 2007.
[2] J. Afzali, A. M. Drzewiecki, and K. Balog. Pointree: A test collection for narrative-driven point of interest recommendation. In Proceedings of the 44th International ACM SIGIR Conference
on Research and Development in Information Retrieval, SIGIR '21, page 2478 2484, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380339, doi: URL https: otgg///oioooo/oo 1144/144488833 3463243
10.1 A41114444444353446050
[3] Z. Akkalyoneu Yilmaz, W. Yang, H. Zhang, and J. Lin. Cross-domain modeling of sentence- level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNIP)) pages 3490-3496, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/w/DD19-13322 URL https://
www.bollooooooggg1--11 1352
[4] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.0444 S.ti:2302.04023, 2023.
[5] I. Beltagy, K. Lo, and A. Cohan. Scibert: A pretrained language model for scientific text. In P Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, 2019.
[6] T. Bogers and M. Koolen. Defining and supporting narrative-driven recommendation. In Pro- ceedings of the Eleventh ACM Conference on Recommender Systems, RecSys '17, page 238-242, New York. NY. USA. 2017. Association for Computing Machinery. ISBN 978 478444634 doi: 10.1145/3109859.33 109893. Utt/https:// //dotp org 10. /30095990999991/ 0108899..319989 3109893
[7] S. Bowman, G. Angeli. C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, 2015.
[8] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics. doi: 10. (0.18853/v1/DB 1075. URL https://aclantholll
107/
[9] S. Brin and L. Page. The anatomy of scale hypertextual web search engine. Computer networks and ISDN systems, 30(1-7):107-111 1998.
[10] Brown, R. Consortium, and Y. Zhou. Large expert-curated database for benchmarking document similarity detection in biomedical literature search. Database, 2019. 10 2019. ISSN doi: 10.1093/dattbbas//bz0555 URL https:://oo// org 1093/ /at03///aabba////// baz085 baz085.
[11] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert-Vonr G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, Bhess, Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877- cccpppprrrcppppprrpp.peppppp
1901. Curran Associates, Inc., 2020. URL f
[12] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, page 89-96, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931 1805 doi: 10. 102351. 1003363. //doi.org/10. 1145/110235111111 1102363
URL https:
[13] S. Buttcher, C. L. Clarke, and G. V. Cormack. Information retrieval: Implementing and evaluating search engines. Mit Press, 2016.

---

[14] J. Chan, J. C. Chang, T. Hope, D. Shahaf, and A. Kittur. Solvent: A mixed initiative system for finding analogies between research papers. Proc. ACM Htmn.-Comput Interact., 2(CSCW), nov 2018. doi: 10. 145/3274300. htl.hhtps:://od org/ 10. #145/3274300
[15] D. Chen and W.-t. Yih. Open-domain question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 34-37, On- line, July 2020. Association for Computational Linguistics. doi: 10.18665/w1/22220acll- -uttrriiia... 8. httpstss: http://acaaanthhogoo/o org/2020 nuc--tutoraass...
[16] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv
preprint arXiv:2204.003/1 2022.
[17] Clark, Cowhey, O. Etzioni, T. Khot, Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
arXiv: 1803.05457, 2018.
[18] A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. S. Weld. Specter: Document-level representation learning using citation-informed transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270-22000 2282, 2020.
[19] N. Craswell, B. Mitra, E. Yilmaz, and D. Campos. Overview of the trec 2020 deep learning track, 2021.
[20] K. Dimitriadou, O. Papaemmanouil, and Y. Diao. Explore-by-example: An automatic query steering framework for interactive data exploration. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data, pages 517-528, 2014.
[21] B. Ding, C. Qin, L. Liu, L. Bing, Joty, and B. Li. Is gpt-3 good data annotator? arXiv preprint arXiv:2212. 10450, 2022.
[22] G. Faggioli, L. Dietz, C. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas,
M. Potthast, B. Stein, and H. Wachsmuth. Perspectives on large language models for relevance judgment, 2023.
[23] T. Formal, C. Lassance, B. Piwowarski, and S. Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109. 109.10086, 2021.
[24] T. Gao, A. Fisch, and D. Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 38 3816-3830, 2021.
[25] Gao, X. Yao, and D. Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/2000 111663//122211..mmmmmm......
URL https://
aclanthology. org/2021.111111 mmnnp-main..... 552.
[26] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumeé III, and
K. Crawford. Datasheets for Datasets. arXiv: /oriv:/803.000 880,00000 [cs]. Jan. 2020.
[27] Gilardi, M. Alizadeh, and M. Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303 15000..03333..5660 2023.
[28] R. Greene, T. Sanders, L. Weng, and A. Neelakantan. New and improved embedding model.
hhooo//log//eweee and improved-emmedddingg model/ 2022. Accessed:
2023-06-03,
[29] R. Guha, R. McCool, and E. Miller. Semantic search. In Proceedings of the 12th International Conference on World Wide Web, WWW '03, page 700-709, New York, NY, USA, 2003. Association for Computing Machinery. ISBN 1581136888 doi: 10.1 10.114577751522.77550 URL
[30] Hsu. Multiple comparisons: theory and methods. CRC Press, 1996.
[31] Y. Hu, Y. Koren, and C. Volinsky. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE international conference on data mining, pages 263-272. Ieee, 2008
[32] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147 PMLR, 2022.
12

---

[33] Q. Jin. Z. Yuan, G. Xiong, Q. Yu, H. Ying, C. Tan, M. Chen, S. Huang, X. Liu, and S. Yu. Biomedical question answering: A survey of approaches and challenges. ACM Comput. Surv.. 55(2), jan 2022. ISSN 0360-0300. doi: 10.1 10.1145/3490238. URL http::tts///e //doi org/ 10. 1145/ 3490238
[34] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, 2020.
[35] O. Khattab and M. Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 48, 2020.
[36] T. Khot, D. Khashabi, K. Richardson, P. Clark, and A. Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Téchnologies, pages 1264-1279, 2021.
[37] R. M. Kinney, C. Anastasiades, R. Authur, I. Beltagy, J. Bragg, A. Buraczynski, I. Cachola,
S. Candra, Y. Chandrasekhar, A. Cohan, M. Crawford, D. Downey, J. Dunkelberger. O. Etzioni, Hu, Huff, D. King, S. Kohlmeier, B. Kuehl, M. Langan, D. Lin, H. Liu, K. Lo. J. Lochner, K. MacMillan, T. Murray, C. Newell,
R. Evans, S. Feldman, J. Gorney, D. W. Graham,
S. Rao, S. Rohatgi. P. L. Sayre, Z. Shen, A. Singh, L. Soldaini, S. Subramanian, A. Tanaka, A. D. Wade, L. M. Wagner, L. L. Wang, C. Wilhelm, C. Wu, J. Yang, A. Zamarron, M. van Zuylen, and D. Weld. The semantic scholar open data platform. ArXiv, abs/2301.10140, 2023.
[38] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2211332 Curran Associates, Inc., 2022. URL http:tttt//pprcceedingg wtw.eeeediiinnnnninsssreeeeeeeee ct/pcc/pppppppc.cccp Saitapprrfilesppaeeeee Filer/paper/ Paper- Conffrr-Coffeeence.e pdf
[39] L. Kuhn, Y. Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.0966 2023.
[40] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
[41] J. Lee, K. Lee, and G. Kim. Personalized academic research paper recommendation system. arXiv preprini arXiv: 3304.5457,20 2013.
[42] Z. Li, J. Cai, S. He, and H. Zhao. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3203-3214, 2018.
[43] B. Lin, Z. Yao, J. Shi, S. Cao, B. Tang. S. Li, Y. Luo, J. Li, and L. Hou. Dependency parsing via sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7339-73377 7353, 2022.
[44] M. Lissandrini, D. Mottin, T. Palpanas, and Y. Velegrakis. Example-based search: A new frontier for exploratory search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR 19, page 1411-1412, New York, NY. USA, 2019. Association for Computing Machinery. ISBN 97814503617299 doi: 00.145533311144333333
URL http:: http::::/dooi.or//11// /145/33311111 33313387
[45] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict:
A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35 2023.
[46] T. Liu, Y. Zhang, C. Brockett, Y. Mao, Z. Sui, W. Chen, and B. Dolan. A token-level reference- free hallucination detection benchmark for free-form text generation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: pages www.:2222..1-1..444441111 URL batttss:/aclaanthooooo wwwpollooooooggggggoooooo 2022.022222222 ao2..acllo..................ol.. -logg. 464
10. 1
13

---

[47] Y. Lu, Y. Liu, J. Liu, Y. Shi, Z. Huang, S. F. Y. Sun, H. Tian, H. Wu, S. Wang, D. Yin, and H. Wang. Emii-seaacch: Bridging cross-encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval, 2022.
[48] S. MacAvaney and L.. Soldaini. One-shot labeling for automatic relevance estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, page 2230-2235, New York, NY. USA, 2023. Association for Computing Machinery. ISBN 9781450394086.
doi: 10.1 01144///53918..359222 URL
https://doi..org/ org/10. /141/1355511 3539618 3592032
[49] S. MacAvaney, S. Feldman, N. Goharian, D. Downey, and A. Cohan. ABNIRML: Analyzing the behavior of neural IR models. Transactions of the Association for Computational Linguistics. 2022
10:224-233 239, 2022. doi: 10.1162/tucl a 00457. URL
tacl- 1.13
[50] Manakul, A. Liusie, and M. J. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint (tt.:23330008996 2023.
[51] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048- 11064, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguisties. URL htttps://aclaataattti httphhoooooy.rg//222222/23 org/2022. emnlp commlp-main main. 759
[52] M. Morin and M. Willetts. Non-determinism in tensorflow resnets. arXiv preprint arXiv:2001. /396, 2020.
[53] S. Mysore, T. O' Gorman, A. McCallum, and H. Zamani. Csfcube test col- lection of computer science research articles for faceted query by example. In
J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Pro- cessing Systems Track Datasets and Benchmarks, volume 1. Curran, 2021. URL 2021/file/
newrips. cttpiippp.cpppppprffrieeeee fiaesrfffllsppprrreeeeeeee Paper rounde.ruddd. pdf
[54] S. Mysore, A. Cohan, and Hope. Multi-vector models with textual guidance for fine-grained scientific document similarity. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4453 4470, 2022.
[55] S. Mysore, A. McCallum, and H. Zamani. Large language model augmented narrative driven recommendations, 2023.
[56] M. Neves, D. Butzke, and B. Grune Evaluation of scientific elements for text similarity in biomedical publications. In Proceedings of the 6th Workshop on Argument Mining, pages 124-135, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/ 11/W19-4515. URL httpps://clanttholiet otg//199.... 4515
[57] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary. R. Majumder, and L. Deng. Ms marco: human generated machine reading comprehension dataset. choice, 2640:660, 2016.
[58] M. Ostendorff. Contextual document similarity for content-based literature recommender systems. arXiv preprint arXiv:2008. 00202, 2020.
[59] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[60] C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06413 2023.
[61] S. Reddy, D. Chen, and C. D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266 2019.
[62] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIPP), pages 3982-3992, 2019.
14

---

[63] R. Ren, Y. Qu, J. Liu, W. X. Zhao. Q. She, H. Wu, H. Wang, and J.-R. Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825-2835, 2021.
[64] M. Saeidi, M. Bartolo, P. Lewis, Singh, T. Rocktäschel, M. Sheldon, G. Bouchard, and S. Riedel. Interpretation of natural language rules in conversational machine reading. In Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087-2097, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D11-- 1233. URL https: http:///aaaaatolgy..ooooo
1233
[65] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. Le Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Leurning Representations, 2022.
[66] K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3715-3734, 2022.
[67] K. Sparek Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1) ():11 11-1,, 1972.
[68] Y. Sun, I.. Zhang, G. Cheng, and Y. Qu. Sparqa: skeleton-based semantic parsing for complex questions over knowledge bases. In Proceedings the AAAI conference on artificial intelligence, pages 8952-8959, 2020.
[69] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[70] Törmberg. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304..665 065888, 2023.
[71] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv preprint 2302. 2023.
[72] A. Trotman, A. Puurula, and B. Burgess. Improvements to bm25 and language models examined. In Proceedings of the 2014 Australasian Document Computing Symposium, pages 58-65, 2014.
[73] E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soboroff, and L. L. Wang. Trec-covid: Constructing a pandemic information retrieval test collection. SIGIR Forum, 54(1), feb 2021. ISSN 0163-5840. doi: 10. 10.1145//4519664.4515665 htRl:ttps:://doo
org/ 10. 101111141144990 3451964..199 3451965
[74] I.. Wang, N. Yang, X. Huang, B. Jiao, L.. Yang, D. Jiang, R. Majumder, and F, Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03553, 2022.
[75] I. Wang, N. Yang, X. Huang, B. Jiao, L. Yang. D. Jiang, R. Majumder, and F. Wei. Simim: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:220..02578. 2022.
[76] S. Wang, A. Shrivastava, and S. Livshits. Treepiece: Faster semantic parsing via tree tokeniza- tion. arXiv preprint arXiv:2303.1 17161, 2023.
[77] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11171, 2022.
[78] Y. Wang, Y. Hou, H. Wang, Z. Miao, S. Wu, Q. Chen, Y. Xia, C. Chi, G. Zhao, Z. Liu, X. Xie, H. Sun, W. Deng, Q. Zhang, and M. Yang. neural corpus indexer for document retrieval. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25600-25614. Curran Associates, Inc., 2022. URL htt.
www.:es..eeiiii.. cc/ wwwp:ippppcppppp.p..eeeep pappp_rppplll files/ wttpees/ppppp//2////122 Paper-Confeeeeeee Conferencee pdf
[79] Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. Le. Finetuned language models are zero-shot learners. arXiv preprint
2021.
15

---

[80] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv: 20X.::200607682, 2022.
[81] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc.. 2022. URL UtL/ nwwrteiinnnnnee...spe.e..... cc/ pwpp//ppper____p files 96656661SSA44441111100///164--p2
pdf
[82] D. Weissenborn, G. Wiese, and L. Seiffe. Making neural QA simple as possible but not simpler. In Proceedings of the 2 Conference on Computational Natural Language Learning (CoNLL 2017), pages 271-280, Vancouver, Canada, Aug. 2017. Association for Computational Linguistics, doi: 18.18653/v 1/K17-1028. URL acl aaattpoo///..a org/
[83] E. W. Weisstein. Bonferroni correction. https://matwworld. wolfrum. com/, 2004.
[84] A. Williams, N. Nangia, and S. Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume (Long Pupers), pages 1112-1122, 2018
[85] L. Xiong, C. Xiong, Y. Li, K.-F, Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk. Approx- imate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint WxXiv22077008800 2020.
[86] Y. Yang, W.-t. Yih, and C. Meek. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference empirical methods in natural language processing, pages 2013-2018, 2015.
[87] Z. Yang, P. Qi, Zhang. Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa:
A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, 2018.
[88] H. Zhang, Cai, Xu, and Wang. Complex question decomposition for semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4477-4488 2019.
[89] M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith. How language model hallucinations can snowhall. arXiv preprint arXiv:2305. 13534, 2023.
[90] Y. Zhang, Z. Wang, and J. Shang. Clusterlim: Large language models as a guide for text clustering. arXiv preprint arXiv:2305.14877 2023.
[91] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate before use: Improving few-shot
performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
[92] B. Zhou, K. Richardson, X. Yu, and D. Roth. Learning to decompose: Hypothetical question decomposition based on comparable texts. arXiv preprint arXiv:2210.16865, 2022.
16


Reviews:
[{"review_id": "N0aIaFmLNF", "reviewer": "Reviewer_XCsh", "summary": "", "questions": "", "limitations": "As mentioned above, the limitations section is a bit light, its more about the future work possibilities but there could be more discussions on how the existing methods (including both query decomposition and anno-GPT) have their limitations in achieving higher metric scores.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\nThe idea and process behind generating sub-queries or aspects of a complex query seems to be sound. The evaluation of chatGPT generated annotations and comparison with human annotations is thorough. \n\nThe method of auto-generating annotations for complex queries leads to a significant reduction in costs and time (1000 times and 500 times respectively). This would be useful for the broader research community in  \n\nI like the fact that the complexity of the dataset is compared with other datasets for several models. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "c6Dzflb8hZ", "reviewer": "Reviewer_gbm8", "summary": "", "questions": "", "limitations": "Not discussed.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper presents a useful resource for an interesting task.\n- The dataset seems well documented.\n- The annotation experiment in the paper is reasonably conducted and could be illustrative for future work.\n- The experiments in the paper seem thorough.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "O6XxSgXt5O", "reviewer": "Reviewer_uQNo", "summary": "", "questions": "", "limitations": "I haven't found an explicit mention of self-reported limitations. \nMy suggestions in \"Opportunities for Improvement\", if not addressable directly, can be listed as such.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The major contribution is the introduction of a dataset of human-authored query case which are long and are composed of multiple aspects.\nThis is an imporant challange as in specialized domains humans do requires long and multi-faceted questions.\nThis can trigger furhter development of novel approaches in IR.\nThe hierarchical aspect-based structure for instance allows to investigate models to automaically break down queries into aspects\nand multi-turn question answering systems.\n\nThe second relevant contribution is the AnnoGPT. The authors detailad a systematic and scientifically sound approach to deploy LLM such as ChatGPT to annotate datasets with relevance labels, a task which is notably laborious and time consuming.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tqVTSyl2W4", "reviewer": "Reviewer_xrVv", "summary": "", "questions": "", "limitations": "A clear limitation of this work is that it is restricted to ML-flavoured scientific papers. I would be willing to bet the style of query for complex scientific questions varies quite a bit across fields. As it stands, it is sort of glossed over by mentioning that Anno-GPT _could hypothetically_ be used to annotate examples from other domains.\n\nHowever, I would have expected more of a discussion on this limitation in the paper and its possible consequences, e.g. would a system trained or developed on DORIS-MAE be reasonably expected to extend to other domains? Or would an entirely new dataset need to be produced? What would be the benefits or trade-offs of a _mixed domain_, multi-level aspect-based query annotated dataset?", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Complex, multifaceted queries for scientific document retrieval is an important and exciting area of research.\n- There is a lack of labelled data and benchmarks for this task, so the DORIS-MAE dataset fills an obvious need and benefits the IR research community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "F1PbofC860", "reviewer": "Reviewer_vmkV", "summary": "", "questions": "", "limitations": "No limitation section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The effort to provide dataset on long queries with multiple aspect is a meaningful direction. There are some interesting perspectives in the paper, such as just use ChatGPT to annotate sub-aspect, which is a simpler task.\n\n2. The paper is clearly written. The annotation process is clear and the process to make sure ChatGPT annotation quality is well discussed, which the reviewer really appreciate.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a new dataset (DORIS-MAE) consisting of 50 complex computer science queries, each paired with 100 ranked relevant document abstracts. The queries are organized hierarchically into semantic aspects, facilitating annotation. The authors introduce AnnoGPT, an annotation framework utilizing ChatGPT to expedite document labeling, demonstrating its efficiency compared to human annotators. The paper benchmarks state-of-the-art models on the DORIS-MAE dataset, revealing its high query complexity. Furthermore, it compares these models' performance on DORIS-MAE to standard public datasets like MS-MARCO, confirming the dataset's increased difficulty.  The reviewers have expressed a favorable view of the work. It is evident that the authors have dedicated significant efforts towards addressing the reviewers' comments and concerns. In light of the diligence exhibited by the reviewers in their evaluation, it is advisable for the authors to meticulously consider and incorporate all pertinent feedback into their paper prior to the final submission.
Decision: Accept (Poster)