SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The hypothesis testing results support the use of ChatGPT for annotation. At deployment, the pipeline annotated all 165,144 aspect-paper pairs within a span of 24 hours, at a cost under $150. By contrast, human experts typically require approximately 4 minutes per question pair, resulting in an estimated 11,146 hours to annotate the entire dataset. The deployment resulted in a time reduction by a factor of 500, and a cost reduction by a factor of 1,000, without sacrificing annotation quality. Upon completion of the annotation process, we utilized Equations [1] and [2] to compile the results and compute the final rankings for both full-query and sub-query test cases. Anno-GPT could potentially utilize any high-performance LLM to replace ChatGPT and can be adapted for other expert-level tasks, given the availability of a small set of domain expert annotations for validation.
5 Retrieval Results
This section presents the results of testing 17 models discussed in Section 2 on the DORIS-MAE dataset. When available, we trained the model on our CS corpus and denoted the best version as trained in domain (ID), see full training details in Appendix D.1. To contextualize their performance on DORIS-MAE, we compare the results with these models' previously reported performances on various IR datasets, including MS MARCO [57], LoTTE [66], NQ [40], and Wiki-QA [86].
5.1 DORIS-MAE Benchmarking Results
In our benchmarking process for DORIS-MAE, we use complex queries as inputs to these models. We employ a variety of metrics common in the IR/NIR literature for the evaluation, including R@5, R@20, R-Precision (RP), NDCG $^{\text{exp}}_{10\%}$ , MRR@10, and MAP. For fairness, we adopted an alternative approach for models like RocketQA-v2 [63] and ColBERT-v2 [66] that were not designed to handle long queries. For these cases, we allow the models to process the input as either a single text string or sentence-by-sentence, and report the maximum performance achieved. Uniquely among the models that we consider, the ASPIRE models (TSAPire/OTAspire) are designed to handle multi-aspect queries. For brevity, we only report the higher number among these two options for models in Table 2. For more detailed results, refer to Appendix C
To better interpret the results, we compare against a random ranking baseline. In general, the models show consistent behavior, with larger and more general-purpose models (like E5-LargeV2, RocketQA-v2, ada-002, Specter-v2) faring better than the smaller and more specialized ones (like SciBERT [5], ColBERT-v2, BM25, TF-IDF). Though the Aspire models were designed for multi-aspect queries, they do not have strong performance on the complex queries in DORIS-MAE.
When we compare the DORIS-MAE performance of these models with their reported results on traditional retrieval datasets MS MARCO (in Table 3) and NQ (in Table 4), we observe a significant reduction in their performances on DORIS-MAE. We choose metrics for comparison based on what is available in previously published work. The results highlight the challenges posed by DORIS-MAE and suggest gaps in the ability of existing methods to generalize well to complex query retrieval.
Finally, in Table 5, we make a comparison with the model performances on specialized retrieval datasets such as CSFCube [53], RELISH [10], and TRECCOVID [73]. The comparison reveals a
Table 2: Query level performance on full DORIS-MAE. Standard errors are estimated by bootstrapping. ID means a model is trained in domain.
Method	R@5	R@20	RP	NDCGexp10%	MRR@10	MAP
Random	4.41	18.48	16.29	7.31	3.59	19.63
E5-L-v2[74]	16.51 ± 2.05	43.77 ± 2.14	37.46 ± 2.44	25.90 ± 2.15	14.85 ± 2.73	40.49 ± 2.32
RocketQA-v2[63]	15.63 ± 1.88	45.41 ± 2.43	34.36 ± 2.32	30.30 ± 2.26	20.87 ± 3.12	40.18 ± 2.23
ada-002[28]	15.38 ± 1.95	42.84 ± 2.53	35.81 ± 2.67	27.46 ± 2.48	19.88 ± 3.21	40.37 ± 2.55
SimCSE[25]	14.90 ± 1.89	42.62 ± 2.40	35.27 ± 2.34	26.88 ± 2.36	21.19 ± 3.47	39.02 ± 2.35
SPLADE-v2[25]	14.78 ± 1.89	40.14 ± 2.33	31.65 ± 2.38	26.08 ± 2.00	17.82 ± 2.99	37.23 ± 2.26
SPECTER-v2[18]	14.50 ± 2.15	43.36 ± 2.50	33.41 ± 2.33	25.65 ± 2.23	17.19 ± 2.96	37.12 ± 2.10
SPECTERID	13.32 ± 1.76	42.52 ± 2.37	31.55 ± 2.28	21.27 ± 2.03	14.48 ± 2.78	36.02 ± 2.19
TSAspire[54]	14.26 ± 1.80	41.25 ± 2.40	33.81 ± 2.47	26.63 ± 2.05	15.59 ± 2.59	37.00 ± 2.29
SentBERT[62]	14.09 ± 1.88	44.69 ± 2.47	33.79 ± 2.41	21.88 ± 2.07	13.23 ± 2.69	37.75 ± 2.28
OTAspire[54]	13.34 ± 1.56	42.25 ± 2.53	33.63 ± 2.38	25.52 ± 2.29	14.18 ± 2.66	36.70 ± 2.22
ANCEFirstP[85]	13.21 ± 2.02	34.54 ± 2.20	30.51 ± 2.50	20.30 ± 2.02	13.87 ± 2.64	34.53 ± 2.35
SPLADE-v2[23]	11.80 ± 1.86	36.59 ± 2.12	29.90 ± 2.20	21.35 ± 2.12	14.30 ± 2.77	33.98 ± 2.23
LLAMA[71]	12.74 ± 1.82	34.51 ± 2.36	28.33 ± 2.14	16.65 ± 1.68	11.78 ± 2.45	31.29 ± 1.99
SimLM[75]	12.68 ± 1.77	35.67 ± 2.49	28.90 ± 2.42	18.91 ± 1.86	11.29 ± 2.44	33.06 ± 2.34
BM25[72]	8.47 ± 1.80	30.50 ± 2.38	21.94 ± 2.03	13.23 ± 1.97	9.19 ± 2.46	25.99 ± 1.68
ColBERT-v2[66]	8.45 ± 1.46	27.86 ± 2.29	22.33 ± 2.01	12.57 ± 1.71	6.69 ± 2.15	25.80 ± 1.83
TF-IDF[67]	10.71 ± 1.48	29.22 ± 2.25	24.79 ± 2.06	18.25 ± 2.01	12.41 ± 2.53	28.77 ± 1.81
ERNIE[47]	6.49 ± 0.94	22.58 ± 1.72	20.18 ± 1.82	9.66 ± 1.18	3.77 ± 1.06	22.71 ± 1.65
SciBERT[5]	5.13 ± 1.25	17.99 ± 1.69	17.13 ± 1.88	7.50 ± 1.34	3.41 ± 1.57	20.34 ± 1.64
consistent level of difficulty between DORIS-MAE and these completely human-annotated datasets, indicating that DORIS-MAE presents a similarly challenging retrieval task.
Table 3: Comparison with MS MARCO. Stats collected from [85,66,63,75,23].
Ranking Method	MS MARCO MRR@10	DORIS MRR@10
ANCE	33.0	13.87
ColBERT-v2	39.7	6.69
RocketQA-v2	41.9	20.87
SimLM	41.1	11.29
SPLADE-v2	36.8	14.30
Table 4: Comparison with NQ. Stats collected from [66].
Ranking Method	NQ R@20	DORIS R@20
ANCEFirstP	81.9	34.54
BM25	59.1	30.50
RocketQA-v2	83.7	45.41
SimLM	85.2	35.67
Table 5: Comparison with CSFCube, TRECCOVID, and RELISH. Results are from [53], [54].
Ranking Method	RSF	TRECCOVID MAP	RELISH MAP	DORIS-MAE
RP	CSFCube MAP	R@20	RP	MAP	R@20
TSAspire	-	40.26	-	26.24	61.29	33.81	37.00	41.25
OTAspire	-	40.79	-	30.92	62.57	33.63	36.70	42.25
Specter-v2	18.32	-	52.12	28.24	60.62	33.41	37.12	43.36
TF-IDF	14.59	-	39.69	-	-	24.79	28.77	29.22
BM25	13.50	-	42.73	-	-	21.94	25.99	30.50
Up until now, the hierarchical aspect-based structure that Anno-GPT utilizes has been hidden from all the evaluated models because of inability of existing methods to break down a complex query automatically. Even though these structures are not readily available for real-life retrieval methods, they may still hold value once this query decomposition process can be fully automated. To explore this potential, we conduct an experiment where instead of using the original query, each retrieval
method had access to a concatenated string of all aspects within a query, excluding sub-aspects. The results, as illustrated in Table 6 show that Sentence-BERT performs best on four metrics: R@5/20, RP, and MAP. These findings suggest that the use of aspect information could be potentially beneficial to guide retrieval methods.
The hierarchical structure of our dataset can be used to create additional, less complex tasks involving only parts of the query. For instance, by pulling out parts of the query corresponding to 2 Aspects, we are able to generate over 1000 test cases. For this task, we found a significant increase in the number of relevant abstracts. After re-evaluating all models for the sub-query DORIS-MAE, we observed model performance consistent with those seen in previous benchmarks, as indicated in Table 7. Comparing Table 7 with Table 2, we observed noticeably higher numbers for metrics such as RP, NDCG<sup>exp</sup>, MRR and MAP, which are indicators of better model performances on this sub-query task. Since models remain unchanged, this suggests the sub-query task is less challenging than full-query task. This is intuitive since sub-queries are less complex and contain fewer aspects. Overall, the creation of these sub-query tasks underscores the adaptability of our dataset, which could accommodate a range of task complexities under the setting of DORIS-MAE.
Table 6: Ranking performance given model access to aspects. Full table is in Appendix C
Method	R@5	R@20	RP	NDCGexp10%	MRR@10	MAP
ada-002	14.09	42.23	33.56	26.54	20.20	37.62
SentBERT	17.73	45.34	35.67	25.00	15.52	39.87
RocketQA-v2	13.83	43.81	32.59	27.45	16.08	37.90
Table 7: Ranking performance on sub-query (2 Aspects) task. Full table is in Appendix C
Method	R@5	R@20	RP	NDCGexp10%	MRR@10	MAP
ada-002	13.49	40.24	47.35	39.00	24.33	51.67
SentBERT	12.15	36.71	45.08	34.78	20.71	48.96
RocketQA-v2	12.79	39.19	46.47	38.78	23.72	50.81
5.3 Supervised Learning on DORIS-MAE
To assess the utility of our dataset for training IR models, we conducted an experiment where we allocated 40 queries for training and the remaining 60 for testing. Using supervised contrastive learning (SCL), we optimized a margin-based triplet loss as presented in Equation 3. Each triplet, represented as  $(A,P,N)$ , consisted of a query and two abstracts. The higher-ranked abstract in the training data served as the positive instance (i.e.,  $P$ ), with the other functioning as the negative instance (i.e.,  $N$ ). Given each query's candidate pool size of  $\geq 100$ , we derived multiple triplets. This process yielded 3,000 triplets from the designated 40 training queries. Subsequently, we fine-tuned an E5-L-v2 model, a RoBERTa-based text embedding variant with 355 million parameters, over a single epoch.
$$
L (A, P, N) = \max  \left(\frac {<   A , N >}{\left| \left| A \right| \right| \cdot \left| \left| N \right| \right|} - \frac {<   A , P >}{\left| \left| A \right| \right| \cdot \left| \left| P \right| \right|} + m, 0\right) \quad A, P, N \in \mathbb {R} ^ {n}, m > 0 \tag {3}
$$
When evaluating the model on the 60 test queries, we noted a marked improvement across all metrics with the SCL-trained model in contrast to the pre-trained baseline. This comparison can be found in Table 8. These positive outcomes, achieved using supervised contrastive learning on DORIS-MAE, underscore the utility of our train/test split for model fine-tuning.
Note that Table 2 reports models performance on the full DORIS-MAE dataset. We also report models performance on our proposed test set of 60 queries in Appendix C.4
This paper introduces a novel task, Scientific DOdocument Retrieval using Multi-level Aspect-based quEries (DORIS-MAE), aimed at modeling the process of information retrieval in the context of
Table 8: Comparison of SCL vs pretrained. Standard errors are estimated by bootstrapping.
Method	R@5	R@20	RP	NDCGexp10%	MRR@10	MAP
SCL-trained E5-v2	19.57 ± 2.33	52.45 ± 3.17	44.47 ± 3.11	34.67 ± 3.17	23.16 ± 4.28	49.15 ± 3.14
pretrained E5-v2	14.70 ± 1.72	42.38 ± 2.59	38.24 ± 2.94	26.31 ± 2.94	14.53 ± 3.69	40.62 ± 2.85
scientific research. We also present a dataset for DORIS-MAE generated using the Anno-GPT framework.
The results show room for improvement in the performance of current retrieval methods when dealing with DORIS-MAE. Future studies may explore modifications to model architectures and training procedures to better address complex, multifaceted queries.
An understanding of how noise in aspect annotation affects the overall task is an interesting point for future investigation, as it can shed light on how errors in the automated annotations may affect the final candidate pool ranking in DORIS-MAE.
The hierarchical structure of complex queries, as exemplified in DORIS-MAE, is an area that warrants further attention. Future work might include the development of more sophisticated automated query breakdown methods, potentially drawing from advances in question decomposition [36, 92], sequence-to-sequence modelling [42, 43] and semantic parsing [76, 88, 68].
Limitations: DORIS-MAE currently contains queries and abstracts from the computer science domain. Consequently, models trained on this dataset may not generalize well to other disciplines. An extended, multi-domain version of DORIS-MAE is a logical direction for future work. The task of determining aspect relevance is challenging due to the complexity of the abstracts. Improved annotation guidelines and training (for both humans and models) may address this challenge. Finally, while we harnessed LLMs to streamline the annotation, the generation of queries and their aspect decomposition remains manual. We found the development of a reliable, automated query generation pipeline difficult, but anticipate that advances in LLMs might bridge this gap in the near future.
[80] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
[81] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. richter, F. Xia, E. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
[82] D. Weissenborn, G. Wiese, and L. Seiffe. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271-280, Vancouver, Canada, Aug. 2017. Association for Computational Linguistics. doi: 10.18653/v1/K17-1028. URL https://aclanthology.org/K17-1028.
[83] E. W. Weisstein. Bonferroni correction. https://mathworld.wolfram.com/, 2004.
[84] A. Williams, N. Nangia, and S. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, 2018.
[85] L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020.
[86] Y. Yang, W.-t. Yih, and C. Meek. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013-2018, 2015.
[87] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, 2018.
[88] H. Zhang, J. Cai, J. Xu, and J. Wang. Complex question decomposition for semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4477-4486, 2019.
[89] M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534, 2023.
[90] Y. Zhang, Z. Wang, and J. Shang. Clusterllm: Large language models as a guide for text clustering. arXiv preprint arXiv:2305.14871, 2023.
[91] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
[92] B. Zhou, K. Richardson, X. Yu, and D. Roth. Learning to decompose: Hypothetical question decomposition based on comparable texts. arXiv preprint arXiv:2210.16865, 2022.