title: Evaluation of Hypothetical Document and Query Embeddings for Information Retrieval Enhancements in the Context of Diverse User Queries
Creators: name: Jostmann, Marten
Publisher: name: University of Muenster, Institutional Repository MIAMI
Year: 2024
Subjects: subject: 004 Computer science
subject: 330 Economics

description: The task of Information Retrieval (IR), such as obtaining documents, is an important task and the foundation of question-answering and fact-checking systems. The complexity of these retrieval systems evolved over time to enable an effective search of a variety of documents based on users' specific information requirements. However, the quality of the retrieval results highly depends on the question phrasing and structure heavily influenced by the user's age, expertise, background, and aim. This thesis will focus on closing the gap between user-specific query structures and the content of different documents in a large collection. To close this gap Hypothetical Document Embeddings (HyDE)
and Hypothetical Query Embeddings (HyQE) will be investigated utilizing Large Language Models (LLMs). HyDE tries to generate pseudo documents that mimic the content and structure of potentially relevant documents, therefore enabling a more effective document-to-document comparison. HyQE reverses the retrieval process by generating pseudo queries that potentially be posed by the users, allowing query variations to be covered directly by the generation process. Both approaches were evaluated with the help of two real-world test collections out of different domains. HyDE and HyQE could both achieve promising results in terms of effectiveness and robustness. The experiments indicated that these approaches outperform traditional retrieval methods such as BM25 and plain
embedding methods.
description: Figures ..... V
Tables ..... VI
Listings ..... VII
Abbreviations ..... VIII
1 Introduction ..... 1
2 Natural Language Processing ..... 3
2.1 Tokenization ..... 3
2.2 Word Embeddings ..... 4
2.3 Transformers ..... 6
2.3.1 Architecture ..... 6
2.3.2 Generative Pre-Trained Transformer ..... 9
3 Information Retrieval ..... 11
3.1 Motivation and Use-Cases ..... 11
3.2 Data and Information Retrieval ..... 12
3.3 Structure of Information Retrieval Systems ..... 13
3.4 Traditional Retrieval Methods ..... 15
3.5 Large Language Models for Information Retrieval ..... 17
3.5.1 Dense Retrieval using Text Embeddings ..... 17
3.5.2 Query and Document Expansion ..... 21
4 Problem Space Definition ..... 26
4.1 Query Variations and Implications ..... 26
4.2 Objectives and Research Question ..... 28
4.3 Research Methodology ..... 29
5 Design and Development ..... 32
5.1 Hypothetical Document Embeddings ..... 32
5.1.1 Indexing and Retrieval Process ..... 33
5.1.2 Query-Document Concatenation ..... 36
5.1.3 Hypothetical Document Generation ..... 37
5.2 Hypothetical Query Embeddings ..... 40
5.2.1 Indexing and Retrieval Process ..... 41
5.2.2 HyQE for Document Reranking ..... 43
5.2.3 Query Generation ..... 44
5.3 Pipeline Architecture ..... 48
5.3.1 Architectural Building Blocks ..... 48
5.3.2 Dense Embedding Vector Database ..... 50
5.3.3 Retrieval Implementation Details ..... 51
6 Evaluation of Information Retrieval Methods ..... 53
6.1 Experimental Setup ..... 53
6.1.1 Evaluation Procedure and Metrics ..... 53
6.1.2 Dataset Description ..... 56
6.1.3 Creation of the Testset ..... 58
6.2 Effectiveness and Robustness Evaluation ..... 60
6.2.1 Evaluation of the Effectiveness ..... 60
6.2.2 Evaluation of Query Variation Robustness ..... 63
6.2.3 Evaluation of HyQE Reranking Effectiveness ..... 66
6.3 Efficiency and Cost Analysis ..... 68
6.3.1 Indexing and Retrieval Efficiency ..... 68
6.3.2 Indexing and Retrieval Costs ..... 70
7 Discussion of Evaluation Results ..... 72
8 Conclusion and Future Research ..... 76
Appendix ..... 78
References ..... 85